Results 1 to 18 of 18
  1. #1
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    Losing Tons of Money Because of Server Problems?
    Thought I would share something that I've been struggling for awhile. Perhaps it will help others (Merchants, pay attention!) Ever since Christmas I've been "aware" that I've been losing money during high traffic periods for some reason. Its always been pretty confusing to me, because I have way overbought on hardware. e.g. hardware load balancer, solid state drives, dual CPU with 8 cores each, etc. I guess I've been somewhat aware that my setup could be "disk bound", but I've been in a state of denial. It just shouldn't happen. (queue the microsoft sucks music).

    One of the problems with diagnosing this issue is that traffic is supposed to be distributed in a "smart" manner by the load balancer. I do pin requests to a server by IP address though in order to take advantage of in memory caching of dynamic content. In theory, I should be able to run a local resource monitoring tool on both servers and see when problems arise. I confess, the windows resource monitor is a cryptic POS which I find difficult to use and understand.

    So... yesterday I decided to try more advanced web monitoring to see what would happen. I stumbled upon a website called "alert site by smartbear". Here they are: http://alertsite.com The basic idea is to monitor your website just as it would be used by the end user. So, full page loads, examining page objects, checking out the javascript, etc. They have a 30 day full featured free trial, FYI. I was looking for "proof" that my revenue slowdowns were due to website slowdowns.

    So I setup a couple of monitors on key pages and had the entire page and all of its elements checked every 5 minutes through the main address (e.g. load balancer). So checkout the attached image!

    In the alertsite interface you can click on the dot's in the graph and get a breakdown of the time taken for each element on the page. It appears to me that I have a disk response issue. My IO system is too busy and that causes the slowdowns. I must speed up or eliminate file I/O, in some manner. Today I am offloading all static images to my CDN. I might also offload my javascript. I am creating a sub-domain for each file type to parallelize the downloads. My goal is to get all the dots in the attached graph down to 1 - 1.5 seconds. I'll report back if my revenue spikes once I fix this performance.

    So, alertsite is very useful, but... They are way high end. Prepare to be confused. I had a salesman call me seconds after signing up for the trial. I thought it was weird that he offered me a consultation with a solutions engineer, then I tried using the darn thing... As soon as I logged in I thought to myself, damn, they are the Omniture of website monitoring. Super powerful, super customizable, super confusing...

    So, how did I know I was losing money? Merchants, once again, pay attention... I track YOY sales growth and correlate it to traffic levels. Over the holidays I saw something weird. On Black Friday and Cyber Monday my YOY sales growth tanked when my YOY traffic growth spiked. The trend established in the week leading up to Black Friday totally reversed, even when other members on ABW were talking about record days....
    Attached Images Attached Images
    Merchants, any data you provide to Google Shopping should also be in your affiliate network datafeed. More data means more sales!

  2. Thanks From:

  3. #2
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    Location
    Texas
    Posts
    1,268
    very interesting. your setup never ceases to amaze superCool

    are you sure these are users and not bots? funny how the bad response is almost evenly distributed even across the middle of the night/early morning and through the day.

  4. #3
    ...and a Pirate's heart. Convergence's Avatar
    Join Date
    June 24th, 2005
    Posts
    6,918
    It's incredible how the slightest "lag" in performance can result in lost dollars.

    Basically, was it page load speeds due to additional traffic that was hurting you or do you think because of slow response times you could have be been dinged by the search engines and traffic was throttled to your site?
    Salty kisses, Sandy toes, and a Pirate's heart...

  5. Thanks From:

  6. #4
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    Quote Originally Posted by superCool View Post
    very interesting. your setup never ceases to amaze superCool

    are you sure these are users and not bots? funny how the bad response is almost evenly distributed even across the middle of the night/early morning and through the day.
    Oh, right now it is bots, without a doubt. But they are crowding out the real users. I've got rogue bots, which I've been IP banning, but they are on a botnet of home computers with a virtually unlimited supply of IP addresses. I've also got a huge amount of various valid search engine bots that I NEED to give unrestricted access.

    What caused me to look again... Bing webmaster tools was telling me that my crawl-delay directive for the bing bot was inhibiting my presence in their search engine. (I told it to request at most 1 page every second). I needed to take that restriction away so I could have more pages in their index, but I needed to address the, "heavy bot activity = lower revenue" problem...
    Merchants, any data you provide to Google Shopping should also be in your affiliate network datafeed. More data means more sales!

  7. Thanks From:

  8. #5
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    Quote Originally Posted by Convergence View Post
    It's incredible how the slightest "lag" in performance can result in lost dollars.

    Basically, was it page load speeds due to additional traffic that was hurting you or do you think because of slow response times you could have be been dinged by the search engines and traffic was throttled to your site?
    I think it was both. I think Google is throttling and the user experience suffers.
    Merchants, any data you provide to Google Shopping should also be in your affiliate network datafeed. More data means more sales!

  9. Thanks From:

  10. #6
    ...and a Pirate's heart. Convergence's Avatar
    Join Date
    June 24th, 2005
    Posts
    6,918
    Recently added a new server on steroids - hate setting up new servers, BTW.

    Our main site is setup a little different as each niche is a sub-domain on it's own account. This allows a heavy traffic niche to be moved to it's own server if need be. Also, allows multiple bots from one search engine to index the site at once instead of one search engine bot on only "one site".

    We've been fortunate - traffic/sales/commissions have been on the rise since the holidays. But noticed when we were setting up the new server that one of the niche categories/sub-domains has really dropped off (the increases were on the other niche categories/sub-domains). The category that dropped off USED to be the bread and butter of the site. Now it's a fraction. We're in the process of paying more attention to that niche - it's either a server or software issue. Haven't figure out which, yet. Going over stats to try and figure out exactly when the drop occurred - Before or after the new server.

    That's why we like our niche category/sub-domain setup - if one section has an issue, the entire site isn't penalized...
    Salty kisses, Sandy toes, and a Pirate's heart...

  11. Thanks From:

  12. #7
    Beachy Bill's Avatar
    Join Date
    November 20th, 2005
    Posts
    8,266
    I'm sure there are many merchants with a much less robust system than you have. That may be causing them and, consequently, us (you, me and all other affiliates) a loss of sales during high traffic loads.

    I'd be willing to bet there are even some merchants running an e-commerce site on Five-Buck-A-Month-Hosting.
    Bill / Marketing Blog @ 12PM - Current project: Resurrecting my "baby" at South Baltimore..
    Cute Personal Checks and Business Checks
    If you are too busy to laugh you are too busy.

  13. Thanks From:

  14. #8
    ...and a Pirate's heart. Convergence's Avatar
    Join Date
    June 24th, 2005
    Posts
    6,918
    Quote Originally Posted by Bill View Post
    I'm sure there are many merchants with a much less robust system than you have. That may be causing them and, consequently, us (you, me and all other affiliates) a loss of sales during high traffic loads.

    I'd be willing to bet there are even some merchants running an e-commerce site on Five-Buck-A-Month-Hosting.
    That can be true if the images are hosted on the merchant's server. Network issues/slowdowns can occur as well.

    Our goal is by year's end to host all images. Nothing a fist full of hundreds won't cure, lol...
    Salty kisses, Sandy toes, and a Pirate's heart...

  15. #9
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    Quote Originally Posted by Bill View Post
    I'm sure there are many merchants with a much less robust system than you have. That may be causing them and, consequently, us (you, me and all other affiliates) a loss of sales during high traffic loads.
    I know this to be true and it inhibits their growth and of course limits our revenue. If you try out affiliate links during Black Friday or Cyber Monday, you will find many of the smaller merchant sites are completely bogged down.
    Merchants, any data you provide to Google Shopping should also be in your affiliate network datafeed. More data means more sales!

  16. #10
    ...and a Pirate's heart. Convergence's Avatar
    Join Date
    June 24th, 2005
    Posts
    6,918
    Quote Originally Posted by isellstuff View Post
    I know this to be true and it inhibits their growth and of course limits our revenue. If you try out affiliate links during Black Friday or Cyber Monday, you will find many of the smaller merchant sites are completely bogged down.
    But this is a different issue altogether - their server and not yours (unless you're pulling in images from their server)...
    Salty kisses, Sandy toes, and a Pirate's heart...

  17. #11
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    Quote Originally Posted by Convergence View Post
    But this is a different issue altogether - their server and not yours (unless you're pulling in images from their server)...
    I think we were both looking at this thread as a "help" and "evangelism" thread for Merchants, independent of my problem. In other words, look what isellstuff found, Merchants, do you have the same problem?
    Merchants, any data you provide to Google Shopping should also be in your affiliate network datafeed. More data means more sales!

  18. Thanks From:

  19. #12
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    Ok, so here's an update...

    I was already using a content delivery network for my product images, but a couple of days ago I offloaded ALL of my images to the CDN. This was all the little graphics I used on my site in the layout/UI. Think buttons, background images, arrows, etc.

    Offloading all of these small images to the CDN had a really significant positive affect on performance. I was actually really surprised by how much faster my website became. I should have done this a long time ago. There has also been a nice bump in revenue over the last two days. I hope it is related.

    I've also been running extensive performance logging on my servers to check disk queue length. I never saw a problem over hours of testing. My servers are not disk bound.

    I've been running alertsite monitoring via different geographical regions in the US and the results vary widely. So its beginning to look like network latency. The next step might be to combine all of my javascript files into one or two files that are also offloaded to a CDN. I could also combine all of my small images into one sprite. What you want is to minimize the number of network calls for each page. Right now, I've got 30 or more... This could be reduced down to 2-3.

    I've also got to get a better load balancer. I'm thinking about trying the Citrix Netscaler VPX. http://www.citrix.com/netscalervpx I might make the transition in June. I want to have finer control on traffic patterns so I can get a better idea of how my setup is operating. Right now my existing LB is too much of a "black box"
    Last edited by isellstuff; May 18th, 2014 at 01:14 PM.
    Merchants, any data you provide to Google Shopping should also be in your affiliate network datafeed. More data means more sales!

  20. #13
    ...and a Pirate's heart. Convergence's Avatar
    Join Date
    June 24th, 2005
    Posts
    6,918
    Quote Originally Posted by isellstuff View Post
    What you want is to minimize the number of network calls for each page. Right now, I've got 30 or more... This could be reduced down to 2-3.
    Yikes - 30? We're still trying to get it under 6 from around 10 (depending on the page)...
    Salty kisses, Sandy toes, and a Pirate's heart...

  21. #14
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    Quote Originally Posted by Convergence View Post
    Yikes - 30? We're still trying to get it under 6 from around 10 (depending on the page)...
    I'm including all the images.... But yeah, I guess its way excessive. I just launched the IE dev tools, cleared my browser cache and requested a page. There were 68 items requested. I should do something about that huh?
    Merchants, any data you provide to Google Shopping should also be in your affiliate network datafeed. More data means more sales!

  22. #15
    ...and a Pirate's heart. Convergence's Avatar
    Join Date
    June 24th, 2005
    Posts
    6,918
    Are you only looking at external calls/requests - outside your domain?
    Salty kisses, Sandy toes, and a Pirate's heart...

  23. #16
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    Quote Originally Posted by Convergence View Post
    Are you only looking at external calls/requests - outside your domain?
    I was counting everything. My primary page has 22 small images that are used for layout. I've got two internal JS files. 5 external JS files, 15 product images, and a bunch of other resources loaded by third parties.

    I've been researching how to minimize network latency. There is a bit of conflicting information. Do domain shard, don't domain shard. Use a sprite, use small images to give the allusion of fast loading, etc. I guess the right answer has changed as browsers have advanced and network speeds have improved.

    Geographically distributing all of your content is probably the best answer. Too expensive for me, aside from the CDN.
    Merchants, any data you provide to Google Shopping should also be in your affiliate network datafeed. More data means more sales!

  24. Thanks From:

  25. #17
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    This stuff is complex.... Here's another possible issue..., aggravated by a stupid limitation from my host. So right now my load balancer is a "shared" load balancer in that I rent a portion of a LB from Softlayer. This means that I get 1,000 sessions at any one time. The session is defined by the "keep-alive" time-out, currently 15 seconds which can be extended by subsequent page requests. My LB will drop connections over 1,000 and not tell me and also use interpretation on the connection graphs such that a spikes lasting a few seconds are smoothed out. (WTF, right?)

    So, not that this is a big issue yet, but it will be towards the end of the year. It's a social media DDOS. So basically I'm using a scheduler to do my social media posts now. I try to pick a time when people are most likely to interact with my posts. So I might have 3-4 posts go out at roughly the same time each day to twitter, gplus, facebook. All links route through my site in some fashion for tracking purposes or to entice people into using my site. It appears that Facebook likes to show the posts to a large portion of the people within 10 minutes of the post.

    So there you go, its a "slashdot" effect initiated by my own posts. I saw something similar last time I bought stumbleupon traffic.

    So, bringing this back around to merchants.... My links usually redirect through my site and straight to a merchant site. My question to merchants... What happens to your website performance when 2,000 additional people arrive at your site to window shop within a span of 10 minutes? Are you monitoring for slowdowns so you know that you need a hardware upgrade?
    Merchants, any data you provide to Google Shopping should also be in your affiliate network datafeed. More data means more sales!

  26. Thanks From:

  27. #18
    OPM and Moderator Chuck Hamrick's Avatar
    Join Date
    April 5th, 2005
    Location
    Park City Utah
    Posts
    16,646

  28. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Conspiracy Theory, or are affiliates really losing money?
    By RickPlmr in forum Blocking Tips/Advice/scripts
    Replies: 25
    Last Post: September 19th, 2007, 11:32 AM
  2. Problems with CJ continue. Merchants are losing sales because of it.
    By UncleScooter in forum Commission Junction - CJ
    Replies: 23
    Last Post: October 4th, 2005, 02:24 PM
  3. Fix Reporting, I'm losing lots of money
    By kelly626 in forum Commission Junction - CJ
    Replies: 197
    Last Post: March 3rd, 2005, 03:48 PM
  4. Advice: You could be losing money using invalid links...
    By Ron Bechdolt in forum Commission Junction - CJ
    Replies: 16
    Last Post: February 20th, 2004, 12:37 AM
  5. EPC v.s. EPM - We’re all losing money (your comments, please)
    By tag in forum Commission Junction - CJ
    Replies: 12
    Last Post: October 18th, 2002, 03:54 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •