Results 1 to 5 of 5
  1. #1
    http and a telephoto
    Join Date
    January 18th, 2005
    Location
    NYC
    Posts
    17,708
    The bots are killing my bandwidth this month, without me seeing an increase in surfer traffic to go with it. Across the board my sites have tripled and quadrupled their bandwidth usage and it is from a combination of msn, yahoo, overture (why?), google, gigablast (what the heck is this? Whatever it is it *is* sending me actual traffic), and teoma. One site was consumed by yahoo, the rest have msn bot at the top of the bandwidth charts......

    Any clues? I know msn is getting ready to launch in a few weeks, yahoo and google are battling, but are they doing me any good sucking up all my bandwidth? How long before traffic exceeds bots? Any way to see if all this crawling is worth it? Sales are up a little on a couple of sites, but mostly sales and adsense are status quo.
    Deborah Carney
    TeamLoxly.com BookGoodies.com ABCsPlus.com

  2. #2
    Full Member
    Join Date
    January 18th, 2005
    Posts
    270
    Have you done any analysis of the pages that are being asked for? There is a meta tag that will turn away the robots if you tell them to only visit every seven, ten, (you pick the number) days.

    As for MSN, who knows? They've promised a roll out of a new search engine for sometime this year, which could mean July or December or maybe even next year. We haven't done much more than allow it to spider our sites, and while it does it agressively at times, it's not to the punishment point yet.

    One of the problems that we're running into lately are people with access to site suckers/site copiers who think that it's o.k. to hit up our sites with hundreds or thousands of page requests at a time so that they can "surf offline" to save their precious time. We don't tolerate this well, and we've had a day or two where this type of traffic exceeded the search engines... it's all bandwidth and we're quick to 'deny from' on this type of request... something else to check, especially if your site is getting popular.

    CodeJockey.

  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,419
    I'll go ahead and say this trend will likely only worsen.

    Domains bound affiliates will continue to be fodder food for others looking to take advantage of the content they provide for their own gain.

    Is their some sort of "BotAllow" flag to speficy only bots that are allowed, the need for review and constant adding of unknown bots doing mischievious scans to a restrict list is a royal pain in the rear.

  4. #4
    Newbie
    Join Date
    January 18th, 2005
    Posts
    24
    Yup, you can specify down to the individual bot who gets into what or based upon what they are going to do with it (but you'll probably need to do the more complex things with a robots.txt file.)

    http://www.robotstxt.org/wc/exclusion.html


    Brian

  5. #5
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Texas, USA
    Posts
    579


    Loxly,

    Here is the robots.txt code I use:


    # /robots.txt file for http://www.os2warplinks.com/

    User-agent: *
    Disallow: /tmp/
    Disallow: /logs/
    Disallow: /cgi-bin/
    Disallow: /images/
    Disallow: /stylesheets/

    Some bots still hit the images file pretty hard.

    RadarCat
    http://www.os2warplinks.com

  6. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Replies: 4
    Last Post: June 7th, 2004, 11:56 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •