Results 1 to 9 of 9
  1. #1
    Full Member
    Join Date
    January 18th, 2005
    Posts
    303
    Hi,

    I got AV scooter practically living on my server for the last week or so.
    Its getting hungrier by the minute.

    I have a dedicated server with about 8 websites running on it now and AV is taking about 50 to 60% of the load. (lot a rewrites going on)

    I tried to ban it (disalow) but i'm not sure if I miss a opportunity then.
    Up to this day I never seen any traffic coming from AV (from the SERPs I mean)
    OTOH it could bring some traffic after they completed the crawl?

    What would you do?
    Ban and save bandwith and server load, or let it get some because it could bring me some decent traffic?

    ---------------------------------------
    But it beats a real j.o.b.

  2. #2
    Full Member ellen-s4y's Avatar
    Join Date
    January 18th, 2005
    Posts
    489
    I'm disallowing Scooter in all but one directory. It's not allowed anywhere near my rewrite stuff. I did this 2 months ago when it ate about 3000 pages and indexed roughly 1% of them.

    I've seen one, count them, ONE, referrer from AV this month. That's up from zero, last month.

    I hate to ban any legitimate spiders, but this one is a little too hungry, especially when few people actually use AV.

    Inktomi is another hungry bugger. Their bots are only allowed in certain directorys. It's YahooSeeker is a pig the way it eats, it has more access than Scooter, but then, I get referrers from Yahoo, almost as many as Google.

    I'd say 95% of my SE traffic comes from Yahoo and Google.

  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    767
    I never ban any legitimate search engine.
    Bandwith taken now for reading pages could very well mean many extra visitors down the road...

  4. #4
    ABW Adviser Panel Dynamoo's Avatar
    Join Date
    January 18th, 2005
    Location
    Opposite the Slough of Despond
    Posts
    5,465
    I don't ban them because:

    a) I don't know how, and
    b) Don't have direct access to the logs to see how much they're eating up

    1% of my SE traffic comes from AltaVista. 87% from Google+Yahoogle+AOL. 9% from MSN.

    ________
    "All your commission are belong to us." - Slimeware Corporation

  5. #5
    Full Member
    Join Date
    January 18th, 2005
    Posts
    243
    Something to also keep in mind...Altavista is owned by Yahoo (acquired through it's overture purchase).

    You never know what may develop next year. Yahoo may even merge the contents of the Inktomi/altavisa databases into one massively crawled database.

    If it is unduly stressing your server, then by al means, disallow it. However, there may be other work-arounds you cna use to insure it only hits what you want it to hit.

    I've also seen one of my larger (PR7) sites being hit hard by paractically all crawlers this week (scooter, ask jeeves/teoma, googlebot, and the infoamous MSN bot, which has chewed up about 20,000 pages in the last 3 days).

    Personally, with the rapidly changing SE landscape (microsoft may even acquire google), I'm loathe to disallow any legtimate spider, for fear that it's index may get rooled into one of the big 3's databases.

    Erik

    ==============================
    The AllPiercings.com Affiliate Program: 20% LIFETIME Commissions, 5% 2nd Tier Commissions, Affiliate Store Builder, Raw Datafeeds, and Advanced Tools.
    http://www.Allpiercings.com

  6. #6
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    4,423
    I agree on the not banning. I never get msn traffic, this past week I have seen MSN go to the number 2 spot for one of my sites. The site was mentioned on MSNBC, so I can only assume that helped?

    Chet

  7. #7
    Full Member
    Join Date
    January 18th, 2005
    Posts
    303
    Thanks for the replies.

    I banned them on 2 sites, let them eat on the others, load is pretty descent now.
    Just looked at the logs, scooter has gotten over 50K pages the last couple of day's, I think thats enough for now
    If they show me some traffic for that I will remove the "deny".

    One of the sites is hardly visited by anyone, bots or real people, so could make a nice studycase for the effect AV has.

    BTW, I like to monitor with the combination:
    TOP, for monitoring the complete server and
    tail -f on specific domains.

    A other thing I have seen a lot of suddenly is "FunWebProducts", anyone else noticed?

    ---------------------------------------
    But it beats a real j.o.b.

  8. #8
    Full Member
    Join Date
    January 18th, 2005
    Posts
    480
    I banned it too (as well as TEOMA). It chews up my server resources and spits nothing back in return.

    I will unblock it if Yahoo decides to use it.

  9. #9
    ABW Ambassador Andy's Avatar
    Join Date
    January 18th, 2005
    Posts
    4,178
    I'd be careful about banning a known good bot. As mentioned, you never know what might develop down the road. There must be a reason for all the spidering going on, it costs them money as well.

    What I get are a few unidentified robots, and I've considered disallowing them, but I hate to do that since I'm not really sure what they're doing. If I discover they're up to no good, I'll disallow them immediately, but until then I'm going to just keep an eye on them.

    Andy

    _______________
    <font color="red">Call the Exterminators! We've Got PARASITES!</font>

  10. Newsletter Signup

+ Reply to Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •