Results 1 to 8 of 8
  1. #1
    Action Jackson - King of the World
    Join Date
    January 18th, 2005
    Posts
    2,201
    Is anyone else getting hounded by the Allthe web spider. This is taking up more bandwidth than Google on my main mall site!

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,356
    So ban it.

    You could rank #1 for "free sex videos" or "Britney Spears Nude" and you'd be lucky to get 6 hits a month from them.

  3. #3
    2005 Linkshare Golden Link Award Winner  ecomcity's Avatar
    Join Date
    January 18th, 2005
    Location
    St Clair Shores MI.
    Posts
    17,328
    Britney could gladly whack me 6 times a month for making free sex videos with her. See what that speed reading class did for me?
    Webmaster's... Mike and Charlie

    "What have you done today to put real value into a referral click...from a shoppers viewpoint!"

  4. #4
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    Um, what AlltheWeb spider ? Can you post the User-Agent name for this as AlltheWeb is now part of Yahoo and Yahoo's spider is Slurp.

  5. #5
    Action Jackson - King of the World
    Join Date
    January 18th, 2005
    Posts
    2,201
    Agent: Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler

    I don't see any mention of Slurp and I'm banned for this site by Yahoo

  6. #6
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    There's a thread about this crawler at WmW :
    http://www.webmasterworld.com/forum35/2107.htm

    Doesn't tell you much, except that if it's eating bandwidth, you're probably safe to exclude it with robots.txt .

  7. #7
    Action Jackson - King of the World
    Join Date
    January 18th, 2005
    Posts
    2,201
    Thnks Mark I ahve it disallowed now

    I can't imagine it would make much difference to my traffic:P

  8. #8
    Member
    Join Date
    January 18th, 2005
    Posts
    65
    are you saying that alltheweb does nothing for you?
    what volume of traffic are you getting from it?

    I'm seeing a little but nothing to write home about.

    Saw this site that explains the relationship of all the web with the entire internet

    according to it it's totally worthless!!
    the link is here
    http://www.bruceclay.com/searchengin...nshipchart.htm

    maybe its not worth worrying about

  9. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. AllTheWeb-Check Them Out New Stuff
    By Trust in forum Search Engine Optimization
    Replies: 1
    Last Post: March 5th, 2003, 05:36 PM
  2. AllTheWeb Halloween Skin
    By Trust in forum Search Engine Optimization
    Replies: 0
    Last Post: October 30th, 2002, 09:33 PM
  3. Fast/AllTheWeb Update
    By johnniebry in forum Search Engine Optimization
    Replies: 3
    Last Post: September 16th, 2002, 11:10 PM
  4. Fast Alltheweb
    By josephmonuit in forum Search Engine Optimization
    Replies: 3
    Last Post: September 14th, 2002, 06:01 AM
  5. What does Alltheweb power?
    By wly in forum Search Engine Optimization
    Replies: 3
    Last Post: November 22nd, 2001, 02:56 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •