Results 1 to 5 of 5
  1. #1
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,403
    robot of www.buscaplus.com
    robot of unknown.Level3.net
    robot of buildrack92.sv.av.com
    robot of drone10.sv.av.com
    spider from cr005r01-3.sac2.fastsearch.net
    spider.lon4.fastsearch.net
    c-24-98-213-204.atl.client2.attbi.com
    robot from bigip1a-snat.sv.av.com
    ZyBorg/1.0 Dead Link Checker
    usma-cce.usma.army.mil
    Inktomi slurp
    crawl11-public.alexa.com
    fixcr002.sac2.fastsearch.net

    and so on.

    I just stopped researching and still 90% of logfile rests. A lot of bandwidth I have to pay for.

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Nunya, Business
    Posts
    23,684
    Spider from a Spanish search engine. They were talking about this at WebmasterWorld awhile back:
    "Seem to be a whole network of caching proxy servers gone mad - the range belongs to Spanish ISP / Telco "Telefonica". "the whole network of proxy servers have gone mad requesting the page every few seconds. "

  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    Carneol, aside from the mad spanish spider and another dodgy looking one, the rest are major search engines - Inktomi, Fast/AllTheWeb, AltaVista, WiseNut. Oh, and Alexa. Not really sure what you are complaining about here. Sure you pay the bandwidth but in return the nice search engines send you people who buy stuff. Seems fair enough to me.

    Search Engine Positioning - 1 Design 4 Life

  4. #4
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,403
    Marymark,
    I would not complain if these spiders will com 5 times or 10 times, but they come nearly each 5 minutes, and there is another anoying thing: my private site stats show e.g. 15 clicks for a site, in the same timespan cj.com shows 65 clicks for the merchant on that site. Of course someone could have clicked more than 1 times on a link, but I experienced that since weeks. I suspect a spider to click my links.

    carneol

  5. #5
    ABW Adviser Panel Dynamoo's Avatar
    Join Date
    January 18th, 2005
    Location
    Opposite the Slough of Despond
    Posts
    5,465
    I just checked my busiest site and 2.1% of the traffic comes from crawlers.

    Just with some playing about with numbers I reckon that each page attracts roughly 1Mb of spider traffic a month.. that seems to be vaguely consistent across the sites I've looked at.

    So.. a 1000 page site could have 1Gb of transfer eaten up just by spiders alone.

    ________
    "All your commission are belong to us." - Slimeware Corporation

  6. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Found My new Home
    By InfoScout in forum Introduce Yourself
    Replies: 4
    Last Post: May 24th, 2010, 07:30 PM
  2. Not Found
    By Witzer in forum Rakuten LinkShare - LS
    Replies: 0
    Last Post: October 19th, 2009, 08:30 AM
  3. How are websites found?
    By Merchant Consultant Team in forum Midnight Cafe'
    Replies: 9
    Last Post: September 24th, 2007, 07:19 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •