Results 1 to 13 of 13
  1. #1
    All Around Web Guy Cursal's Avatar
    Join Date
    January 18th, 2005
    Posts
    829
    How Many Robots In a Log?
    Just as a general question:

    How many spiders and robots visit your "best" website in a month?
    (Best could be sales or traffic please specify)

    My best traffic site gets around 13 different ones with

    Googlebot, Inktomi Slurp, MSNBot, being the top visitors

    Thanks for your input

  2. #2
    MasterMike HardwareGeek's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,810
    On any given day i have over 60 GOogle bots, 30, yahoo bots, 10 ask.com and 15 MSN search

    for the entire month i get atleast 10k of bots in total.

  3. #3
    Member Art's Avatar
    Join Date
    August 21st, 2005
    Location
    Europe
    Posts
    155
    This isn't our "best" website but ... (April '06)
    Googlebot 137450 hits
    MSNBot 61990 hits
    Inktomi Slurp 19420 hits
    Unknown robot (identified by 'spider') 25330 hits
    AskJeeves 13600 hits
    Alexa (IA Archiver) 770 hits
    WISENutbot 570 hits
    Unknown robot (identified by 'crawl') 470 hits
    Unknown robot (identified by 'robot') 390 hits
    Voyager 70 hits
    [URL=http://www.affiliatesnetwork.info/links/][SIZE=1][COLOR=DarkOrange][B]Affiliatesnetwork[/B][/COLOR][/SIZE][/URL] | [URL=http://findhotel.info/hotels/][COLOR=DarkOliveGreen][SIZE=1][B]Book a hotel[/B][/SIZE][/COLOR][/URL] | [URL=http://affiliate-marketing.affiliatesnetwork.info/][SIZE=1][B][COLOR=DarkOrange]Affiliate Marketing Blog[/COLOR][/B][/SIZE][/URL]

  4. #4
    All Around Web Guy Cursal's Avatar
    Join Date
    January 18th, 2005
    Posts
    829
    Quote Originally Posted by HardwareGeek
    On any given day i have over 60 GOogle bots, 30, yahoo bots, 10 ask.com and 15 MSN search

    for the entire month i get atleast 10k of bots in total.
    How can you tell them apart?

    Do they all have unique names or ???

    I'm not talking hits on a site or hits on robot.txt files

    Just the number of UNIQUE Robots or Spiders.

    I have xx,xxx for hits too but only 13 different named spiders/robots/crawlers etc.

  5. #5
    All Around Web Guy Cursal's Avatar
    Join Date
    January 18th, 2005
    Posts
    829
    Quote Originally Posted by Art
    This isn't our "best" website but ... (April '06)
    Googlebot 137450 hits
    MSNBot 61990 hits
    Inktomi Slurp 19420 hits
    Unknown robot (identified by 'spider') 25330 hits
    AskJeeves 13600 hits
    Alexa (IA Archiver) 770 hits
    WISENutbot 570 hits
    Unknown robot (identified by 'crawl') 470 hits
    Unknown robot (identified by 'robot') 390 hits
    Voyager 70 hits
    So this is 10

  6. #6
    notary sojac Herb ԿԬ's Avatar
    Join Date
    January 18th, 2005
    Location
    Central/Western NY State
    Posts
    7,741
    I think there are many more out there, just not labeled as robots. when you see an unknown visitor hitting every page you have in a row, it's probably a 'bot.

    one hopes it is there doing innocent work . . .

  7. #7
    Member
    Join Date
    April 11th, 2006
    Location
    Michigan
    Posts
    70
    If you really want to know the name of all robots. Please click on this links to see the complete database of robots

    http://www.robotstxt.org/wc/active/html/index.html
    Burhan Tanweer
    Marketing Manager
    sales/at/awebhosts.com
    [URL=http://www.awebhosts.com]www.awebhosts.com [/URL]

  8. #8
    notary sojac Herb ԿԬ's Avatar
    Join Date
    January 18th, 2005
    Location
    Central/Western NY State
    Posts
    7,741
    however, it may be true also that there are people running site-skimming software on their servers that have never been listed. I remember articles about this years ago. Seems to me that listing somewhere was nice, but not necessary.

  9. #9
    Roll Tide mobilebadboy's Avatar
    Join Date
    January 18th, 2005
    Location
    Mobile, Alabama
    Posts
    1,220
    robotstxt.org's list is quite incomplete and outdated.

    Shawn Kerr (.com) | Disney World | SEC Football

  10. #10
    Member Art's Avatar
    Join Date
    August 21st, 2005
    Location
    Europe
    Posts
    155
    Quote Originally Posted by Cursal
    How can you tell them apart?

    Do they all have unique names or ???
    If you check raw access logs you should see their names.
    I suppose they all have unique names.

    Quote Originally Posted by Cursal
    I'm not talking hits on a site or hits on robot.txt files

    Just the number of UNIQUE Robots or Spiders.

    I have xx,xxx for hits too but only 13 different named spiders/robots/crawlers etc.
    Talking about the names only, these are the most active bots/spiders
    I see in the server log files:
    Googlebot, msnbot, Gigabot, Exabot, Yahoo! Slurp, Mediapartners-Google,
    Ask Jeeves/Teoma, Inktomi Slurp, Alexa

    From time to time I also notice visits from these:
    WISENutbot, Voyager, Baiduspider, snap.com beta crawler, Nomad,
    LinkWalker, findlinks

    16 different bots/spiders
    [URL=http://www.affiliatesnetwork.info/links/][SIZE=1][COLOR=DarkOrange][B]Affiliatesnetwork[/B][/COLOR][/SIZE][/URL] | [URL=http://findhotel.info/hotels/][COLOR=DarkOliveGreen][SIZE=1][B]Book a hotel[/B][/SIZE][/COLOR][/URL] | [URL=http://affiliate-marketing.affiliatesnetwork.info/][SIZE=1][B][COLOR=DarkOrange]Affiliate Marketing Blog[/COLOR][/B][/SIZE][/URL]

  11. #11
    Plazan Merchant Neil's Avatar
    Join Date
    February 25th, 2005
    Location
    cyprus
    Posts
    1,764
    we average around 25 diffirent ones each month.
    but many of them on a daily basis.
    Find us at shareasale.com 12% commission
    Shareasale Merchant 7191
    PLAZAN SKIN CARE As seen on TV . Used by Jennifer Lopez

  12. #12
    MasterMike HardwareGeek's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,810
    Quote Originally Posted by Cursal
    How can you tell them apart?

    Do they all have unique names or ???
    You look at your logs and find something with a hostname of Yahoo Slurp, MSN bot or Google Bot. Then you find how many Unique IPs have visited your site.

    Right now Yahoo is hitting me hard.

  13. #13
    ABW Ambassador buy_online's Avatar
    Join Date
    January 18th, 2005
    Location
    Richmond, VA
    Posts
    3,234
    Quote Originally Posted by Neil
    we average around 25 diffirent ones each month.
    but many of them on a daily basis.
    Yup, same here.

    Although, I have been slammed for bandwidth by Inkt*mi this month like never before (many gigs). Made some changes to that site, so maybe that's it. The IP is from South Africa (weird).

    Anyway, the answer is - it depends. Content changes, site scope (size and document make-up), popularity and longevity (heckuva formula).

    Fred

  14. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  2. How To Block Robots...?? Help..
    By AddHandler in forum Programming / Datafeeds / Tools
    Replies: 11
    Last Post: August 12th, 2007, 05:51 PM
  3. Happy Log vs. Web Log Expert - Looking For Comments
    By sweat in forum Search Engine Optimization
    Replies: 6
    Last Post: July 27th, 2003, 10:01 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •