Results 1 to 9 of 9
  1. #1
    ABW Ambassador
    Join Date
    January 17th, 2005
    Posts
    2,160
    I thought it may be useful to compile a list of active web robots.

    Here goes for the ones that have spidered Trivio over the last few days.

    TrueRobot/1.4
    Source - Unknown

    Slurp
    Source - Inktomi

    Googlebot/2.1
    Source - Google

    Openbot/3.0+
    Source - Openfind.com

    ia_archiver
    Source - Alexa

    almaden
    Source - almaden.ibm.com

    psbot/0.1
    Source - picsearch.com

    Scooter-W3-1.0
    Source - AltaVista

    Marvin
    Source - Infoseek


    Please add / ammend this list for info.


    Henry
    Source - Mirago

    Zeus
    Source - Keeping my eye out for e-mails [img]/infopop/emoticons/icon_wink.gif[/img]

    [ 09-05-2001: Message edited by: Guffy ]

  2. #2
    ABW Veteran Student Heyder's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,482
    BlitzsucheBOT

    DSurf15a 21

    Mozilla/2.0 (compatible; Ask Jeeves)

    Acoon Robot v1.50.001 (www.acoon.de)
    lwp-trivial/1.28

  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Winterpeg, the Mosquito Capital of Canada
    Posts
    2,299
    its just me with my need for info again.
    Do these come through on your stas reports? How does one find what robots may have come through?
    I use wbtrends but never saw any references such as these or anything that may indicate something spidering.
    Does using a firewall such as Zone Alarm not allow these things in? me in my confused mode # 394 for the week ending Sept5 >>>>> :confused:
    :confused:

    [ 09-05-2001: Message edited by: walleye warrior ]

    [ 09-05-2001: Message edited by: walleye warrior ]

  4. #4
    ABW Ambassador
    Join Date
    January 17th, 2005
    Posts
    2,160
    @ walleye warrior

    A firewall just stops unwelcome guests to YOUR computer.

    You'll need to check your log files on the server or use something like axs tracker to get the info.

    The free hit counters and such often don't give this info - gumball tracker used to but I think the site is down now.

  5. #5
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Winterpeg, the Mosquito Capital of Canada
    Posts
    2,299
    log files on server, axs tracking???
    better give me the newbie tag back. I even look at my puter wrong it crashes,wouldnt even have a clue where to start looking for those things
    And u thought I was kidding on the confused part huh

  6. #6
    ABW Veteran Student Heyder's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,482
    home.net might not give you access to those stats.

  7. #7
    ABW Ambassador
    Join Date
    January 17th, 2005
    Posts
    2,160
    @ walleye warrior
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>log files on server, axs tracking??? <HR></BLOCKQUOTE>
    There will come a day when you will need to host on a server with cgi access and all the extras. It's like a natural progression as you gain experience.
    Don't worry - I found it :confused: at first but you'll soon get to grips with it.
    Go here to get an idea of what a decent tracking script can give you.
    Take a look at the demo to view the stats and see for yourself.

    It's just a question of when your ready......

    Take a look at your stats from the bravenet counter.
    Specifically - Visitor Info - Web Browser. My guess is that it will only tell you minimum browser details but you may get lucky and it may tell more info.
    If you just see stuff like:
    MSIE v5.x
    MSIE v4.x
    Netscape v5.x

    Thats not good.

    If you get like:

    Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
    Qanto search bot (www.qanto.com) AKA "Qantobot"

    That's the robots and bravenet are to be commended.


    Had a visit yesterday from:
    Qantobot
    Source - qanto.com
    Never heard of them before - anyone know if it was worth the spider from them?

    [ 09-06-2001: Message edited by: Guffy ]

  8. #8
    Master of Design AlexBet's Avatar
    Join Date
    January 18th, 2005
    Location
    Toronto, Canada
    Posts
    1,364
    @ walleye warrior

    I suppose you use "WebTrends Live" for your tracking and I guess you don't have such a report. I use "WebTrends Log Analyzer" and I do have such statistics. I analyze my raw log files with it. It is not web based but it is a program that you install on your computer. I download my raw log files every day and after I run the report, I archive them for future use.

    Here is my list of spidersfor the last month:

    Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
    Scooter-W3-1.0
    FAST-WebCrawler
    ArchitextSpider
    Lycos_Spider_(modspider)
    Mozilla/4.0 (compatible: FDSE robot)
    WebTrends
    scooter-3.0.vns
    scooter-venus-3.0.vns
    Openfind data gatherer, Openbot
    KIT-Fireball
    Mozilla/4.0_(compatible;_MSIE_5.0;_Windo ws_95)_TrueRobot/1.4 libwww/5.2.8
    teomaagent1 [crawler-admin@teoma.com]
    teomaagent crawler-admin@teoma.com
    Gulliver
    marvin
    DittoSpyder
    Mozilla/4.0 (compatible; MSIE 4.0; Windows NT; Site Server 3.0 Robot) Datadesign
    Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; FREE; MSIECrawler)
    Mozilla/3.0 (compatible; Fluffy the spider; http://www.searchhippo.com/; info@searchhippo.com)

  9. #9
    ABW Veteran Student Heyder's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,482
    Does T-Rex ever do anything? I see him all the time but I haven't seen my site indexed yet.

  10. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  2. Merchant active but not active
    By shanmaree in forum ShareASale - SAS
    Replies: 2
    Last Post: March 21st, 2006, 04:38 AM
  3. CJ, eBay & Active/Non-active Links
    By Emjay in forum Commission Junction - CJ
    Replies: 0
    Last Post: March 5th, 2006, 07:50 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •