Results 1 to 5 of 5
  1. #1
    Web Ho - Design B!tch ~Michelle's Avatar
    Join Date
    January 18th, 2005
    Location
    Michigan
    Posts
    2,040
    I was just checking my stats and there were a couple of crawls I didn't recogonize. Does anyone know who they belong to?

    ExactSeek Crawler/0.1
    libwww-perl/5.48
    Firefly/1.0

    and Websponsors was there too. But I know who they are. *G*

    ~Michelle

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    Never heard of Firefly, but ExactSeek is a meta search, though it does have an add URL page (so I guess it spiders from submissions to that). Libwww-perl could be almost anything. It is a set of perl modules that deals with extracting info from websites, running spiders and so on. Could be from almost anywhere.

    That version of libwww-perl is about 2 years old, though; maybe more. You would expect to see 5.63 and 5.64. Just exclude it in your robots txt if you don't want crawls from it.

    Search Engine Positioning - 1 Design 4 Life

  3. #3
    Full Member
    Join Date
    January 18th, 2005
    Posts
    322
    According to www.4webhelp.com/spiders/ Firefly is a spider from www.fireball.de and libwww-perl/5.48
    is from Alexa. I don’t know how accurate there info is though.
    I’ve had a spider named Jonzilla/666 on my site a lot, anybody know about this fiendish sounding thing?

    Jason
    “That’s the song I’ve been sangin for years, that’s the way the wild wind blows” – Robert Earl Keen

  4. #4
    ABW Ambassador sjangro's Avatar
    Join Date
    January 18th, 2005
    Location
    Boston
    Posts
    1,529
    libwww-perl could be anybody. That's a standard Perl library for programatically acting like a web browser, reading web pages, spidering, etc. Using that you could write a simple spider in 5 minutes.

    Any number of search engines may use that library. It's probably the ones you care less about if they don't even have the wherewithall or courtesy to change the agent string to declare themselves.

    Do a reverse DNS lookup on the IP address. It's just as likely to be some random person on a cable modem as a corporation.

    From my experience Alexa's spider has "ia_archiver" as the agent.

    The ExactSeek Crawler would be from, you guessed it, ExactSeek: http://www.exactseek.com [img]/infopop/emoticons/icon_smile.gif[/img]

    --scott

    --
    Scott Jangro
    sjangro@befree.com

  5. #5
    Web Ho - Design B!tch ~Michelle's Avatar
    Join Date
    January 18th, 2005
    Location
    Michigan
    Posts
    2,040
    Thanks guys!

    ~Michelle

  6. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. ExactSeek
    By westgroup in forum Search Engine Optimization
    Replies: 4
    Last Post: July 19th, 2006, 09:45 AM
  2. Google Crawled Site but Not Listed
    By Rodski in forum Search Engine Optimization
    Replies: 11
    Last Post: January 15th, 2006, 11:21 PM
  3. Purchasing links on sites & in ExactSeek etc
    By westgroup in forum Building Traffic, Newsletters & Advertising
    Replies: 2
    Last Post: June 9th, 2005, 05:55 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •