Results 1 to 7 of 7
  1. #1
    ABW Ambassador Andy's Avatar
    Join Date
    January 18th, 2005
    Posts
    4,178
    I've seen this in my logs quite a bit, but it never really hit my site very hard, pulling an average of just several pages a day. Last night, it hit me really hard, pulling every page file allowed by robots.txt in the site.

    Who or what is this? Anyone know? Goggled it, but just found other sites' traffic logs making reference to it.

    Andy

    AFFILIATE MARKETING STANDARD: The site upon which the initial action to buy occurs is the site the commission is paid to. Period.

  2. #2
    ABW Adviser Panel Dynamoo's Avatar
    Join Date
    January 18th, 2005
    Location
    Opposite the Slough of Despond
    Posts
    5,465
    Erm does it not have any kind of name?

    ________
    All your commission are belong to us.

  3. #3

  4. #4
    ABW Ambassador Andy's Avatar
    Join Date
    January 18th, 2005
    Posts
    4,178
    AWStats identifies it as "Unknown Robot (identified by 'crawl')" without the quotations. That's exactly what shows up in the logs, and that's all that shows up. Which is why I'm a bit concerned. AWStats doesn't identify it as a site grabber, but whatever it is, it doesn't play by the rules set out on the robots.txt either.

    Andy

    AFFILIATE MARKETING STANDARD: The site upon which the initial action to buy occurs is the site the commission is paid to. Period.

  5. #5
    ABW Adviser Panel Dynamoo's Avatar
    Join Date
    January 18th, 2005
    Location
    Opposite the Slough of Despond
    Posts
    5,465
    How about a source IP address?

    ________
    All your commission are belong to us.

  6. #6
    ABW Ambassador Andy's Avatar
    Join Date
    January 18th, 2005
    Posts
    4,178
    216.167.*.*

    It has been back to my site as of the day before yesterday, but only pulled a couple of pages. Very strange.

    Andy

    AFFILIATE MARKETING STANDARD: The site upon which the initial action to buy occurs is the site the commission is paid to. Period.

  7. #7
    Full Member
    Join Date
    January 18th, 2005
    Posts
    246
    A lot of ordinary webmasters are running their own crawlers, sampling the web. I've had this/these visiting a few of my sites too lately, don't know if it's the same one you have seen. Got an exact ip?

    -- Less is more --

  8. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Google Ads Not Identified...
    By ADesertRose in forum Midnight Cafe'
    Replies: 2
    Last Post: March 21st, 2006, 11:59 AM
  2. Using Specifically Identified Categories
    By jmmktent in forum WebMerge (Fourthworld.com)
    Replies: 2
    Last Post: November 16th, 2005, 12:31 PM
  3. Google Robot Searching for robot.txt
    By ahmar in forum Search Engine Optimization
    Replies: 4
    Last Post: December 26th, 2004, 01:26 PM
  4. unknown referrer
    By Celicaphile in forum Midnight Cafe'
    Replies: 2
    Last Post: January 9th, 2003, 09:53 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •