Results 1 to 5 of 5
  1. #1
    Full Member
    Join Date
    January 18th, 2005
    Posts
    236
    Hi
    On my stats programme it says that the page request with most errors is the robots.txt page.

    Can anyone shed any light on this at all. I take it that its something to do with search search engines crawling the sites.

    Look forward to the replies and learning something new.

    regards John

  2. #2
    Full Member
    Join Date
    January 18th, 2005
    Posts
    236
    Hi
    Typical, just did a simple search on google and found the following site:
    http://www.robotstxt.org/

    Ah well at least I now know what its about and what to do to stop the error message.

    regards John

  3. #3
    Full Member
    Join Date
    January 18th, 2005
    Posts
    315
    >>Ah well at least I now know what its about and what to do to stop the error message.<<

    Couldn't find that info in my very cursory glance at the site (which is a pretty niftly little resource, actually). But it's probably worth noting that I heard somewhere you should never put up a blank robots.txt file since that will stop certain robots from slurping your site - Lycos, for example. So those who want to stop the robots errors need to have something on the robots.txt page that they make.

  4. #4
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    I just let those errors pile up...I can see at a glance how well-spidered a site is that way, and I don't have to fish out the calls for robots.txt from my log files when they are neatly listed in the error log!

  5. #5
    Full Member
    Join Date
    January 18th, 2005
    Posts
    315
    I actually do the same thing - robots.txt is the only error that's welcome in my log file. I like seeing them pile up! <IMG src=http://www.abestweb.com/ubb/icons/icon38.gif>

  6. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Looking for a Good Robots.txt file
    By Doug247 in forum Midnight Cafe'
    Replies: 8
    Last Post: July 24th, 2007, 06:18 PM
  2. editing Robots.txt file
    By KODea in forum Search Engine Optimization
    Replies: 5
    Last Post: August 28th, 2006, 03:02 PM
  3. robots.txt file
    By UncleScooter in forum Midnight Cafe'
    Replies: 2
    Last Post: October 15th, 2004, 06:39 PM
  4. Do I need a "robots.txt" file?
    By unclejesse in forum Search Engine Optimization
    Replies: 20
    Last Post: June 12th, 2004, 11:02 AM
  5. robots.txt file question
    By Doc Sawyer in forum Search Engine Optimization
    Replies: 4
    Last Post: December 7th, 2002, 07:10 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •