Results 1 to 5 of 5
  1. #1
    Full Member ahmar's Avatar
    Join Date
    January 18th, 2005
    Posts
    481
    I have noticed that in my stats google crawls through my site looking for robot.txt. I dont have any robot.txt file on the server. So ultimately that results in 404 error. I always thought that robot.txt file should only be used if one wants to scare the robots away.

    Does anybody know what google is looking for and why it only looks for robot.txt? Any suggestions please?
    <DT>[size=1][color=navy]"The best measure of a man's honesty isn't his income tax return.[/color][/size]<DT>[size=1][color=navy]It's the zero adjust on his bathroom scale." Arthur C. Clark[/color][/size]</DT>

  2. #2

  3. #3
    Full Member ahmar's Avatar
    Join Date
    January 18th, 2005
    Posts
    481
    Thanks Mikey for the link . Found the answer:

    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR> 404 Redirects that lead to another page:
    Quite common is the website without a robots.txt that seamlessly redirects the request to another page. Often that redirect is done without generating a server status error or redirect status message. It is then up to the spider to figure out if it is looking at a robots.txt or an html file. Although it should not cause you any problems, can you afford to risk it? To fix it without reconfiguring your server, place a blank robots.txt file in your root.
    <HR></BLOCKQUOTE>
    <DT>[size=1][color=navy]"The best measure of a man's honesty isn't his income tax return.[/color][/size]<DT>[size=1][color=navy]It's the zero adjust on his bathroom scale." Arthur C. Clark[/color][/size]</DT>

  4. #4
    MasterMike HardwareGeek's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,810
    I have mines with the following

    User-Agent: Scooter
    Disallow:

  5. #5
    Web Ho - Design B!tch ~Michelle's Avatar
    Join Date
    January 18th, 2005
    Location
    Michigan
    Posts
    2,040
    I have the following in mine.

    User-agent: *
    Disallow: /cgi-bin/

    User-agent: Googlebot-Image
    Disallow: /

    User-agent: BDFetch
    Disallow: /

    User-agent: NPBot
    Disallow: /

    User-agent: Zao
    Disallow: /

    User-agent: Zao/0.2
    Disallow: /

    User-agent: TurnitinBot
    Disallow: /

    User-agent: psbot/0.1
    Disallow: /

    SetEnvIfNoCase User-Agent "NPBot" evil=1
    Deny from env=evil
    ~Michelle
    "All I ask is a chance to prove that money can't make me happy."
    "Work to become, not to acquire." -- Confucius

  6. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Robot.txt example...
    By eggerda in forum Search Engine Optimization
    Replies: 7
    Last Post: September 18th, 2003, 11:56 AM
  2. Robot.txt versus Amazon.PL
    By beggers in forum Cusimano.com Scripts
    Replies: 12
    Last Post: March 18th, 2003, 04:16 PM
  3. A Lesson About Robot.txt
    By seaslug44 in forum Search Engine Optimization
    Replies: 1
    Last Post: February 22nd, 2002, 07:49 PM
  4. Robot txt
    By mousejockey in forum Programming / Datafeeds / Tools
    Replies: 12
    Last Post: January 14th, 2002, 05:05 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •