Results 1 to 3 of 3
  1. #1
    Full Member
    Join Date
    January 18th, 2005
    Location
    K&Y
    Posts
    448
    A strange thing on many of my sites: Google looks for robots.txt, gets a 404 error and that's it! It can happen like 5 times in a row, but nothing gets indexed. Google doesn't even look at the homepage. No major traffic changes, PR actually went up on one of the sites, but nothing is getting indexed for several days.
    Well, I just created a robots.txt file for all my sites, and I'm waiting to see Google's reaction.

  2. #2
    More Cheesier Than Ever Cheesehead's Avatar
    Join Date
    January 18th, 2005
    Location
    Land of The NFL Champs!
    Posts
    2,942
    Is this something new?

    I was always under the impression that the robots.txt file was optional. You use this if you don't want the page spidered. I had read this in a few places.
    This World is Not My Home
    We're gonna go inside, we're gonna go outside, inside and outside. . . And then we're gonna go go go and we're not gonna stop til we get across that goalline! Quotes from the movie Rudy, 1993

  3. #3
    Full Member
    Join Date
    January 18th, 2005
    Posts
    417
    Yeah, I hardly ever use robot.txt and google indexes my sites no problems.. optional

  4. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. No change to Robots.txt but google says pages blocked
    By JI7009 in forum Search Engine Optimization
    Replies: 7
    Last Post: November 7th, 2009, 11:34 AM
  2. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  3. Google and robots.txt character limit
    By John Powell in forum Search Engine Optimization
    Replies: 6
    Last Post: February 8th, 2006, 01:00 PM
  4. Google, Ask Jeeves ignoring robots.txt?
    By ~Michelle in forum Search Engine Optimization
    Replies: 5
    Last Post: April 16th, 2005, 06:05 PM
  5. Do you use a robots.txt?
    By Mr. Sal in forum Voting Booth
    Replies: 11
    Last Post: November 12th, 2003, 07:29 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •