Results 1 to 3 of 3
November 18th, 2004, 08:06 PM #1
- Join Date
- January 18th, 2005
A strange thing on many of my sites: Google looks for robots.txt, gets a 404 error and that's it! It can happen like 5 times in a row, but nothing gets indexed. Google doesn't even look at the homepage. No major traffic changes, PR actually went up on one of the sites, but nothing is getting indexed for several days.
Well, I just created a robots.txt file for all my sites, and I'm waiting to see Google's reaction.
November 19th, 2004, 07:45 AM #2
Is this something new?
I was always under the impression that the robots.txt file was optional. You use this if you don't want the page spidered. I had read this in a few places.This World is Not My Home
We're gonna go inside, we're gonna go outside, inside and outside. . . And then we're gonna go go go and we're not gonna stop til we get across that goalline! Quotes from the movie Rudy, 1993
November 19th, 2004, 08:05 AM #3
- Join Date
- January 17th, 2005
Yeah, I hardly ever use robot.txt and google indexes my sites no problems.. optional
By JI7009 in forum Search Engine OptimizationReplies: 7Last Post: November 7th, 2009, 10:34 AM
By mayfly in forum Search Engine OptimizationReplies: 10Last Post: August 26th, 2009, 04:13 PM
By John Powell in forum Search Engine OptimizationReplies: 6Last Post: February 8th, 2006, 12:00 PM
By ~Michelle in forum Search Engine OptimizationReplies: 5Last Post: April 16th, 2005, 05:05 PM
By Mr. Sal in forum Voting BoothReplies: 11Last Post: November 12th, 2003, 06:29 PM