Results 1 to 3 of 3
April 5th, 2005, 08:08 AM #1Yahoo Ignores to robots.txt
Part of my robots.txt file reads
and I continue to get crawled. They take about 5GB a month. Normally this would be a good thing but do not index the site.
Anyone know what I can do.
April 5th, 2005, 09:21 AM #2
My only guess is to check your server logs to make sure that the robots.txt is being read sucessfully. Maybe it's unable to read it so it's not using it?
consultant by day, affiliate by night
April 5th, 2005, 10:22 AM #3
Also make sure you upload the file as Unix (happened to me once).
Most ftp program have a mode where you can choose Unix format.
Some html editors don't have a choice / mode and can default to the wrong thing.
Use this tool to convert your file to Unix if in doubt:
Then make sure you use an ftp tool set to Unix when you upload it to your root directory.
http://www.searchengineworld.com/cgi-bin/robotcheck.cgi (Start here!!)
http://www.searchengineworld.com/rob...s_tutorial.htm (Good resource!)
By Trust in forum Blogging, Mobile and Social MediaReplies: 6Last Post: August 31st, 2010, 06:46 PM
By mayfly in forum Search Engine OptimizationReplies: 10Last Post: August 26th, 2009, 05:13 PM
By Rhia7 in forum Midnight Cafe'Replies: 0Last Post: April 18th, 2009, 12:34 AM
By Mr. Sal in forum Voting BoothReplies: 11Last Post: November 12th, 2003, 07:29 PM
By reflections in forum Programming / Datafeeds / ToolsReplies: 5Last Post: December 26th, 2002, 06:22 PM