Results 1 to 4 of 4
June 6th, 2010, 11:08 AM #1robots.txt not working ??
superCool is seeing pages in google that he thinks should be protected by robots.txt
the robots file is in the root directory and looks like this
but this url is showing up in google (using search for site:example.com)
that url is affected by a rewrite rule in htaccess. could that be causing this? wouldn't google apply the robots.txt rules to the final url (as shown above)?
did superCool do something wrong? gooble webmaster tools shows the correct robots info
June 6th, 2010, 11:32 AM #2
maybe /ddd/*.html or /ddd/*.* might fix it?
Any way to add noindex to the meta?
Make sure it is not showing in your sitemaps and you can remove the URI from goofle's index in GWT.
June 6th, 2010, 03:29 PM #3
- Join Date
- June 18th, 2006
- The Call is coming from Inside the House!
You can disallow pages or directories in Webmaster Tools. They have to be currently throwing a 404 or be blocked by robots.txt.
I have seen the same problem. It seems like Google is not really following robots.txt for some sites. It indexes folders it shouldnt.
I have recently started using multiple versions of folders I dont want indexed.
I dont really have an answer yet if this is helpful.
June 6th, 2010, 04:18 PM #4
superCool will give those suggestions a shot. it's a bit of a befuddling frustration when you follow every example you find on the net but it still doesn't work. we'll see what happens. superCool's new site is just now being indexed (at a snail's pace) and half the pages are the ones that shouldn't be included . oh well, that's the way it goes sometimes
By Trust in forum Blogging, Mobile and Social MediaReplies: 6Last Post: August 31st, 2010, 05:46 PM
By mayfly in forum Search Engine OptimizationReplies: 10Last Post: August 26th, 2009, 04:13 PM
By Rhia7 in forum Midnight Cafe'Replies: 0Last Post: April 17th, 2009, 11:34 PM
By Mr. Sal in forum Voting BoothReplies: 11Last Post: November 12th, 2003, 06:29 PM
By reflections in forum Programming / Datafeeds / ToolsReplies: 5Last Post: December 26th, 2002, 05:22 PM