Results 1 to 7 of 7
October 27th, 2009, 09:41 PM #1Robots.txt for WordPress?
Is there a robots.txt file that comes with WordPress? If so, is it editable from the control panel? If not, is it a good idea to add a robots.txt file?
October 27th, 2009, 10:09 PM #2
I use the robots meta plugin where you can edit the robots.txt in control panel. It also allow you to "disallow" indexing of archives, categories, author archives, date archives, tag archives, any many other indexing issues by just checking off the boxes.
October 27th, 2009, 10:15 PM #3
Wow -- I didn't know there was a robots meta plugin.~Rhia7 -- Remember the 7
October 28th, 2009, 07:04 AM #4
Thanks for that info. I just downloaded the plugin. Wow, there are a lot of checkboxes. Now I need to find out which ones are a good idea to have checked as far as SEO.
October 28th, 2009, 08:01 AM #5
- Join Date
- June 18th, 2006
- The Call is coming from Inside the House!
It is a good idea to have a WP robots.txt file.
I have a standard one that I upload and tweak a bit for each website. I usually exclude the plugins and themes folders, tag, categories, and archives folders.
It is very easy to upload it and then edit with your FTP program on the fly, then verify it is reachable at yoursite.com/robots.txt.
A direct link to the sitemap.xml.....
October 28th, 2009, 10:29 AM #6
I don't know how long this plugin has existed but just found it recently too. Before that I had to manually add in the disallows by hand since the robots were indexing directories that were insignificant. Pretty darn irritating too. Anyways some info here in this post, good advice here: http:// yoast.com/articles/wordpress-seo/#noindexfollow
I have in my plugin:
I pretty much have all checked except for these 3 boxes
- hide verbose explanation of settings
- add noarchive meta tag (not sure why)
- redirect search result pages when referrer is external
Also I keep trying to check the box for not indexing the login pages but it keeps getting unchecked. So I have that manually put into my robots.txt.
October 30th, 2009, 08:43 AM #7
I use a robots.txt file to "block" some directories, but most of them are not related to WP.
There is one rule you should use to prevent your site for duplications:
By Eric Ewe in forum Search Engine OptimizationReplies: 4Last Post: February 5th, 2013, 09:58 PM
By mayfly in forum Search Engine OptimizationReplies: 10Last Post: August 26th, 2009, 05:13 PM
By Rhia7 in forum Midnight Cafe'Replies: 0Last Post: April 18th, 2009, 12:34 AM
By Mr. Sal in forum Voting BoothReplies: 11Last Post: November 12th, 2003, 07:29 PM
By reflections in forum Programming / Datafeeds / ToolsReplies: 5Last Post: December 26th, 2002, 06:22 PM