Results 1 to 7 of 7
  1. #1
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Robots.txt for WordPress?
    Is there a robots.txt file that comes with WordPress? If so, is it editable from the control panel? If not, is it a good idea to add a robots.txt file?

  2. #2
    ABW Ambassador ladidah's Avatar
    Join Date
    October 15th, 2007
    Location
    MA
    Posts
    1,888
    I use the robots meta plugin where you can edit the robots.txt in control panel. It also allow you to "disallow" indexing of archives, categories, author archives, date archives, tag archives, any many other indexing issues by just checking off the boxes.

  3. #3
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Wow -- I didn't know there was a robots meta plugin.
    ~Rhia7 -- Remember the 7
    Twitter me

  4. #4
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    ladidah,

    Thanks for that info. I just downloaded the plugin. Wow, there are a lot of checkboxes. Now I need to find out which ones are a good idea to have checked as far as SEO.

  5. #5
    SEO: A Specialty - Web Design: Slow or outsourced andbeyond's Avatar
    Join Date
    June 18th, 2006
    Location
    The Call is coming from Inside the House!
    Posts
    1,332
    It is a good idea to have a WP robots.txt file.

    I have a standard one that I upload and tweak a bit for each website. I usually exclude the plugins and themes folders, tag, categories, and archives folders.

    It is very easy to upload it and then edit with your FTP program on the fly, then verify it is reachable at yoursite.com/robots.txt.

    A direct link to the sitemap.xml.....

  6. #6
    ABW Ambassador ladidah's Avatar
    Join Date
    October 15th, 2007
    Location
    MA
    Posts
    1,888
    I don't know how long this plugin has existed but just found it recently too. Before that I had to manually add in the disallows by hand since the robots were indexing directories that were insignificant. Pretty darn irritating too. Anyways some info here in this post, good advice here: http:// yoast.com/articles/wordpress-seo/#noindexfollow

    I have in my plugin:

    I pretty much have all checked except for these 3 boxes

    - hide verbose explanation of settings
    - add noarchive meta tag (not sure why)
    - redirect search result pages when referrer is external

    Also I keep trying to check the box for not indexing the login pages but it keeps getting unchecked. So I have that manually put into my robots.txt.

  7. #7
    Newbie finalwebsites's Avatar
    Join Date
    October 30th, 2009
    Posts
    9
    I use a robots.txt file to "block" some directories, but most of them are not related to WP.
    There is one rule you should use to prevent your site for duplications:

    User-Agent: Googlebot
    Disallow: /index.php

  8. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Replies: 4
    Last Post: February 5th, 2013, 09:58 PM
  2. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  3. Robots.txt
    By Rhia7 in forum Midnight Cafe'
    Replies: 0
    Last Post: April 18th, 2009, 12:34 AM
  4. Do you use a robots.txt?
    By Mr. Sal in forum Voting Booth
    Replies: 11
    Last Post: November 12th, 2003, 07:29 PM
  5. robots txt
    By reflections in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: December 26th, 2002, 06:22 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •