Results 1 to 7 of 7
  1. #1
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Nunya, Business
    Posts
    23,684
    Robots Txt Plugin, Do You Use One?
    Just wondering if anybody uses a Robots txt. plugin for Wordpress, or do you even need one?

    I have XML Sitemap Generator, All In One SEO plugin, not sure if any of those have anything to do with it.
    And checking the robots txt of all my blogs and they're all:

    User-agent: *
    Disallow:

    Sitemap: http: // mysite .com/ sitemap . xml . gz

    Good enough?

  2. #2
    Full Member gcarson's Avatar
    Join Date
    November 13th, 2009
    Posts
    383
    I also tell google to stay out of wp-admin and what not. The issue I had was, google webmaster was telling me my site was taking 9 seconds on average to download a page. When I checked the pages they were showing as estimates, they were wp-admin pages (not even sure how they get there). So I think it might be important to add that if you care how long google thinks it takes for your site to load.

  3. #3
    ABW Ambassador ladidah's Avatar
    Join Date
    October 15th, 2007
    Location
    MA
    Posts
    1,888
    You can leave the robots.txt blank if you want but one of my site I had the same problem as gcarson so I had some directives in there from way back when to stop indexing certain directories. Sometimes, G would go in and index plugin files, etc.

    If I see certain bots consuming too much bandwidth and they are up to no good I try to disallow but that doesn't mean they listen. Then I resort to the htaccess file.

    I like to go to askapache[dot]com for lot of good info although it's over my head. I use the same plugins as you do but no robots.txt plugin.

  4. #4
    http and a telephoto
    Join Date
    January 18th, 2005
    Location
    NYC
    Posts
    17,708
    http://yoast.com/wordpress/meta-robo...dpress-plugin/

    This one manages your robots meta tags instead of robots text. Found it in the presentation from Jon Henshaw of Raven Tools on Supercharging your WordPress blog. The slideshow of the presentation is here:

    http://www.slideshare.net/affsum/2d-...-press-for-seo
    Deborah Carney
    TeamLoxly.com BookGoodies.com ABCsPlus.com

  5. #5
    Newbie mrthe9th's Avatar
    Join Date
    August 27th, 2010
    Posts
    6
    I use XML sitemap generator and SEO platinum but just now about robot.txt plugin. nice joining this forum
    Last edited by mrthe9th; August 27th, 2010 at 12:15 PM.

  6. #6
    Member niche's Avatar
    Join Date
    September 4th, 2007
    Posts
    197
    I used the robot text file on one of my websites (not wordpress) and I didn't really notice any difference in SERPs or traffic or indexing or anything. So I didn't bother with the plugin on my wordpress blogs
    [URL="http://www.affiliatemarketingintro.com"]Affiliate Marketing Basics[/URL]

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Nunya, Business
    Posts
    23,684
    Alright, thanks. I guess I don't need it. I was checking out the meta robots tags plugin also and it looks like I can do all that with my SEO plugin. And the sites I have already with what I posted about above:

    User-agent: *
    Disallow:

    Sitemap: http: // mysite .com/ sitemap . xml . gz

    seem to do fine. I can't think of a reason I guess that I need one, I just started hearing a lot about them so wanted to check it out.

  8. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  2. Robots.txt
    By Rhia7 in forum Midnight Cafe'
    Replies: 0
    Last Post: April 18th, 2009, 12:34 AM
  3. Google wants nothing but robots.txt!
    By login in forum Search Engine Optimization
    Replies: 2
    Last Post: November 19th, 2004, 09:05 AM
  4. Do you use a robots.txt?
    By Mr. Sal in forum Voting Booth
    Replies: 11
    Last Post: November 12th, 2003, 07:29 PM
  5. robots txt
    By reflections in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: December 26th, 2002, 06:22 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •