Results 1 to 5 of 5
  1. #1
    Merchant & ABW Ambassador
    Join Date
    May 31st, 2006
    Location
    Houston TX
    Posts
    4,731
    Exclamation Robots.txt file suppressing Wordpress /tags/ from showing up in search engines
    Hi All

    This is bugging me and although not critical, it feels like a small torn that i need to get it out.

    I noticed that about 2 dozen pages are not being crawled due the robots file. It is the "/tags/" that is omitted from search engines, along with some feeds.

    I confirmed that by doing a google search and also google/Webmaster

    The listing does show up if i force it but I get this on Google
    TITLE PAGE
    A description for this result is not available because of this site's robots.txt learn more


    I don't get it. Here is my robots.txt

    User-agent: *

    Disallow: /wp-admin/

    Sitemap: http://www.my domain.com/sitemap.xml

  2. #2
    ABW Ambassador Bob Lawrence's Avatar
    Join Date
    July 2nd, 2007
    Posts
    1,090
    User-agent: *
    Allow: /

    Disallow: /wp-admin/

    Try that Eric
    Last edited by Bob Lawrence; February 3rd, 2013 at 01:28 PM. Reason: copies his disallow down.
    Where's the Great Life of Affiliate Marketing Hiding?

  3. Thanks From:

  4. #3
    Merchant & ABW Ambassador
    Join Date
    May 31st, 2006
    Location
    Houston TX
    Posts
    4,731
    Bob-

    Thanks. It is crawling the main pages and the index page.
    Not sure if your suggestion will make a different.

    I was looking at
    http://www.overstock.com/robots.txt

    and it looks like it is the same as mine

  5. #4
    Moderator
    Join Date
    April 6th, 2006
    Posts
    2,689
    Is it possible the plugin that generates your tag pages is IN the wp-admin folder..?

    Just a thought.

  6. #5
    Merchant & ABW Ambassador
    Join Date
    May 31st, 2006
    Location
    Houston TX
    Posts
    4,731
    Quote Originally Posted by teezone View Post
    Is it possible the plugin that generates your tag pages is IN the wp-admin folder..?

    Just a thought.
    Good thought. I don't think so cos I run the same plugin on a different site and it is crawled. Thanks for the 2 cents.

    I might try a diff plugin or strip it all together.

  7. Newsletter Signup

Closed Thread

Similar Threads

  1. editing Robots.txt file
    By KODea in forum Search Engine Optimization
    Replies: 5
    Last Post: August 28th, 2006, 03:02 PM
  2. robots.txt file
    By UncleScooter in forum Midnight Cafe'
    Replies: 2
    Last Post: October 15th, 2004, 06:39 PM
  3. Do I need a "robots.txt" file?
    By unclejesse in forum Search Engine Optimization
    Replies: 20
    Last Post: June 12th, 2004, 11:02 AM
  4. robots.txt file question
    By Doc Sawyer in forum Search Engine Optimization
    Replies: 4
    Last Post: December 7th, 2002, 07:10 PM
  5. robots.txt file????? HELP
    By john dundas in forum Search Engine Optimization
    Replies: 4
    Last Post: August 18th, 2002, 05:36 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •