Results 1 to 8 of 8
  1. #1
    Newbie
    Join Date
    August 28th, 2009
    Posts
    9
    No change to Robots.txt but google says pages blocked
    Hello,

    I've made no changes to my robots.txt file. but suddenly google webmaster tools says my pages are blocked robots.txt, including my homepage. Anyone know of anything that would make this happen?

    Thanks

  2. #2
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    What do you have in your robots.txt?

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  3. #3
    Newbie
    Join Date
    August 28th, 2009
    Posts
    9
    Sitemap: hxxp ://correct site.com/sitemap.xml

    User-agent: *

    Disallow: /wp-content/
    Disallow: /wp-icludes/
    Disallow: /cgi-bin
    Disallow: /cgi-bin/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /archives/
    Disallow: /category/
    Disallow: /category/*/*
    Disallow: /tag/*
    Disallow: /tag/
    Disallow: /wp-*
    Disallow: /login/
    Disallow: */trackback
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*?*
    Disallow: /*?

    User-agent: All
    Allow: /

    User-agent: duggmirror
    Disallow: /

  4. #4
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    Quote Originally Posted by JI7009
    Sitemap: hxxp ://correct site.com/sitemap.xml

    User-agent: *

    Disallow: /wp-content/ You're blocking your content pages here - can't index your posts/content pages?
    Disallow: /wp-icludes/
    Disallow: /cgi-bin
    Disallow: /cgi-bin/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /archives/ You're blocking your archives pages here - google can't index these pages.
    Disallow: /category/ You're blocking category pages here - Googlebot can follow links to find other pages or index.
    Disallow: /category/*/* Same as above
    Disallow: /tag/*
    Disallow: /tag/
    Disallow: /wp-*
    Disallow: /login/
    Disallow: */trackback
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: /*?*
    Disallow: /*?

    User-agent: All
    Allow: /

    User-agent: duggmirror
    Disallow: /

    You're not only blocking the Googlebot - you're blocking all bots from indexing your site. Do you have any traffic at all?

    recommended WP robots.txt:

    User-agent: *
    Disallow: /cgi-bin
    Disallow: /wp-admin
    Disallow: /wp-includes
    Disallow: /wp-content/plugins
    Disallow: /wp-content/cache
    Disallow: /wp-content/themes
    Disallow: /trackback
    Disallow: /feed
    Disallow: /comments
    Disallow: /category/*/*
    Disallow: */trackback
    Disallow: */feed
    Disallow: */comments
    Disallow: /*?*
    Disallow: /*?
    Allow: /wp-content/uploads

    # Google Image
    User-agent: Googlebot-Image
    Disallow:
    Allow: /*

    # Google AdSense
    User-agent: Mediapartners-Google*
    Disallow:
    Allow: /*

    # Internet Archiver Wayback Machine
    User-agent: ia_archiver
    Disallow: /

    # digg mirror
    User-agent: duggmirror
    Disallow: /

    Sitemap: http://www.example.com/sitemap.xml



    Read More About It Here: http://codex.wordpress.org/Search_En..._for_WordPress

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  5. #5
    Newbie
    Join Date
    August 28th, 2009
    Posts
    9
    I just updated my robots.txt with what you gave me.

    Not sure what happened. i havent changed anything in months. But up untill Oct. 18 i was doing alright. then google webmaster tools started telling me my pages were blocked by my robots.txt.

    Anyways, i'll keep my eyes on it for the next few days, and see what happens. Really appreciate the help. Thanks!

  6. #6
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    Quote Originally Posted by JI7009
    I just updated my robots.txt with what you gave me.

    Not sure what happened. i havent changed anything in months. But up untill Oct. 18 i was doing alright. then google webmaster tools started telling me my pages were blocked by my robots.txt.

    Anyways, i'll keep my eyes on it for the next few days, and see what happens. Really appreciate the help. Thanks!
    All I did was do a search on Google for - WordPress robots.txt

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  7. #7
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    I've already banned him permanently for very crude remarks he made about females in one of his post.

    Apparently he thinks this is a porno forum.

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  8. #8
    Analytics Dude Kevin's Avatar
    Join Date
    January 18th, 2005
    Location
    Rochester, NY
    Posts
    5,904
    Our store blog started showing this behavior last week. I went and checked the source, and a meta robots tag had been added blocking everything. Under Privacy Settings in Wordpress, the switch had been flipped to make the blog private,

    Admittedly, even though it's our store, at this point I am using a free template and plug ins. (Well, the store blog... our store is powered by Volusion.)

    I'm assuming that one of the plug ins or the theme itself had done that, as I ceretainly would NEVER make the blog private.

    It seems the meta tag overrides the robots.txt as well.

    Another possibility is that there's a mischievous hacker around, making switches to that setting under the radar. Who knows.

    But something to mindful of if you're a Wordpress user.
    Kevin Webster
    twitter: levelanalytics

    Kayak Fishing
    Web Analytics and Affiliate Marketing

  9. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Blocking Certain Forum Pages with robots.txt
    By Rob_BAM in forum Search Engine Optimization
    Replies: 3
    Last Post: May 29th, 2012, 08:58 AM
  2. Google and robots.txt character limit
    By John Powell in forum Search Engine Optimization
    Replies: 6
    Last Post: February 8th, 2006, 01:00 PM
  3. Google wants nothing but robots.txt!
    By login in forum Search Engine Optimization
    Replies: 2
    Last Post: November 19th, 2004, 09:05 AM
  4. how do i get slurp to crawl my pages, it only crawls my robots.txt
    By shamrock in forum Search Engine Optimization
    Replies: 10
    Last Post: September 10th, 2002, 08:09 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •