Results 1 to 4 of 4
  1. #1
    Newbie
    Join Date
    March 11th, 2012
    Location
    New Jersey
    Posts
    22
    Blocking Certain Forum Pages with robots.txt
    Recently, I added forum to my website and I've noticed that there are a lot of pages not relevant to the content of my site (FAQ, help, etc). I use URL rewriting for all the content pages on the forum so I can recognize pages that do not attribute content by ? in the URL.

    I figured that inclusion of all those pages could negatively affect my SEO efforts and Google was showing forum as most significant keyword on my site.

    Couple of weeks ago I decided to block those pages in robots.txt by adding this line for all user agents:
    Disallow: /forum/*?

    Ever since, when I log into my Webmaster Tools in Google, I get warning stating that there are severe health issues with important pages being blocked by robots.txt. The pages I get forwarded to are the ones I don't want showing in search results. Up to now I've been just ignoring those warnings thinking that they would go away.

    What should I do about this?
    Should I just let it be the way it is?
    Should I request those pages to be removed?
    Should I remove that line from robots and allow Google to spider everything?

    Thanks,
    Robert

  2. #2
    ...and a Pirate's heart. Convergence's Avatar
    Join Date
    June 24th, 2005
    Posts
    6,918
    Quote Originally Posted by Rob_BAM View Post
    Couple of weeks ago I decided to block those pages in robots.txt by adding this line for all user agents:
    Disallow: /forum/*?

    Ever since, when I log into my Webmaster Tools in Google, I get warning stating that there are severe health issues with important pages being blocked by robots.txt. The pages I get forwarded to are the ones I don't want showing in search results. Up to now I've been just ignoring those warnings thinking that they would go away.

    What should I do about this?
    Should I just let it be the way it is?
    Should I request those pages to be removed?
    Should I remove that line from robots and allow Google to spider everything?

    Thanks,
    Robert
    Yeah, we upset the Google by blocking our FAQ directory on hour web hosting site after the Google had already had it indexed for several years. Has had no ill effects other than the annoying "critical health issue" warnings.

    What happens is the Google continues to try and access the directory and is throwing a temper tantrum.

    The Google needs to get over it...
    Salty kisses, Sandy toes, and a Pirate's heart...

  3. #3
    Newbie
    Join Date
    March 11th, 2012
    Location
    New Jersey
    Posts
    22
    I've noticed that sometimes the warning is there, sometimes it's not. I don't see any negative effects with ranking for my keywords so I'll just keep ignoring it.

  4. #4
    Newbie
    Join Date
    May 25th, 2012
    Posts
    14
    I don't think it will give any nagetive effect to your website but what i recommend is:

    What should I do about this?

    - If your website pages are not replacing SERP of forum page, you should not lost those traffic by blocking those forum page.

    Should I just let it be the way it is?
    Should I request those pages to be removed?

    - If you are fine with traffic loss then yes..

    Should I remove that line from robots and allow Google to spider everything?

    - If you think that the traffic coming to those forum pages from Google SERP is your prospect customers then you should put some call to action to your forum pages that can divert those traffic to your website and can covert into sell.

  5. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. No change to Robots.txt but google says pages blocked
    By JI7009 in forum Search Engine Optimization
    Replies: 7
    Last Post: November 7th, 2009, 11:34 AM
  2. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  3. Blocking a spider from a directory - Robots.txt
    By Javi in forum Search Engine Optimization
    Replies: 2
    Last Post: November 4th, 2003, 12:41 PM
  4. how do i get slurp to crawl my pages, it only crawls my robots.txt
    By shamrock in forum Search Engine Optimization
    Replies: 10
    Last Post: September 10th, 2002, 08:09 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •