Results 1 to 4 of 4
  1. #1
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    robots.txt not working ??
    superCool is seeing pages in google that he thinks should be protected by robots.txt

    the robots file is in the root directory and looks like this

    User-agent: *
    Disallow: /aaaaaaa.html
    Disallow: /bbbbbbb.php
    Disallow: /ccccccccc/
    Disallow: /ddd/

    but this url is showing up in google (using search for

    that url is affected by a rewrite rule in htaccess. could that be causing this? wouldn't google apply the robots.txt rules to the final url (as shown above)?

    did superCool do something wrong? gooble webmaster tools shows the correct robots info


  2. #2
    ABW Ambassador 2busy's Avatar
    Join Date
    January 17th, 2005
    Tropical Mountaintop
    maybe /ddd/*.html or /ddd/*.* might fix it?
    Any way to add noindex to the meta?
    Make sure it is not showing in your sitemaps and you can remove the URI from goofle's index in GWT.

  3. #3
    SEO: A Specialty - Web Design: Slow or outsourced andbeyond's Avatar
    Join Date
    June 18th, 2006
    The Call is coming from Inside the House!
    You can disallow pages or directories in Webmaster Tools. They have to be currently throwing a 404 or be blocked by robots.txt.

    I have seen the same problem. It seems like Google is not really following robots.txt for some sites. It indexes folders it shouldnt.

    I have recently started using multiple versions of folders I dont want indexed.


    Disallow: /folder
    Disallow: /folder/
    Disallow: /folder/*.*

    I dont really have an answer yet if this is helpful.

  4. #4
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    superCool will give those suggestions a shot. it's a bit of a befuddling frustration when you follow every example you find on the net but it still doesn't work. we'll see what happens. superCool's new site is just now being indexed (at a snail's pace) and half the pages are the ones that shouldn't be included . oh well, that's the way it goes sometimes

    thanks friends

  5. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Robots Txt Plugin, Do You Use One?
    By Trust in forum Blogging, Mobile and Social Media
    Replies: 6
    Last Post: August 31st, 2010, 06:46 PM
  2. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  3. Robots.txt
    By Rhia7 in forum Midnight Cafe'
    Replies: 0
    Last Post: April 18th, 2009, 12:34 AM
  4. Do you use a robots.txt?
    By Mr. Sal in forum Voting Booth
    Replies: 11
    Last Post: November 12th, 2003, 07:29 PM
  5. robots txt
    By reflections in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: December 26th, 2002, 06:22 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts