Results 1 to 11 of 11
  1. #1
    ABW Ambassador
    Join Date
    January 4th, 2006
    Location
    USA
    Posts
    2,477
    Restricted by robots.txt without robots.txt?
    I went to check one of my sites status on google webmaster tool today and found out that there are hundreds of Crawl errors showing. The errer detail shows: URL restricted by robots.txt. Detected range from Jul 30-Aug 4.

    The weird thing is, I never had any robots.txt file set up. How would this happen? What should I do with it?

    I'd appreciate any helps.

  2. #2
    Moderator
    Join Date
    April 6th, 2006
    Posts
    2,689
    Do you use a CMS like Joomla, or Wordpress...?

    Sometimes these apps come with a predefined robots.txt - it may have been installed by default.

  3. #3
    ABW Ambassador
    Join Date
    January 4th, 2006
    Location
    USA
    Posts
    2,477
    Quote Originally Posted by teezone
    Do you use a CMS like Joomla, or Wordpress...?

    Sometimes these apps come with a predefined robots.txt - it may have been installed by default.
    Teezone, thanks for your reply.
    No, it's a Frontpage site. But I did use a template. Would that be the template's problem? Which file should I look into?

  4. #4
    15 years and counting
    Join Date
    January 18th, 2005
    Posts
    6,121
    If you use WordPress and this plugin XML Sitemap Generator for WordPress 3.1.4, check in the configuration page. You have an option "Add sitemap URL to the virtual robots.txt file. " Adding a virtual robots.txt is like adding a real one.

    Google webmaster tool is pretty stable right now but a few months ago before the last release they were driving me crazy with their robot.txt. One day they wanted one, next one I had to delete it.

    If you don't use WP, don't do anything for a few days, it will come back to normal. These guys at Google webmaster tool are nuts, always trying new code. Just resubmit your sitemap.

  5. #5
    ABW Ambassador
    Join Date
    January 4th, 2006
    Location
    USA
    Posts
    2,477
    Quote Originally Posted by Zeus
    If you don't use WP, don't do anything for a few days, it will come back to normal. These guys at Google webmaster tool are nuts, always trying new code. Just resubmit your sitemap.
    I sure hope it will back to normal. Since I don't recall if there are so many "Restricted by robots.txt" pages last time I checked.

    What scared me the most is the large number of urls that are "Restricted by robots.txt".

    Just want to make sure there is nothing weird set up with the template. I was trying to find if there is any robots file hiding inside the files that came with the template. Not sure what to look. Any advice?

  6. #6
    15 years and counting
    Join Date
    January 18th, 2005
    Posts
    6,121
    What you can do with Frontpage is to open a blank robots.txt page and upload it. It could solve your problem.
    Open FrontPage
    Open your website
    Selection "create new" > page in text mode
    Save as robots.txt in the root
    and upload.

    The page has to be in "text" mode not "html". You can also create the blank page in notepad.

    If you don't want your page blank you can add
    User-agent: *
    Disallow:

    It will not restrict anything.

  7. #7
    ABW Ambassador
    Join Date
    January 4th, 2006
    Location
    USA
    Posts
    2,477
    Thanks Zeus. I will add a blank robots.txt and see if it works. I hate to see hundreds of urls get restricted by google bots.

    I was afraid that it was the free template I'm using caused the problem. Since you guys didn't comment on that part, I assume it is not the case.

  8. #8
    15 years and counting
    Join Date
    January 18th, 2005
    Posts
    6,121
    Be patient with Google webmaster tool. I was ready to lose my sanity with them. All the robots.txt problems I had were coming from them, not me. It took me some time to discover they have a few idiots working there, too.

  9. #9
    ABW Ambassador 2busy's Avatar
    Join Date
    January 17th, 2005
    Location
    Tropical Mountaintop
    Posts
    5,636
    I have seen that error in GWT but when you use their tools to test the same URL they claim is restricted, it is not restricted. I don't pay any attention then because there isn't anything you can do to correct it. I've never seen hundreds of supposedly restricted pages though.

  10. #10
    ABW Ambassador
    Join Date
    January 4th, 2006
    Location
    USA
    Posts
    2,477
    Quote Originally Posted by 2busy
    I've never seen hundreds of supposedly restricted pages though.
    That's why I'm kind of scared since there are so many pages got the Restricted by robots.txt error on this site.

    My other sites get one or two restricted once in a while. I won't pay much attention to them either.

    I've add a robots.txt file to the site. Hopefully the problem will go away soon.

  11. #11
    Newbie
    Join Date
    August 26th, 2009
    Posts
    5
    Webmaster Tools is fried for me. Can't verify a site even though the webpage is present.

  12. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Robots.txt
    By Rhia7 in forum Midnight Cafe'
    Replies: 0
    Last Post: April 18th, 2009, 12:34 AM
  2. Does this do what I hope in robots.txt?
    By micheck in forum Programming / Datafeeds / Tools
    Replies: 8
    Last Post: January 20th, 2008, 07:40 PM
  3. Do you use a robots.txt?
    By Mr. Sal in forum Voting Booth
    Replies: 11
    Last Post: November 12th, 2003, 07:29 PM
  4. robots txt
    By reflections in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: December 26th, 2002, 06:22 PM
  5. robots.txt file????? HELP
    By john dundas in forum Search Engine Optimization
    Replies: 4
    Last Post: August 18th, 2002, 05:36 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •