Results 1 to 3 of 3
  1. #1
    ABW Ambassador Doug247's Avatar
    Join Date
    January 18th, 2005
    DE USA
    Yahoo Ignores to robots.txt
    Part of my robots.txt file reads

    User-agent: YahooSeeker/1.2
    Disallow: /

    User-agent: Slurp
    Disallow: /

    User-agent: YahooSeeker
    Disallow: /

    and I continue to get crawled. They take about 5GB a month. Normally this would be a good thing but do not index the site.

    Anyone know what I can do.

    TIA DOug

  2. #2
    Not Verif-Lidated infoTim's Avatar
    Join Date
    January 18th, 2005
    Sunny Florida
    My only guess is to check your server logs to make sure that the robots.txt is being read sucessfully. Maybe it's unable to read it so it's not using it?

    - Tim
    consultant by day, affiliate by night

  3. #3
    Lite On The Do, Heavy On The Nuts Donuts's Avatar
    Join Date
    January 18th, 2005
    Winter Park, FL
    Also make sure you upload the file as Unix (happened to me once).
    Most ftp program have a mode where you can choose Unix format.
    Some html editors don't have a choice / mode and can default to the wrong thing.

    Use this tool to convert your file to Unix if in doubt:

    Then make sure you use an ftp tool set to Unix when you upload it to your root directory.

    Further guidance: (Start here!!) (Good resource!)

  4. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Robots Txt Plugin, Do You Use One?
    By Trust in forum Blogging, Mobile and Social Media
    Replies: 6
    Last Post: August 31st, 2010, 05:46 PM
  2. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 04:13 PM
  3. Robots.txt
    By Rhia7 in forum Midnight Cafe'
    Replies: 0
    Last Post: April 17th, 2009, 11:34 PM
  4. Do you use a robots.txt?
    By Mr. Sal in forum Voting Booth
    Replies: 11
    Last Post: November 12th, 2003, 06:29 PM
  5. robots txt
    By reflections in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: December 26th, 2002, 05:22 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts