Results 1 to 2 of 2
  1. #1
    ABW Ambassador Doc Sawyer's Avatar
    Join Date
    January 18th, 2005
    Southern California Desert
    It doesn't look like a disallow would have much effect now that OpenBot has already been to my site:
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR> Openbot does try to follow the robots.txt by filtering out URLs that are specified in the robot exclusion database. However, updating the robot exclusion database is done in batch mode once a day, and there is a chance that openbot has retrieved part of your web pages listed in your robots.txt before your robots.txt be fetched and processed into our robot exclusion database. Once openbot has noticed your robots.txt and learned the rule, it will not grab web pages listed in your robots.txt after then.<HR></BLOCKQUOTE>

    Does OpenFind "contribute" to other SE's as they claim or is it one of reasons I get so much spam from asia?



    "Half this game is ninety percent mental." -- Phillies Manager, Danny Ozark

  2. #2
    ABW Ambassador
    Join Date
    January 17th, 2005
    United Kingdom
    Yeah, exclude it. It probably isn't the reason why you are getting lots of spam from Asia though. I'm sure I read an interview with the owner where he said he was dropping OpenFind to work on a new search engine, but I'm not sure.

    Search Engine Positioning - 1 Design 4 Life

  3. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. List of Robots To Disallow?
    By Uncle Rico in forum Midnight Cafe'
    Replies: 25
    Last Post: February 13th, 2008, 05:13 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts