Results 1 to 2 of 2
March 29th, 2003, 05:43 AM #1
It doesn't look like a disallow would have much effect now that OpenBot has already been to my site:
<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR> Openbot does try to follow the robots.txt by filtering out URLs that are specified in the robot exclusion database. However, updating the robot exclusion database is done in batch mode once a day, and there is a chance that openbot has retrieved part of your web pages listed in your robots.txt before your robots.txt be fetched and processed into our robot exclusion database. Once openbot has noticed your robots.txt and learned the rule, it will not grab web pages listed in your robots.txt after then.<HR></BLOCKQUOTE>
Does OpenFind "contribute" to other SE's as they claim or is it one of reasons I get so much spam from asia?
"Half this game is ninety percent mental." -- Phillies Manager, Danny Ozark
March 29th, 2003, 11:44 PM #2
- Join Date
- January 17th, 2005
- United Kingdom
Yeah, exclude it. It probably isn't the reason why you are getting lots of spam from Asia though. I'm sure I read an interview with the owner where he said he was dropping OpenFind to work on a new search engine, but I'm not sure.
Search Engine Positioning - 1 Design 4 Life
By Uncle Rico in forum Midnight Cafe'Replies: 25Last Post: February 13th, 2008, 05:13 PM