Results 1 to 8 of 8
June 23rd, 2006, 11:02 AM #1Robots.txt questions
what do I need to have in my robots.txt file to only allow google, msn, and yahoo?
Also is there a way to ban scraper sites or other crappy sites from linking to your site?
Hope all is well with everyone,
June 23rd, 2006, 01:19 PM #2
Nice link thanks.
Question though If I disallow all with the specific bots still crawl?
June 23rd, 2006, 01:37 PM #3
Robots.txt won't do anything to stop the scraper bots. Most don't even bother reading it, others read and ignore. I'm testing out a spider trap right now. I've caught a few, but I still see a lot of suspicious activity.But are you still master of your domain?
June 23rd, 2006, 04:28 PM #4
Have to agree with NewCastleB.
I have added all kinds of edits to my robots.txt file to keep out the crap, with little affect.
I love the log spam I get...lol
June 24th, 2006, 01:30 AM #5
criminals aren't going to listen to your robots.txt requests
June 24th, 2006, 05:29 AM #6
I just seen my css code indexed after adding a robots.txt example
http://www.example.cpm/abc.css how would I go about blocking that from getting indexed since it is in my main root folder and not in a seperate /folder/ or should I not even care.
June 24th, 2006, 05:50 AM #7
- Join Date
- January 18th, 2005
- Los Angeles
June 24th, 2006, 07:21 AM #8
Thanks - Isnt it funny that you cant get a page listed in yahoo that you want, but you can get your css file into the search engine without even trying.
I went to yahoo and put one of my urls in and the first thing that popped up was the css file.
By mayfly in forum Search Engine OptimizationReplies: 10Last Post: August 26th, 2009, 04:13 PM
By Rhia7 in forum Midnight Cafe'Replies: 0Last Post: April 17th, 2009, 11:34 PM
By Hardaka in forum Newbie Affiliate FAQs & Helpful ArticlesReplies: 11Last Post: December 20th, 2007, 07:49 PM
By Mr. Sal in forum Voting BoothReplies: 11Last Post: November 12th, 2003, 06:29 PM
By reflections in forum Programming / Datafeeds / ToolsReplies: 5Last Post: December 26th, 2002, 05:22 PM