Results 1 to 4 of 4
  1. #1
    Full Member
    Join Date
    January 17th, 2005
    Looking for ideas for site to dump a pest bot
    I have been getting blasted by a couple of rude spider bots - they won't pay any attention to robots.txt so I had to add them to my 'deny from' black list in the Apache configuration file. That works - the pests are denied access to my server and they get a 403 Forbidden error. That sort of makes them mad so they repeat trying to access the page I blocked them from - 30 or 40 times at 4/second before they move to another of my pages and repeat... Yesterday they hit me 9000 times. I can go into the Apache configuration and set it so the client that receives a 403 error is redirected some place - I need clean suggestions for where to send them. Thoughts???


  2. #2
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Wikipedia. The rogues want content? Give them LOADS of content...server-crashingly huge loads of content...

  3. #3
    Merchant & ABW Ambassador
    Join Date
    May 31st, 2006
    Houston TX
    Or send them to google groups.

  4. #4
    Newbie persistent's Avatar
    Join Date
    December 12th, 2007
    Unethical spiders and robots.txt
    I've always wondered about Robot.txt. If I remember right, it lists where the spider should not go, and not index. Seems like the unethical spiders would find that file and go to those files and directories first!

  5. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. What Does This Pest Want?
    By John Powell in forum Blogging, Mobile and Social Media
    Replies: 2
    Last Post: October 24th, 2008, 04:08 PM
  2. Replies: 0
    Last Post: January 13th, 2005, 12:26 AM
  3. Replies: 34
    Last Post: September 22nd, 2004, 09:15 PM
  4. WOW..yesterday was 'bot' day at my site!
    By frank3iii in forum Search Engine Optimization
    Replies: 3
    Last Post: October 22nd, 2003, 03:06 PM
  5. Any opinions on Pest Patrol?
    By swimmer in forum Blocking Tips/Advice/scripts
    Replies: 0
    Last Post: July 13th, 2003, 04:23 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts