I've been having nothing but fun this month with 3 spiderbots coming after some of my domains - SBIderbot, Twiceler, and MJ12bot have been attacking some of my domains at the rate of one or more per second.

My robots.txt files specifically tell them to go away - they read but ignore this file.

I tried to ban their specific IP# - they changed IP#

Googling a bit I found other souls being attacked and finally found a solution that works for me (so far...)

I enter the Apache configuration file and instead of banning a specific IP or range of IP's, I ban the signature name for the bots.

SetEnvIfNoCase User-Agent "^MJ12bot(.*)1\.0\.8" stay_out0

<Directory "C:/Apache/www">
Order allow,deny
Deny from env=stay_out0
Allow from all

They give me a bunch of ERROR 403 Forbidden's but I can ignore these