Results 1 to 4 of 4
  1. #1
    Full Member
    Join Date
    January 18th, 2005
    Posts
    396
    I know I'm Special but not this Special
    For the last five weeks I have been bombarded with spiderbots from all over Europe. These bots will use a particular IP and attempt to read my pages following my Google sitemap at a rate of 5/second or more. They attempt several hundred pages and then quit. Within 10 minutes another IP from a different part of Europe takes its try. I'm blocking them with a 'honey trap' at the start of my index.php and my real customers seem to be getting through but I am testing HostGator's bandwidth. Even with my blocking these bots add 100+ megabytes/day to my bandwidth

    Each IP uses many different 'Agent's so I can't block by IP or agent - this is a 10 second burst example using IP: 87.16.143.118 from Italy:

    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET)
    Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET)
    Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET)
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
    Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
    Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; InfoPath.1)
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET)
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
    Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)

    HostGator pointed out that if I block all of these Agents I have blocked all of the folks who use any version of Internet Explorer - not a good idea

    I can't be alone with this - what are you doing - what have you done?

    Charles

  2. #2
    general fuq mrbshouse's Avatar
    Join Date
    January 18th, 2005
    Location
    Argieville
    Posts
    1,381
    when the agent can't be blocked i just block the whole damn ip block...ready to block the whole ripe network altogether soon..

    is your site in Italian? or If the ip was coming from a server farm...aside from google msn yahoo and a handful of others....who cares..block the ips ;-)

  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    551
    Stick the ip and the time it hit your site in a database, if the ip hits more than twice/thrice in one second, block it for a while.

  4. #4
    Full Member
    Join Date
    January 18th, 2005
    Posts
    396
    Unfortunately they use many IP#s - I've seen only a few duplications in 5 weeks - maybe 400+ different IPs and since they come from Norway to Ukraine to Greece to Spain I haven't come up with a way to block all of the IP ranges - my trap that blocks them is about 98% effective and I do send them a random portion of a paper of the theoretical physics 'string theory' with each block - let them parse that.

  5. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Nothing Special
    By Mack in forum Virtual Family and Off-Topic
    Replies: 4
    Last Post: December 18th, 2007, 03:46 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •