Results 1 to 10 of 10
  1. #1
    Member
    Join Date
    May 18th, 2005
    Posts
    119
    Bad Bots Help
    Is there software or way to detect or trap / ban bad or rouge spiders that ignore robot.txt . I know there is probably no 100% solution.
    I tired of scrapers and bandwidth theft from bots that are parasites that I chose not to let in my site. I not looking for a list just a way to detect bots that ignore the robots text file in my root and then possibly auto ban them by IP.

  2. #2
    Crazy like a fox suzigeek's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,096
    I found a spider trap at WmW that was pretty simple. It uses a php script that catches/bans spiders that don't grab the robots.txt file.

    I'm not sure if it works...set it up on one site...

    If you do a search on webmasterworld for spider trap you'll find the post.
    Suz~~GearGirl~~

  3. #3
    Comfortably Numb John Powell's Avatar
    Join Date
    October 17th, 2005
    Location
    Bayou Country, LA
    Posts
    3,432
    Yep there are a couple of long how to posts there and it works. I have it on 5 sites and it's sending me an email every other day saying one got zapped. Just put "PHP spider trap" in G.


  4. #4
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by bumpaw
    Yep there are a couple of long how to posts there and it works. I have it on 5 sites and it's sending me an email every other day saying one got zapped. Just put "PHP spider trap" in G.
    Where do you put the "PHP spider trap"?

    In the meta? In the body? Is there a special code for this or is there a website you could point us to that would offer specific instructions?

  5. #5
    Comfortably Numb John Powell's Avatar
    Join Date
    October 17th, 2005
    Location
    Bayou Country, LA
    Posts
    3,432
    The trap can be found here. It consists of a link in your body to a php file on your server. You block the good bots from the file with robots.txt a few days before implementation.

    When the bad spider gobbles the trap file the script updates your .htaccess file to block it's IP. I added a little to mine so it would send me an email so I could feel the joy.


  6. #6
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Thanks for the code ... it looks a little complicated

  7. #7
    Member
    Join Date
    May 18th, 2005
    Posts
    119
    Quote Originally Posted by bumpaw
    The trap can be found here. It consists of a link in your body to a php file on your server. You block the good bots from the file with robots.txt a few days before implementation.
    Have you had any problems with it blocking good bots?

  8. #8
    Comfortably Numb John Powell's Avatar
    Join Date
    October 17th, 2005
    Location
    Bayou Country, LA
    Posts
    3,432
    Have you had any problems with it blocking good bots?
    No. I never have, but I always wait a few days after adding it to the robots.txt file to give the good ones a chance to read it.


  9. #9
    Full Member
    Join Date
    January 18th, 2005
    Location
    Des Moines, IA
    Posts
    298
    Looks like you have to pay $89 just to see it. Is it available anywhere else?

    Ray

  10. #10
    Comfortably Numb John Powell's Avatar
    Join Date
    October 17th, 2005
    Location
    Bayou Country, LA
    Posts
    3,432
    Looks like you have to pay $89 just to see it. Is it available anywhere else?
    Give it another try. It should be free to register, with only one area for premium folks.


  11. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Featured: Ban Bad Bots And Countries
    By BurgerBoy in forum Programming / Datafeeds / Tools
    Replies: 20
    Last Post: August 8th, 2013, 04:36 PM
  2. BAD BOTS (not sure if this is the right forum)
    By Gordon in forum Suspicious Activity!
    Replies: 16
    Last Post: January 28th, 2007, 11:29 PM
  3. Current bad bots htaccess list?
    By Andy in forum Spam
    Replies: 6
    Last Post: August 17th, 2004, 08:31 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •