Results 1 to 2 of 2
  1. #1
    Join Date
    January 17th, 2005
    Hello -

    The spider using these IP addresses is taking up a LOT of bandwidth! It says it is from South Korea. Here are the IP's:

    No other information is given, not DNS name or anything. My first question is, should I disallow them in the robots.txt (just making sure it isn't one of the 4 or 5 major SE's).

    Second question, what if they don't obey the robots.txt after I ban them? Is it possible to have the host ban these IP's from my server?

    How would you guys handle this, and does anyone have a good template robots.txt file that eliminates alot of this?



  2. #2
    ABW Ambassador
    Join Date
    January 17th, 2005
    Look at a Korean search engine/portal. Might be the one.

    You have to ban them using the spider name in the robots.txt


  3. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. good spider or bad spider?
    By Gordon in forum Newbie Affiliate FAQs & Helpful Articles
    Replies: 3
    Last Post: September 3rd, 2004, 08:38 AM
  2. New spider?
    By womanht in forum Midnight Cafe'
    Replies: 1
    Last Post: August 8th, 2004, 05:00 AM
  3. CJ Spider?
    By appbizz in forum Midnight Cafe'
    Replies: 1
    Last Post: April 7th, 2003, 09:12 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts