Results 1 to 2 of 2
  1. #1
    Member
    Join Date
    January 18th, 2005
    Posts
    189
    Hello -

    The spider using these IP addresses is taking up a LOT of bandwidth! It says it is from South Korea. Here are the IP's:

    218.145.25.81
    218.145.25.17
    220.73.165.203
    220.73.165.75

    No other information is given, not DNS name or anything. My first question is, should I disallow them in the robots.txt (just making sure it isn't one of the 4 or 5 major SE's).

    Second question, what if they don't obey the robots.txt after I ban them? Is it possible to have the host ban these IP's from my server?

    How would you guys handle this, and does anyone have a good template robots.txt file that eliminates alot of this?

    Thanks,

    Dan

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,663
    Look at www.naver.com a Korean search engine/portal. Might be the one.

    You have to ban them using the spider name in the robots.txt

    Wayne

  3. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. good spider or bad spider?
    By Gordon in forum Newbie Affiliate FAQs & Helpful Articles
    Replies: 3
    Last Post: September 3rd, 2004, 09:38 AM
  2. New spider?
    By womanht in forum Midnight Cafe'
    Replies: 1
    Last Post: August 8th, 2004, 06:00 AM
  3. CJ Spider?
    By appbizz in forum Midnight Cafe'
    Replies: 1
    Last Post: April 7th, 2003, 10:12 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •