Results 1 to 4 of 4
  1. #1
    ABW Ambassador Doug247's Avatar
    Join Date
    January 18th, 2005
    Location
    DE USA
    Posts
    931
    Banning a bot
    I want to completely ban Yahoo from crawling on of my sites. What do I need to pu in the robots.txt?

    I don't want the image bot, merchant bot, or any other bot related to yahoo crawling my site.

    TIA,
    Doug

  2. #2
    Full Member Tech Evangelist's Avatar
    Join Date
    March 16th, 2005
    Location
    Mesa, AZ
    Posts
    374
    User-agent: Slurp
    Disallow: /

    More info on the Yahoo Slurp page at:

    http://help.yahoo.com/help/us/ysearc.../slurp-02.html

  3. #3
    ABW Ambassador Doug247's Avatar
    Join Date
    January 18th, 2005
    Location
    DE USA
    Posts
    931
    Thanks. Yahoo takes about 3 GB a month but they booted me from thier index back in DEC. 04. I'm actually trying to disallow them for a while to see if I can get back in after a month or so.


    Thanks,
    Doug

  4. #4
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Texas, USA
    Posts
    579
    China, Yahoo, eBay ???


    Hi,

    I've been having problems with some kind of bot from China pounding my site apparently on behalf of Yahoo and eBay. Close to 2,000 hits last month.

    I looked up the IP# in my Latest Visitors Apache web server log several days ago and entered it into my Apache IP Deny Manager. It seemed to stop the hits.

    Now, several days later, it has switched to another IP# which I promptly entered into my Apache IP Deny Manager. 120 pages and 120 hits today.
    Could it be some kind of crazed programming error?

    Does one have ANY idea what is with China, Yahoo and eBay? I emailed
    the hostmaster in China inquiring about the matter but never recieved a reply.

    Thanks,

    RadarCat, Webmaster
    http://www.os2warplinks.com

  5. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Banning a bot
    By Doug247 in forum Midnight Cafe'
    Replies: 2
    Last Post: April 1st, 2005, 12:34 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •