Results 1 to 8 of 8
  1. #1
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    How to Block ALL Search Bots from Sub Domain?
    And I mean completely. 100%. They are not gettting in. I don't want the bots to have any access anywhere starting at the top level sub domain.

    I will use a pad lock, electronics fence, pit bulls, and anything else that's sharp to keep the bots out.

    Full proof blocking?

  2. #2
    ABW Ambassador
    Join Date
    October 22nd, 2006
    Posts
    1,065
    Make it password protected.

  3. #3
    Merchant & ABW Ambassador
    Join Date
    May 31st, 2006
    Location
    Houston TX
    Posts
    4,731
    http://www.webmasterworld.com/forum93/628.htm

    Q: Suppose I have a.domain.com and b.domain.com, I'd like to block only a.domain.com from Google. Is this possible? if yes, what is the right way to do it.


    A: Yes it is possible because every subdomain will be checked for its own robots.txt located in its root. Therefore in order to block all URLs from a.domain.com then you just need this robots.txt located in its root:
    User-agent: Googlebot
    Disallow: /

  4. #4
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    The problem with Robots.txt is that not all bots abide by it.

  5. #5
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Quote Originally Posted by sam_park
    Make it password protected.
    Good idea, but I still need the url to be accessible. I plan on directing all my outbound affiliate links to a redirect page on a sub domain, if that makes sense.

    So the sub domain url needs to be web accessible for the outbound link redirection and the sub domain needs to be unaccessible to any search engine/bot.

  6. #6
    http and a telephoto
    Join Date
    January 18th, 2005
    Location
    NYC
    Posts
    17,708
    SB, check your PMs please

  7. #7
    Full Member suzie250's Avatar
    Join Date
    May 5th, 2005
    Posts
    416
    If the outbound link is coming from your website, you should be able to block all traffic unless it is coming from that address in the sub domains htaccess.
    I disagree with Kay Jewelers. I would bet on any given Friday or Saturday night more kisses begin with Bud Lite than Kay.

  8. #8
    Full Member suzie250's Avatar
    Join Date
    May 5th, 2005
    Posts
    416
    hmm, after thinking about it, that's not gonna work.

    It would keep them out directly, but not if the bot follows the link. I suppose you could check it's IP address against a database before the redirect but if you have that info, why wouldn't you just block them to begin with?

    You don't say why you want to do that so I'm guessing you are looking for a way to get more accurate counts on real people clicks?

    It seems that most feel Javascript links are the best at tracking because even though some bots can follow them, most do not.
    I disagree with Kay Jewelers. I would bet on any given Friday or Saturday night more kisses begin with Bud Lite than Kay.

  9. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Do networks proactively block bots?
    By Georgie Peri in forum Midnight Cafe'
    Replies: 1
    Last Post: October 23rd, 2010, 12:14 PM
  2. Replies: 4
    Last Post: December 3rd, 2005, 07:08 PM
  3. Have you done a search of your domain name lately?
    By Mr. Sal in forum Search Engine Optimization
    Replies: 25
    Last Post: March 29th, 2005, 09:30 PM
  4. php and search bots
    By awstewar in forum Programming / Datafeeds / Tools
    Replies: 4
    Last Post: February 7th, 2004, 05:33 PM
  5. PPC Search Engine + Domain
    By coingallery in forum Midnight Cafe'
    Replies: 3
    Last Post: September 24th, 2002, 04:19 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •