Results 1 to 11 of 11
  1. #1
    Full Member
    Join Date
    January 18th, 2005
    Posts
    126
    I want to set up a site that would be available to all search engines except Google. Is there any way to block just the Google spider?

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Nunya, Business
    Posts
    23,684
    I think there is a way, i'm sure someone who knows how will post it. Just curious as to why you would want to block Google?

    "Nothing focuses the mind better than the constant sight of a competitor who wants to wipe you off the map."
    --Wayne Calloway

  3. #3
    Defender of Truth, Justice and the Affiliate Way
    Join Date
    January 18th, 2005
    Location
    The Swamp
    Posts
    7,503
    you can block google in your robots.txt file

    Keep Your Hands Off My Cookies

  4. #4
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,082
    Robots Text file?
    I am curious also, why you want to exclude google.
    Cazzie

  5. #5
    ABW Veteran Student Heyder's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,482
    There are many good reasons to block google. I think they have a lot of nerve really.

    1. If you send a spider to google your entire IP class will be blocked by them yet they can spider away anywhere they please.

    what's good for the goose is good for the gander

    2. They hotlink image files and cost you bandwidth.

    3. They get everywhere and include everything even stuff you don't want seen.

    Don't get me wrong I depend on google for a huge chunk of change but I can see many points in blocking them.

  6. #6
    ABW Founder Haiko de Poel, Jr.'s Avatar
    Join Date
    January 18th, 2005
    Location
    New York
    Posts
    21,609
    Block google? Ok LMAO!

    <font size="2" face="Verdana">Haiko


    The secret of success is constancy of purpose. ~ Disraeli
    </font></p>

  7. #7
    Full Member
    Join Date
    January 18th, 2005
    Posts
    126
    Here's why -- we optimized our site for Google, and that worked, but apparently Inktomi considered some of the pages as unsuitable. They dropped it. So now I want to set up a companion site for Inktomi, but I don't want Google to see this as a copy or duplicate and ban both sites. I also believe this is the continuing problem of affiliate sites not being welcomed at the search engines.

    Anyway, if I set up a companion site, is there a way to keep Google away from it so the new site doesn't poison the existing site?

  8. #8
    Schlaumeier cumbrowski's Avatar
    Join Date
    January 18th, 2005
    Location
    Fresno/CA
    Posts
    640
    Tom,
    Go to: http://www.robotstxt.org/wc/robots.html
    You will find there anything you need for your problem.

    Carsten

    Shop-Links.net Partner
    Shop-Links.net

    If you can't move things, try harder!

  9. #9
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    User-agent: Googlebot
    Disallow: /

    Put that in your robots.txt - that will keep them out.

    I am curious to know if you are sure you have been dropped by Inktomi for the reasons you suggest, though.

    Search Engine Positioning - 1 Design 4 Life

  10. #10
    Full Member
    Join Date
    January 18th, 2005
    Posts
    126
    Thanks, Carsten.

    Markymark, I'm not positive about why Inktomi dropped my site, so I'm making a semi-educated guess. This was a paid-for-inclusion site through Position Tech, and suddenly all of the traffic just stopped. Oh, occasionally someone would take the time to page down to search listing item 405 on some obscure search term and we would actually get a visitor, but what had been a pretty high level of traffic just ground to a complete halt.

    I contacted Position Tech and they gave me a canned response. I expected that, and I don't fault them for it. Debugging web sites is not why they are in business.

    I tried changing some of the pages and nothing happened. I even substituted some of the pages but nothing happened. Obviously, we were hit with a huge penalty from Inktomi.

    My feelings on why the site was dropped may well be wrong. But I also have several other sites that are still doing fine with Inktomi. So, I figured that rather than spend an inordinate amount of time pounding away at this, I would just set up a companion site and try again.

  11. #11
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    I was going to suggest substituting a few pages, but you've already done it. Yeah, I think a second site is the way foward. It's what I'd do under the circumstances.

    Search Engine Positioning - 1 Design 4 Life

  12. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. The Google Spider
    By JP Sauve in forum Midnight Cafe'
    Replies: 18
    Last Post: September 8th, 2009, 03:10 PM
  2. Webrescuer Spider interfering with Google
    By darkstar7 in forum Search Engine Optimization
    Replies: 3
    Last Post: March 24th, 2004, 07:10 AM
  3. Dynamic Content Site, Google's scared to spider...
    By weisinator in forum Search Engine Optimization
    Replies: 18
    Last Post: December 9th, 2003, 05:11 PM
  4. does google spider web-protected files
    By oscar in forum Search Engine Optimization
    Replies: 5
    Last Post: December 1st, 2003, 03:03 PM
  5. Google Spider
    By hershey in forum Search Engine Optimization
    Replies: 1
    Last Post: November 27th, 2001, 06:25 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •