Results 1 to 10 of 10
  1. #1
    "An Englishman In New York" TJ's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,282
    I'm sure this cant be good Google has been chewing a bunch of pages from the sub-domain http://xxx.mydomain.com and it already has pages from http://www.xxx.com.

    Am I gonna get penalized for duplicate content? Is there a way to stop this?

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    England
    Posts
    4,327
    Isn't that what the robots.txt is for? I thought you could exclude content from being indexed?

    www.cjshoppingnetwork.com

  3. #3
    "An Englishman In New York" TJ's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,282
    How to set it to allow xxx.com but not xxx.mydomain.com?

  4. #4
    Defender of Truth, Justice and the Affiliate Way
    Join Date
    January 18th, 2005
    Location
    The Swamp
    Posts
    7,503
    I want google to crawl my subdomains. But my subdomains all have unique content, so the penalty issue isn't a factor.

    It's Your Money. You earned it. What are you going to do to make sure you get to keep it?

  5. #5
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR> How to set it to allow xxx.com but not xxx.mydomain.com? <HR></BLOCKQUOTE> .

    Well, all you need to do is put a robots.txt in the root of xxx.mydomain.com like this:

    User-agent: Googlebot
    Disallow: /

    Search Engine Positioning - 1 Design 4 Life

  6. #6
    "An Englishman In New York" TJ's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,282
    But this will prevent google from crawling the pages when it goes to xxx.com too, right?

    The pages for xxx.com are in the folder xxx.mydomain.com.... that's how the add-on domains are set up... I wonder if there is something that can be changed on the server to fix this?

    I cant understand why google is even crawling those pages, there are no links to them ANYWHERE!

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    Ah. I think I know what you're saying now. Correct me if I'm wrong - xxx.com is an add-on domain that points to xxx.mydomain.com. In effect, xxx.com and xxx.mydomain.com are one and the same. However, you are finding that Googlebot is spidering xxx.mydomain.com as well as (instead of ?) xxx.com. Is this right ?

    If so, it shouldn't really be a problem. Google will only display one set of pages and it should be those under the domain xxx.com as that's the one with the links pointing to it. In effect, the duplicate page filter will remove what it sees as one lot of duplicate pages.

    Search Engine Positioning - 1 Design 4 Life

  8. #8
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Utah
    Posts
    1,141
    I believe the answer is to set a 301 redirect from subdomain.domain.com to www.subdomain.com. The only problem is that I don't know how to do it. I know the .htaccess file is used, but I'm not sure what the syntax is.

    Glad I could be of help.

    Michael

  9. #9
    "An Englishman In New York" TJ's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,282
    ok Em... so what dyou think...

    redirect 301 / http://xxx.com/ - Will this cause a loop?

    or this...

    redirect 301 http://xxx.mydomain.com http://xxx.com/ - I have a feeling this is not good syntax

    Thanks to Mark for your input... I'd rather google didn't spider the entire site TWICE

  10. #10
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Utah
    Posts
    1,141
    You definitely don't want to get stuck in a loop. This is one of my projects for the weekend, so I'll let you know what I find out.

    Michael

  11. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Google Crawling nonexistent pages
    By NewcastleB in forum Search Engine Optimization
    Replies: 10
    Last Post: June 10th, 2005, 09:20 AM
  2. Is the (google)bot crawling?
    By AffiliateBuddha in forum Search Engine Optimization
    Replies: 11
    Last Post: February 8th, 2003, 12:49 PM
  3. Replies: 2
    Last Post: June 14th, 2002, 08:12 AM
  4. Google crawling...
    By MsMarySunshine in forum Search Engine Optimization
    Replies: 1
    Last Post: January 27th, 2002, 04:14 PM
  5. Google Bots are deep crawling all my sites
    By cowcool in forum Search Engine Optimization
    Replies: 19
    Last Post: December 31st, 2001, 07:57 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •