Results 1 to 12 of 12
  1. #1
    Full Member 212TomTom's Avatar
    Join Date
    May 31st, 2007
    Location
    Canton, GA
    Posts
    203
    AdsBot - Was I sleeping when they started this?
    Is it just me, or did Google recently slide in a new rule about robots.txt files regarding the fact that the Google bot ignores 'User-agent: *' wildcard statements. Official quote from Google help section:

    Note: In order to avoid increasing CPCs for advertisers who don't intend to restrict AdWords visits to their pages, the system will ignore blanket exclusions (User-agent: *) in robots.txt files.

    Up until I read that, I always thought 'User-agent: *' kept everyone out. I'm usually pretty good about reading the guidelines, so maybe this is news to someone else too.

  2. #2
    Affiliate Manager Afilyit's Avatar
    Join Date
    April 25th, 2008
    Location
    Staten Island, NY
    Posts
    348
    So, what are they going to behave like a "bad bot" now?

  3. #3
    Moderator
    Join Date
    April 6th, 2006
    Posts
    2,689
    Whoaaa... I need to read that.. critical exclusions in some of my robots.txt files, to avoid duplicate content being spidered.

    Why is it always something..??

    Thanks for the catch....

  4. #4
    ABW Ambassador
    Join Date
    June 30th, 2007
    Location
    Syracuse, NY
    Posts
    677
    I'm confused.. so doing something like:

    User-agent: *
    Disallow: /folder1/
    Disallow: /folder2/

    Means google would completely ignore the disallows now?

  5. #5
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    Where in their "help" section is this (link)?

    Quote Originally Posted by 212TomTom

    Note: In order to avoid increasing CPCs for advertisers who don't intend to restrict AdWords visits to their pages, the system will ignore blanket exclusions (User-agent: *) in robots.txt files.
    That quote looks as if it's likely to be just talking about that bot they use to verify pages for AdWords (Adsbot-Google), rather than the general Googlebot. Also, the use of the term "the system" seems to imply a branch other than the search engine.

    I looked up the general page on how to keep Googlebot away here and, while it does give the vague impression that they just can't imagine anyone wanting to do that, (and doesn't admit that a wildcard can work) it doesn't say it'll ignore the wildcard, either.

    So I'd need to see the possibly-offending help page in context, before freaking out.

    Not that I'd put it past G to only update their FAQ in one place, and not where most people are going to see it...
    There is no knowledge that is not power. ~Hemingway

  6. #6
    Moderator
    Join Date
    April 6th, 2006
    Posts
    2,689
    Under Tools (in Webmaster tools) you can now generate robots.txt directly in the google interface. That's new - at least in last few months, as it wasn't there when I was testing versions for a site launch.

    Click on Generate - then Learn More at the top right:

    http://www.google.com/support/webmas...y?answer=40364

    "Googlebot follows the line directed at it, rather than the line directed at everyone."

    Pretty big change (for me anyway)..

    edit: just noticed you provided same link. I interpret that to mean google will now ignore the * flag, and you would need to specify User-agent: Googlebot, Disallow.

  7. #7
    Moderator
    Join Date
    April 6th, 2006
    Posts
    2,689
    As usual, this could be subject to google interpretation, but there's a pretty decent article here: http://searchengineland.com/080327-173946.php

    For me, I can't chance it, have already added Googlebot disallow.. my site would get penalized for duplicate content if they accidentally ignored the wildcard disallow (for any reason). Better safe than sorry, my rankings have been all over the map for 2 new sites...

  8. #8
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    "Googlebot follows the line directed at it, rather than the line directed at everyone."
    The examples I saw showed it having lines for G-bot, but, their examples were giving their bot different instructions than the ones for everyone--not just repeating what had already been done.

    So it seems, to me, that if there IS no line directed at it, it should pick up the clue (unless you're an AdWords advertiser). But it's G, so your mileage may vary.
    There is no knowledge that is not power. ~Hemingway

  9. #9
    Moderator
    Join Date
    April 6th, 2006
    Posts
    2,689
    I'm still surprised to see the ability to generate your robots.txt directly on the google site. Not sure how comfortable I am with an online editor for such an important file. Then again, you'll never see me sign up for the new Google Health records either!

    Also noticed an increase in number of their agents.

    Next time my site goes crazy in SERP, at least I can rest easy knowing it isn't robots.txt

  10. #10
    Full Member 212TomTom's Avatar
    Join Date
    May 31st, 2007
    Location
    Canton, GA
    Posts
    203
    Sorry folks, here's the page where I found that blurb:

    http://adwords.google.com/support/bi...n&answer=38197

    The data I posted applies only to the AdsBot, which is used by the Adwords program to help determine Quality Scores for Adwords.

    Due to personal reasons, the last week has been hell. Sorry for posting only partial data.

  11. #11
    ABW Ambassador purplebear's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,960
    Wasn't aware of this so will have to read more about it.

    " Due to personal reasons, the last week has been hell. Sorry for posting only partial data."

    I'm sorry about whatever that is and hope this week is better

  12. #12
    ABW Ambassador newestuser's Avatar
    Join Date
    February 14th, 2007
    Posts
    505
    sorry about the bad week, but unless the adbot sees the page you're putting their ads on, how will they know what stuff to show?
    Seems like it is reasonable to expect that the adwords bot would ignore the robots for pages with adwords on them.

  13. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Sleeping on the job
    By cpaemarket in forum Virtual Family and Off-Topic
    Replies: 2
    Last Post: October 22nd, 2008, 09:03 PM
  2. Adsbot-Google
    By KODea in forum Search Engine Optimization
    Replies: 1
    Last Post: February 24th, 2008, 04:05 PM
  3. How to wake up sleeping Affiliates?
    By Lisamac in forum Introduce Yourself
    Replies: 16
    Last Post: August 16th, 2005, 02:25 PM
  4. Amazon Sleeping with Morpheus
    By canfree in forum Other Affiliate Networks
    Replies: 12
    Last Post: September 24th, 2002, 02:39 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •