Results 1 to 10 of 10
  1. #1
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    551
    Googlebot's eating well
    On one of my sites Googlebot normally takes about 200~350MB of bandwidth, but this month it has taken 1.05GB of bandwidth.

    I'm not complaining, I just want to find out if anyone else is seeing this.

    Also, does anyone know where I can get a list of Googlebot IP's, I want to check the IP against the user agent just cos I'm paranoid

    Thanks

  2. #2
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    I am also a little suspicious whether any "bad bots" may be masking themselves by claiming to be Googlebot. But I'm pretty sure that most of the queries I'm seeing are from Googlebot, because they follow up on earlier index-page slurps that Googlebot did 6-8 weeks earlier.

    I, too, have seen a huge increase in recent Googlebot activity -- it looks like about 5,000 to 7,000 pages per day are being indexed (out of more than half a million). About 10 days or so ago, the queries seemed almost random -- requesting pages from different directories -- but more recently there seems to be a very linear progression of files, slurping up an entire directory's contents before moving on to the next one.

    There was a big pause in late July and early August, when I was seeing only a few dozen Googlebot queries per day, so this is a relief -- I was worried they were simply not going to index my content pages, after sweeping my index pages in June and early July. Now I'm anxiously waiting for the next Google index update, to see how some of my different experiments will fare in the search results. I expect most to be dismal failures, treated as "thin affiliate" pages, but I'm hopeful that some of my different "content strategies" will bring my pages meaningful placement for some searches.

  3. #3
    Crazy like a fox suzigeek's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,096
    Yes googlebot has been voracious with most all of my sites.
    Waiting to see what happens....hopefully something good for all of us.
    Suz~~GearGirl~~

  4. #4
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    I've noticed a lot of Googlebot activity myself. So far it's downloaded over 10gb and 250,000 pages from me this month. It's so many that my session table got overloaded and started causing problems in my forum. I had to modify some code to ignore Googlebot.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  5. #5
    Dream Cruise megatonloh's Avatar
    Join Date
    June 5th, 2005
    Location
    Look East
    Posts
    418
    Yes mine to plus a lot of other crawlers and spiders.

    I look up for bad bots and found this:

    80.190.240.51 - ipx10044.ipxserver.de 2005-06-18 Googlebot/2.1 (+http://www.google.com/bot.html) N unknown bot, faked UA, hosted by ipxserver.de, DE

    Some name the real google bot url as googlebot/2.1 (+http://www.googlebot.com/bot.html)

    My log shows all the google bot ip start as 66-249-xx-xx

    Anyone can interprete these.

  6. #6
    Full Member Tech Evangelist's Avatar
    Join Date
    March 16th, 2005
    Location
    Mesa, AZ
    Posts
    374
    Google may be trying to catch up to Yahoo after Yahoo's recent announcement that they have 20 billion Web items indexed. Google has only 8,168,684,336 items indexed according to their home page.

    http://www.ysearchblog.com/archives/000172.html

    They could also be trying to identify as many Thin Affiliate sites as they can before the Christmas season starts so that they can penalize the sites and drive us all to AdWords.
    There's good, fast and cheap. Pick any two.
    [url=http://www.topranksolutions.com]Phoenix SEO[/url] :: [url=http://www.tech-evangelist.com/category/affiliate-marketing/]Affiliate Marketing Tutorials[/url]

  7. #7
    Full Member jerseyjim's Avatar
    Join Date
    February 1st, 2005
    Posts
    271
    [B]Yesterdays news, Yahoo crashes again,like when google ran it.
    JJ

  8. #8
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by Tech Evangelist
    They could also be trying to identify as many Thin Affiliate sites as they can before the Christmas season starts so that they can penalize the sites and drive us all to AdWords.
    Us? Does that mean you run Thin Affiliate sites?

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  9. #9
    Full Member jerseyjim's Avatar
    Join Date
    February 1st, 2005
    Posts
    271
    [B]What's a googlebot? I use them for ad sense, and search engines, they find all of my sites since I use the same domain name in all.
    JJ
    [FONT=Times New Roman]Even Einstein read comic books to relax[/FONT] :clap:

  10. #10
    Member johnm's Avatar
    Join Date
    August 24th, 2005
    Location
    minneapolis
    Posts
    65
    Googlebot has been voracious for me this month; freaking out on one of my sites in particular, eating up 12.7 GB worth of pages in the last 25 days alone. ~830k hits from GB, but the other SE bots haven't been nearly as busy.

  11. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Healthy eating during pregnancy????
    By cassandra4 in forum Dieting, Fitness & Health
    Replies: 2
    Last Post: December 5th, 2007, 02:46 PM
  2. Holiday Eating Tips
    By Rexanne in forum Virtual Family and Off-Topic
    Replies: 5
    Last Post: December 19th, 2005, 04:54 PM
  3. Cookie Eating Contest
    By Kcedit in forum Voting Booth
    Replies: 2
    Last Post: October 22nd, 2003, 01:59 PM
  4. What are you eating? What's eating you?
    By Linda - 5starAffiliatePrograms in forum Dieting, Fitness & Health
    Replies: 24
    Last Post: September 4th, 2003, 01:45 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •