Results 1 to 15 of 15
  1. #1
    Member
    Join Date
    January 18th, 2005
    Posts
    189
    Quick question on something : 2 days ago I added a lot more pages to my site and added them to my MAIN page which has my site index on it. The total number of links on my index.html page now is about 175. I noticed the google spider came by and didn't grab these new pages - is this because 175 is too many links to have on the index.html, or is it simply a case of, sometimes the spider only gets a few pages, and sometimes it grabs everything it can?

    Dan

  2. #2
    The "other" left wing davidh's Avatar
    Join Date
    January 18th, 2005
    Location
    Boston
    Posts
    3,492
    It is not uncommmon for googlebot to hit your home page day after day, and only go deeper on an intermittent basis. Hang in there, they'll get around to it. Lately they been doing it a bit more often than before (used to be a once-monthly crawl for most sites)

  3. #3
    ABW Veteran Student Heyder's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,482
    Google has a mind or collection of minds of their own. They can not be predicted. What you are seeing is perfectly normal.

  4. #4
    Member
    Join Date
    January 18th, 2005
    Posts
    189
    Thanks a lot - first time I've had a site with this many pages...hopefully the Googlemeister will do a little digging soon...

  5. #5
    Full Member
    Join Date
    January 18th, 2005
    Posts
    243
    You might want to decrease your homepage links to 100 or less.

    Further, google seems to like pages 100k in size or less (the smaller the better) so try to keep your page size/load times down.

    Erik

  6. #6
    ABW Ambassador John Kruger's Avatar
    Join Date
    January 18th, 2005
    Location
    At my computer
    Posts
    645
    Does the 100k include graphics or simply the raw html?

    John

    Once a year, go someplace you've never been before.

    www.teampb.com

  7. #7
    Full Member
    Join Date
    January 18th, 2005
    Posts
    243
    Overall page size (inclding all html, graphics, etc.) of less than 100k has been my experience as being best for good ol' googlebot.

    In addition, link structure/navigation is vital, as well as a good sitemap (a must). In fact, I felt so strongly abot a sitemap, I had an automated one included in our shopping cart system hust to "help" the crawlers grab a new product page quick (i.e. whenever I add a new product, it is automatically added to the sitemap as well).

    Erik

  8. #8
    Member
    Join Date
    January 18th, 2005
    Posts
    189
    The page size is well under 100K. However, there is a sitemap on the page (it's the main page) I am linking to all major sections of my site, and the amount of links is about 175 - this could be the reason why google hasn't dug deeper yet? Guess I'll have to figure out which 75 links to pull off of the main page sitemap...damn!

    Dan

  9. #9
    ABW Veteran Student Heyder's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,482
    I wouldn't change anything.

  10. #10
    Full Member
    Join Date
    January 18th, 2005
    Posts
    480
    keep the links under 100. They tell you so. It is best to respect their request.

    Graphics do not matter. Googlebot does not grab them. However, over 100K is tough for modem users.

    Googlebot grabs a page... analyzes it, and only later will grab come back to get the new pages. It must first process the page it grabbed to find the next round of links( and to see if they are "worthy" of actually spidering.)


    I agree that google seems to have a mind of its own now. It really is not a good mind, IMHO. Googlebot will hardly to anything several days, and then out of friggen now where, it will slurp down 5 - 10,000 pages in a day. My server gets crushed at times.

  11. #11
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    767
    I agree. You should keep the links under 100.

  12. #12
    Member Clicks4Nut'n's Avatar
    Join Date
    January 18th, 2005
    Posts
    151
    agree. You should keep the links under 100.
    --------------------------------------------

    Suggested for every page or just the home page?

    "Merchant with no cookie get no clicky"

  13. #13
    Member
    Join Date
    January 18th, 2005
    Location
    Neyland Stadium West
    Posts
    56
    Danski,

    Can you pull the sitemap from your home page & put it on a sitmap.html page? Not knowing you site, that's what my generic recommendation would be.

    Good luck,
    rmjvol

  14. #14
    Full Member
    Join Date
    January 18th, 2005
    Posts
    243
    Yep...surprisingly (or maybe not so) we see most of the crawlers hitting our sitemap everyday now. In addition, our new product pages seem to be getting indexed within 48 hours or less (in google at least). Since many of them are 2-3 levels deep within the site, the only way they would be picked up so quick is via the sitemap

    Erik

    The AllPiercings.com Affiliate Program: 20% LIFETIME Commissions, 5% 2nd Tier Commissions, Affiliate Store Builder, Raw Datafeeds, and Advanced Tools.
    http://www.Allpiercings.com

  15. #15
    Member
    Join Date
    January 18th, 2005
    Posts
    189
    Thanks for the recommendation.

    I have the sitemap right below the text on the homepage. I am putting the sitemap on the homepage because if I dedicate a single page to it, and link to it from the homepage, that puts my main product categories at level 3, and the individual products at level 4 (google won't get them in as fast). I am going to try and trim down my sitemap (take some of the content pages that have already been spidered off of it - they are already linked to from other content pages anyway).

    Does this sound like the way to go?

    Any other comments on this?

    Thanks,

    Dan

  16. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Slow down spidering
    By dollardaze in forum Cusimano.com Scripts
    Replies: 2
    Last Post: November 18th, 2004, 07:07 AM
  2. Inktomi Spidering finally
    By cjk in forum Search Engine Optimization
    Replies: 1
    Last Post: January 24th, 2004, 09:14 AM
  3. mega google spidering going on
    By Heyder in forum Search Engine Optimization
    Replies: 8
    Last Post: December 10th, 2002, 12:15 PM
  4. Netfirms and Google Spidering - or lack thereof..
    By JadaKiss in forum Search Engine Optimization
    Replies: 1
    Last Post: May 2nd, 2002, 10:53 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •