Page 1 of 5 12345 LastLast
Results 1 to 25 of 105
  1. #1
    lurk
    Join Date
    March 25th, 2005
    Posts
    355
    Google Sitemaps
    Thought this would be an interesting one to share for all you SEO freaks...


    https://www.google.com/webmasters/sitemaps/login

  2. #2
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Wow, I have a feeling this might become major.

    By placing a Sitemap-formatted file on your webserver, you enable our crawlers to find out what pages are present and which have recently changed, and to crawl your site accordingly.
    They go on to say this will help get more of your site crawled. It includes some very complicated instructions on building an XML sitemap that you serve with your webpages. With a dynamic website I don't even know where to begin.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  3. #3
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    This gets more and more interesting. They let you attribute a priority to your pages so you can tell the bots what pages to visit first when they come to your site. I need to start working on a sitemap generator!

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  4. #4
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    Google must have its *ss on backwards. Get this, last year I made a site with a "sitemap." I called that page... sitemap.html (duh)

    Google IGNORED IT (as in, it didn't put that page in the index despite spidering it...I mean the "no results for this URL" kind of nowhere) until finally I decided, "maybe G doesn't really like sitemaps. I'll rename it."
    So I did. And, G had that page listed within a week.

    But NOW they're not only saying to make sitemaps, but giving out a program to do it!
    I wonder what changed the real policy?

    *throws up hands*
    There is no knowledge that is not power. ~Hemingway

  5. #5
    Special Liaison
    Join Date
    January 18th, 2005
    Location
    Detroit, USA
    Posts
    63
    This is using an XML page, so it is different to your sitemap.html page. I think you just got unlucky, sitemap.htm pages have done alright for me - the main aim is not to rank, but to allow easy access for the spiders to reach your other pages, so the fact your sitemap page didn't rank at all is no great loss.

    I like this idea from Google, interesting to see how it pans out and how it wil be abused

  6. #6
    Lite On The Do, Heavy On The Nuts Donuts's Avatar
    Join Date
    January 18th, 2005
    Location
    Winter Park, FL
    Posts
    6,930
    link to your site should be on your home page as well... in keeping with the theory of helping the crawlers not make a mistake... let them find the sitemap and find it fast.

  7. #7
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    Great find! I just built a sitemap for one of my sites, and I'm looking forward to seeing how well it helps Google spider the site.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  8. #8
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,205
    I have to access the server via command line, I'm not sure my host lets me perform such tasks.

  9. #9
    Full Member
    Join Date
    January 18th, 2005
    Location
    UK
    Posts
    273
    Nice find.

    Would have been even nicer if site generator proggy was available in PHP as well as being Python

    Les

  10. #10
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    718
    Is Google being smarter than we give her credit for here?

    If I wanted to have the BIGGEST web database in the world, without spending the cash needed to keep it running, why not encourage the millions of webmasters out there who need my traffic so badly to essentially store that data for me in a massive 'distributed database'?

    So far as I can see, we would be generating a single file which would significantly reduce Google's crawling requirements - Google would simply examine this file, isolate updates and, possibly, crawl only the updated sections.

    Not that I perceive this to be a bad thing at all, just think it's interesting to think about what led Google to develop this?

  11. #11
    ABW Veteran Mr. Sal's Avatar
    Join Date
    January 18th, 2005
    Posts
    6,795
    I only wonder what would happen if you have over 10k pages on one site.

    The other day I tried the CoffeeSiteMapper.exe on a site with less than 5k pages and the sitemap file generated was over 1.2MB in size, so I didn't uploaded that file to my site because I thought that it was too big for the SE's to crawl.

  12. #12
    "An Englishman In New York" TJ's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,282
    Quote Originally Posted by microdot
    Is Google being smarter than we give her credit for here?

    If I wanted to have the BIGGEST web database in the world, without spending the cash needed to keep it running, why not encourage the millions of webmasters out there who need my traffic so badly to essentially store that data for me in a massive 'distributed database'?

    So far as I can see, we would be generating a single file which would significantly reduce Google's crawling requirements - Google would simply examine this file, isolate updates and, possibly, crawl only the updated sections.

    Not that I perceive this to be a bad thing at all, just think it's interesting to think about what led Google to develop this?
    Quite possibly... I'm intrigued by the line in the output that says
    Notifying: www.google.com

  13. #13
    ABW Ambassador buy_online's Avatar
    Join Date
    January 18th, 2005
    Location
    Richmond, VA
    Posts
    3,234
    Google has recommended a site-map for quite a while now (one-year plus). And now we might be looking at meta tags too.

    Fred

  14. #14
    "An Englishman In New York" TJ's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,282
    Seems to me that it lists WAY too much stuff in the sitemap it creates.... even with filters it seems to be a real pita!

  15. #15
    Super Sh!t Stirrer SSanf's Avatar
    Join Date
    January 18th, 2005
    Posts
    9,944
    Well, their explainantion page is all gobbledy gook to me.

    I tried CoffeeSiteMapper.exe with .shtml files on a Front Page web and it didn't work at all. Has anyone been able to get it to work that way?
    Comments are opinion unless otherwise noted. Remember, pillage first. Then burn. Half of all people in the world have IQs under 100. You best learn to trust ol' SSanf!

  16. #16
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    Quote Originally Posted by Mr. Sal
    I only wonder what would happen if you have over 10k pages on one site.
    You can have up to 50,000 URLs in a sitemap file. If you have more than 50,000 URLs, you can create multiple sitemaps and reference up to 1000 of them in a master sitemap index file. That's 50 million URLs total.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  17. #17
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    I was happy to hear about the 50,000 URLs because it puts less pressure on me to map the entire site, but now this 50 million is a lot more responsibility! I'm just afraid of the added work this is going to create. Hope it's worthwhile!

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  18. #18
    ABW Veteran Mr. Sal's Avatar
    Join Date
    January 18th, 2005
    Posts
    6,795
    Quote Originally Posted by MichaelColey
    You can have up to 50,000 URLs in a sitemap file. If you have more than 50,000 URLs, you can create multiple sitemaps and reference up to 1000 of them in a master sitemap index file. That's 50 million URLs total.
    Michael,

    Right now I don't know if I have to kill you or thank you!

    Are you 100% sure that the search engines (specially Google) will index that many links on just one sitemap page?

    If they do indeed index all of those links on just one sitemap page, I will take you to dinner, but...... If I make a sitemap with just 10k links and wait for that page to load on my Internet Explorer () and see that no SE index that page in the next three months, I will fly to Mansfield, TX and do this or this or even this to you.

    So unless you're 100% sure that they will index all of those links in just one page, don't get me inspired to make a big sitemap page. I still using a dial-up connection and that would take a long time to just check that new sitemap page offline, so let me know when you're sure.

    Sal.

  19. #19
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    I think Google will still only index and spider the number of pages that your PageRank warrants. If Google was spidering 100 pages a day in the past, they'll probably do the same now. The difference being you have control over which 100 pages get spidered and which should be spidered first. Very interesting stuff.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  20. #20
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    Sal: It's not a normal HTML site map. It's a specially formatted XML file that you have to create. Click through the link in the original post above and read the details.

    If you create the file and submit it using the directions they provide, it will let Google know about the pages, but won't necessarily guarantee that they'll be spidered, indexed, or ranked. It's supposed to help Google spider your site more intelligently.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  21. #21
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    My first thought about implementing this was to build a sitemap of my most popular pages. I can use the click stats in my database to build a new sitemap every day. I just need to figure out a good priority range. Wow, this is going to be quite a task. I like how you have the freedom to establish a system of priority as complicated or as simple as you'd like.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  22. #22
    Affiliate Marketer Rogi's Avatar
    Join Date
    January 18th, 2005
    Location
    Melbourne
    Posts
    415
    Mr. Sal

    [taken from url: https://www.google.com/webmasters/si...protocol.html]

    Providing Multiple Sitemap Files
    You can provide multiple Sitemap files, but each file that you provide must have no more than 50,000 URLs and must be no larger than 10MB (10,485,760) when uncompressed. These limits help to ensure that your Web server does not get bogged down serving very large files.

    If you want to list more than 50,000 URLs, you must create multiple Sitemap files. If you anticipate your Sitemap growing beyond 50,000 URLs or 10MB, you should consider creating multiple Sitemap files. If you do provide multiple Sitemaps, you must list them in a Sitemap index file. Sitemap index files may not list more than 1,000 Sitemaps. Your Sitemap index file could be named Sitemap_index.xml.

    The XML format of a Sitemap index file is very similar to the XML format of a Sitemap file. The Sitemap index file uses the following XML tags:

    * lastmod
    * loc
    * sitemap
    * sitemapindex

    Note: A Sitemap index file can only specify Sitemaps that are found on the same site as the Sitemap index file. For example, http://www.yoursite.com/sitemap_index.xml can include Sitemaps on http://www.yoursite.com but not on http://www.mysite.com or http://yourhost.yoursite.com.

    Sample XML Sitemap Index

    The following example shows a Sitemap index in XML format. The Sitemap index lists two Sitemaps:
    <?xml version="1.0" encoding="UTF-8"?>
    <sitemapindex xmlns="http://www.google.com/schemas/sitemap/0.84">
    <sitemap>
    <loc>http://www.mysite.com/sitemap1.xml.gz</loc>
    <lastmod>2004-10-01T18:23:17+00:00</lastmod>
    </sitemap>
    <sitemap>
    <loc>http://www.mysite.com/sitemap2.xml.gz</loc>
    <lastmod>2005-01-01</lastmod>
    </sitemap>
    </sitemapindex>

    Note: Sitemap URLs, like all values in your XML files, must be XML-encoded.

    Sitemap Index XML Tag Definitions

    *

    The loc tag is required and identifies the location of the Sitemap.
    *

    The lastmod tag is an optional tag that identifies the time that the corresponding Sitemap file was modified. It does not correspond to the time that any of the pages listed in that Sitemap were changed. The value for the lastmod tag should be in ISO 8601 format.

    By providing the last modification timestamp, you enable search engine crawlers to retrieve only a subset of the Sitemaps in the index — i.e. a crawler could only retrieve Sitemaps that were modified since a certain date. This incremental Sitemap fetching mechanism allows for the rapid discovery of new URLs on very large sites.
    *

    The sitemap tag encapsulates information about an individual Sitemap.
    *

    The sitemapindex tag encapsulates information about all of the Sitemaps in the file.



    Michael Coley is right. Go ahead and build that big sitemap file.

  23. #23
    ABW Veteran Mr. Sal's Avatar
    Join Date
    January 18th, 2005
    Posts
    6,795
    Thanks for that info Michael and Rogi,

    I will try all that by next week, right now I am on overdrive with a lot shit to do with a lot of new info and sites to update this weekend.

    I will post here when I make that new sitemap and see how it goes.

    Thanks.

    Sal.

  24. #24
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Texas, USA
    Posts
    579
    Lightbulb Google Sitemaps


    Hi, All,

    re: Google Sitemaps

    From the GoogleBlog at:

    http://googleblog.blogspot.com/


    Webmaster-friendly

    6/2/2005 05:29:00 PM

    Posted by Shiva Shivakumar, Engineering Director

    We're undertaking an experiment called Google Sitemaps that will either fail miserably, or succeed beyond our wildest dreams, in making the web better for webmasters and users alike. It's a beta "ecosystem" that may help webmasters with two current challenges: keeping Google informed about all of your new web pages or updates, and increasing the coverage of your web pages in the Google index.

    Initially, we plan to use the URL information webmasters supply to further improve the coverage and freshness of our index. Over time that will lead to our doing an even better job of delivering more search results from more websites. (Danny Sullivan interviewed me about this if you want to read more.)

    This project doesn't just pertain to Google, either: we're releasing it under the Attribution/Share Alike Creative Commons license so that other search engines can do a better job as well. Eventually we hope this will be supported natively in webservers (e.g. Apache, Lotus Notes, IIS). But to get you started, we offer Sitemap Generator, an open source client in Python to compute sitemaps for a few common use cases. Give it a whirl and give us your feedback.
    I hope that this will help content sites that are buried under tons of old sites full of dead links that have been floating around the web for years.

    Best,
    RadarCat, WebMaster
    http://www.os2warplinks.com

  25. #25
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    There was a thread about this in the Midnight Cafe yesterday:

    - URL REMOVED AS I MERGED THE THREADS -

    Quite a bit of discussion there.
    Last edited by Haiko de Poel, Jr.; June 13th, 2005 at 12:11 PM.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

+ Reply to Thread
Page 1 of 5 12345 LastLast

Similar Threads

  1. Google Sitemaps
    By kbrock in forum Search Engine Optimization
    Replies: 4
    Last Post: February 20th, 2007, 11:53 PM
  2. Google Sitemaps: Visited it lately?
    By Herb ΤΏΤ¬ in forum Search Engine Optimization
    Replies: 1
    Last Post: October 13th, 2005, 04:39 AM
  3. Google sitemaps arrggggg
    By suzigeek in forum Search Engine Optimization
    Replies: 15
    Last Post: September 9th, 2005, 04:27 AM
  4. Google Sitemaps and Subdomains
    By Zoe2 in forum Search Engine Optimization
    Replies: 5
    Last Post: June 26th, 2005, 10:24 AM
  5. Google Sitemaps
    By jrod805 in forum Midnight Cafe'
    Replies: 62
    Last Post: June 10th, 2005, 09:31 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •