Results 1 to 16 of 16
  1. #1
    Crazy like a fox suzigeek's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,096
    Google sitemaps arrggggg
    I have been futzing for almost a WEEK TRYING to make a site map for my dynamically generated website(s).. I have tried phpSitemapNG that was mention earlier on this board. I've tried the python google version but I keep getting cannot read config file error and I've tried a bunch of online ones that just don't work w/dynamic or large sites!!!

    Does anyone know of any sitemap creator that will work for dynamic sites with over 10000 pages? I need a sitemap for other purposes also aside from the google sitemap experiment. I was thinking of some kind of scraper or spider software--I don't want to scrape just want a list of my own urls.

    Any help from anyone I'll be eternally grateful My kids will be thankful also - I am so crossed eyed my kids don't recognize me!!!
    Suz~~GearGirl~~

  2. #2
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    I build a program from scratch for my sites. As long as you can generate a list of all URLs, then extract the change date from the physical files for static pages and from your database for dynamic pages, it's fairly simple. If you don't have change dates in your database, I'm not sure how you would do it.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  3. #3
    Full Member asr_guy's Avatar
    Join Date
    June 12th, 2005
    Location
    Canada
    Posts
    310
    Hi Suzi,
    Did you try this Google Sitemap maker thing?.

    I'm no expert at this but came across it.

    Cheers,
    Peter

  4. #4
    Crazy like a fox suzigeek's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,096
    Yes asrguy and I had high hopes for it but it seems do stop working at around 3000-5000 depending and thats after letting it spider for over an hour......

    Micheal I'm fairly new to really grasping my php sites i.e. I hacked a cms and a catalog program to work together... added a bunch of extra features (cross selling and such) which I'm pretty proud of actually ...but I have 2 little children yapping at me and my programming patience is on the wane this week and I imagine the next few weeks so I'm looking for a ready to use solution...


    My dbs aren't using change dates as of yet. I'm still manually dropping then inserting. My next task was automating but then I got the site map bug. It will also help me with shortening my urls... on and on it goes...
    Suz~~GearGirl~~

  5. #5
    Full Member
    Join Date
    January 18th, 2005
    Location
    UK
    Posts
    273
    I have had a look at phpSitemapNG and it seems quite nifty but then I realized for a dynamic database site it makes more sense to do as Michael says and generate a list of the URLs rather than spider the site.

    So in my case I have a category table and my URLS are in the format

    www.abc.com/categories/catnumber/cat-name/

    so I just need to build a list of URLs so they look like

    categories/catnumber1/cat-name1/
    to
    categories/catnumber999/cat-name999/

    and then process this is a similar fashion to the phpSitemapNG XML generation code to produce the XML file.

    I have a newish datafeed site that Google has hardly indexed at all so I have a site that makes a perfect test pilot.

    Hope that helps a bit.

    Les

  6. #6
    ABW Veteran Mr. Sal's Avatar
    Join Date
    January 18th, 2005
    Posts
    6,795
    I have not submitted any sitemap to Google yet, but the other day I discovered that I already had all my links listed in one text file on my cgi-bin.

    Since I use the Fluid Dynamics Search Engine on my site, every time that I re-index the site, I get a new updated text file with all of the links that I would like all the search engines to have when they index my site.

    The FDSE creates a file called search.pending.txt on the searchdata directory and I think that is a good text file to rename as sitemap.txt and submit it to Google, but I still thinking about it.

    On that search.pending.txt file, the only thing I have to remove is the 10 digits that is at the end of all the links, but that's simple to do with the add-replace of WordPad or any other text editor.

  7. #7
    Crazy like a fox suzigeek's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,096
    Yeah I was thinking about it last night and it would be pretty easy to generate a map with the category table and sku's for individual pages. I was just so burnt on solving other issues I couldn't/didn't want to think about writing any more code.

    New day new brain! Thanks for helping me see it more clearly
    Suz~~GearGirl~~

  8. #8
    Full Member
    Join Date
    January 18th, 2005
    Location
    UK
    Posts
    273
    Quote Originally Posted by suzigeek
    Yeah I was thinking about it last night and it would be pretty easy to generate a map with the category table and sku's for individual pages. I was just so burnt on solving other issues I couldn't/didn't want to think about writing any more code.

    New day new brain! Thanks for helping me see it more clearly
    No thanks are due, Abw is a great example of a problem shared is a problem halved.

    One of the things I miss about working in a office is the ability to bounce ideas and problems off one particular person (and he would do the same with me as what one of us missed the other would usually spot).

    Now I have to go and make a cup of coffe or tea and do something completely different for a while.
    Sometimes "a while" can be as short as 10 mins sometimes it can run into days.

    Les

  9. #9
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    If you can get modification dates, that makes things much more useful to Google. With just a list of URLs, they don't know what changed so they still have to spider everything. When you give them the modification dates, they won't need to bother with the URLs that you tell them haven't changed.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  10. #10
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,403
    Try a search for " online sitemaps" and you will get very easy usable results.

    roland

  11. #11
    Member
    Join Date
    January 18th, 2005
    Posts
    64
    I tried a google sitemap for a WP blog and it was a 5 minute job, worked pretty well.

  12. #12
    Not Verif-Lidated infoTim's Avatar
    Join Date
    January 18th, 2005
    Location
    Sunny Florida
    Posts
    1,021
    You know, I usually say if something works, don't mess with it, and I should have listened to my own advice. Google did not like some aspect about the content of my site map and has punished the heck out of the site. It stopped crawling it, and has actually ripped 80% of the pages it had out of the cache. This was a site that was getting good natural SERPS, collecting organic backlinks, all that good stuff. The only pages left now are ones that actually had backlinks. Everything else is toast.

    Bye-bye site map...hopefully things correct themselves back to where they were.
    Tim
    consultant by day, affiliate by night

  13. #13
    Crazy like a fox suzigeek's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,096
    I hate it when that happens!

    Well I gave up on it and I'm concentrating on making my sites navigation to be more easily spidered.

    Sorry Google betrayed you that way! All you were trying to do was help it out. Geez no gratitude!!
    Suz~~GearGirl~~

  14. #14
    ABW Ambassador netnow22's Avatar
    Join Date
    January 18th, 2005
    Location
    Columbia, SC
    Posts
    748
    Quote Originally Posted by infoTim
    You know, I usually say if something works, don't mess with it, and I should have listened to my own advice. Google did not like some aspect about the content of my site map and has punished the heck out of the site. It stopped crawling it, and has actually ripped 80% of the pages it had out of the cache. This was a site that was getting good natural SERPS, collecting organic backlinks, all that good stuff. The only pages left now are ones that actually had backlinks. Everything else is toast.

    Bye-bye site map...hopefully things correct themselves back to where they were.
    THAT IS WORTH REPEATING!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

  15. #15
    Super Sh!t Stirrer SSanf's Avatar
    Join Date
    January 18th, 2005
    Posts
    9,944
    So, if you have a site that is a PR4 and a link to your site map off the home page, the site map would usually be a PR3 and any page off the site map that isn't linked from the home page would be a PR2, right? And, Google doesn't list a PR 2. Is that right?

    Google has found my site map and it is in the directory as well as all my category pages but none of the individual product pages are listed.
    Comments are opinion unless otherwise noted. Remember, pillage first. Then burn. Half of all people in the world have IQs under 100. You best learn to trust ol' SSanf!

  16. #16
    Not Verif-Lidated infoTim's Avatar
    Join Date
    January 18th, 2005
    Location
    Sunny Florida
    Posts
    1,021
    Quote Originally Posted by infoTim
    Bye-bye site map...hopefully things correct themselves back to where they were.
    Just as a followup, after a few weeks of having the site totally dissed by gbot (not even crawling the home page), I revised my sitemap generator and re-submitted. Sure enough, I went from 6 listings to 286,000 overnight.

    Lessons learned:

    1. don't screw up your site map
    2. even if #1 happens, don't delete your site map from Google Sitemaps
    3. I jump thru hoops for G and get better traffic from MSN anyway. :-)
    Tim
    consultant by day, affiliate by night

  17. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Google Sitemaps
    By kbrock in forum Search Engine Optimization
    Replies: 4
    Last Post: February 20th, 2007, 11:53 PM
  2. Google Sitemaps: Visited it lately?
    By Herb ԿԬ in forum Search Engine Optimization
    Replies: 1
    Last Post: October 13th, 2005, 04:39 AM
  3. Google Sitemaps
    By jrod805 in forum Search Engine Optimization
    Replies: 104
    Last Post: July 3rd, 2005, 12:52 AM
  4. Google Sitemaps and Subdomains
    By Zoe2 in forum Search Engine Optimization
    Replies: 5
    Last Post: June 26th, 2005, 10:24 AM
  5. Google Sitemaps
    By jrod805 in forum Midnight Cafe'
    Replies: 62
    Last Post: June 10th, 2005, 09:31 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •