Results 1 to 6 of 6
  1. #1
    Full Member markschok's Avatar
    Join Date
    January 17th, 2005
    Once you have a feed up and running how do you keep your database up to date ?

    any tips or clues on this would be cool.
    ie. How to check your database against the latest datafeed for expired or new products ?

    have a script to auto update ?
    shop/catalog admin ?
    link checker ?
    dump and reload the latest feed ?
    other ?

    I need to do this myself soon and would like to dump everything and reload the latest feed but I have several feeds combined in 1 database so it's looking a bit tricky now, I should have thought about this before I started to mess with the feeds.

  2. #2
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Mansfield, TX
    I have a status field in the table.

    Before I load the feed, I flag all items for that merchant as "pending delete".

    As I update items or insert new items, I change the flag to "valid" (or "updated" and "new").

    After I've processed the entire file, I check to see how many pending delete items I have for that merchant. If it's a reasonable percent, I either delete the "pending delete" items for that merchant or flag them as "deleted". I do this because I've seen times where I get a "partial" datafeed file and I don't want to delete the missing items in that case.

    My routines that use the datafeed tables check the status flag. They still display "pending delete" items but not the truly "deleted" items.

    Michael Coley

  3. #3
    Join Date
    January 17th, 2005
    This may seem like a dumb question, but why wouldn't you just overwrite the entire database with the new feed, deleting all items in the old feed? I guess maybe you have massaged your data already and don't want to have to re-do all that work?


  4. #4
    Pimp Duck popdawg's Avatar
    Join Date
    January 18th, 2005
    Take off eh?
    For me its a matter of bandwidth and downtime.
    My databases are getting huge with the number of products. Also, I don't like my sites being down for more that 1 1/8th of a millionth of a second let alone 30 minutes plus.
    --- that may be part of the reason CJ downtime all the time drives me nuts ---
    Also, like you said, there is a fair amount of processing I do on my databases that complicate things as well.
    My 2c any ways

    Must Build More in 2004
    Game on!!!! - The DawgFather

  5. #5
    ABW Ambassador
    Join Date
    January 17th, 2005
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>why wouldn't you just overwrite the entire database with the new feed<HR></BLOCKQUOTE>
    Exactly what I do. I use the database to generate the pages & menus dynamically and do not want to have to touch the data. Any specials on the pages are generated dynamically also.
    - Import feed into Widgets_new.mdb
    - Upload to server
    - Rename old file Widgets_save.mdb
    - Rename new file Widgets.mdb
    Total down time is the amount of time it take to rename two files on server. Check the site, click through a few pages & if OK, delete Widgets_save.mdb.


  6. #6
    Join Date
    January 17th, 2005
    I have a cron job setup to:

    1) get the datafeed via ftp or http
    2) parse it
    3) convert it to my own search engine database format.

    I run this for all of my datafeeds every night so it picks up new and drops old products daily.

    I also syndicate products to my domains where I have the shopping sites setup. This allows me to have a central site of datafeeds and distributed stores. I have a big server with lots of disk space for my databases and small accounts for the stores.

    I kind of think of it like a central warehouse feeding the datafeeds to wherever they are needed.


  7. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Website Starting Clean Slate: Any ToDo's Before Wipe Clean
    By Uncle Rico in forum Midnight Cafe'
    Replies: 7
    Last Post: September 2nd, 2012, 12:55 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts