Results 1 to 7 of 7
  1. #1
    Newbie
    Join Date
    July 28th, 2011
    Posts
    5
    Datafeed database structure
    Hello everyone,

    I have been lurking around here for awhile reading everything I can find on datafeeds. I have a lot better grasp on it now and have started writing code for my webapp. I have some question with database structures.

    What would be the best way to go about this? I am having a hard time getting it organized in the right way because I have multiple datafeeds that have some of the same products. Should I make one database table just to have all the product info? And then have another with info about the merchant? I am really confused on this whole part and a big reason why I am stuck. If I do it this way where would I put all the info about affiliate buy links for each merchant and all that, this whole thing may have me thinking more than I should. Can someone point me in the right direction?

  2. #2
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    I make a new folder and database for each category.

    If I have a merchants with a feed that is thousands of products I make a folder and database just for that one merchant.

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  3. #3
    Newbie
    Join Date
    July 28th, 2011
    Posts
    5
    Hmm. I am trying to build a review site with one page for each product. Then on that page I would like to list all the merchants that carry that product with price information. This will also include the affiliate link to the product page on the merchants website. Would your solution work for something like this?

    For the categories I am going to create my own structure and create a little utility where I can map each merchants category nodes to my own. This may be tedious but I don't see any other way to do it right now. I would only have to do it once per merchant.

    This is what I am thinkin anyway I am just not sure how I would go about making a database structure. I think the way I am trying to do it I would have to incorporate a master product table. I am just unsure of how to link the products in the master product table with the merchants that carry each product.

    Am I on the right track?

  4. #4
    Moderator BurgerBoy's Avatar
    Join Date
    January 18th, 2005
    Location
    jacked by sylon www.sylonddos.weebly.com
    Posts
    9,618
    I use Price Tapestry to build my sites. It builds price comparison sites and will list each merchant you have for a certain prtoduct and the price at each merchant.

    Affiliate Datafeed Price Comparison Script | Price Tapestry

    Vietnam Veteran 1966-1970 USASA
    ABW Forum Rules - Advertise At ABW

  5. #5
    Member soda's Avatar
    Join Date
    March 26th, 2010
    Location
    Irvine, CA
    Posts
    36
    If you're using an SQL type of database than I can suggest this setup:

    PRODUCTS
    Table with all products from all your data feeds: product id, merchant id, name, brand, description, keywords, merchant category, your category id, clickthrough URL, impression URL, image URL, prices, in stock, special, proto product ID (see below) etc.
    Suggestion: besides an integer ID have a hash ID (e.g. MD5 hash of merchant ID and product SKU -- in theory this should produce predictable unique ID's for all products). This should be the ID other tables use, instead of primary integer ID. Why? Because if one day you delete all products, and re-add them, these IDs will remain same while primary integer ID's will most likely change.

    PROTOTYPE PRODUCTS
    These are master / parent (I prefer prototype) products imported from the main PRODUCTS table. When you have same product in feeds from multiple merchants, you will have only 1 proto product, which will contain hash IDs of its children (duplicates from different feeds), while children in the PRODUCTS table will refer to their respective proto product. Make sure that your proto products have best descriptions and images from the set of children products. Each product in PRODUCTS table has to have a corresponding proto product.
    Other benefit of this is quicker queries if you want to show meta product info on the page (e.g. how many merchants carry this item, min, max, average prices etc).

    CATEGORIES
    This table will contain your categories / classes / taxonomy. I don't know what kind of structure you're using so I will not comment about this.

    MERCHANTS
    This is mostly for the back end use (merchant name, logo, your account #, data feed file location, last data feed import timestamp etc). On the front end you will likely use merchant name and logo.

    BRANDS
    You would probably want to have a master table for brands where you would have normalized brands info like name, name variations (they have to map brand in product feed to the actual brand, because often merchants use different names for same brand, e.g. ABC and ABC INC). You can include some meta data such as brand description, logo URL, number of products of this brand etc.

    PRODUCT IMAGES
    You will need this if you end up downloading and re-sizing images from your merchants. If you decide to do this, it will be a pretty long and resource intensive process. If the only place where you keep images info is PRODUCTS and PROTOTYPE PRODUCTS tables, once, for some reason, you truncate these tables, you will have hard times knowing which images you already have, and would probably have to repeat process of loading and re-sizing images again. If you keep them in a separate table and reference product hash IDs (which would still be same if you completely delete and re-import products), you can quickly lookup images in this table. Will save you lots of time.

    EXCLUDED KEYWORDS
    Keep keywords based on which you will exclude some products on the first pass of processing your data feeds.

    PS you might consider using a no-SQL database like MongoDB to keep denormalized product data. Best place to use this is for the prototype products, as for each parent product you can have all the info you have including children products, multiple images, reviews etc).

  6. Thanks From:

  7. #6
    Newbie
    Join Date
    July 28th, 2011
    Posts
    5
    Smile
    Thanks soda. That was exactly what I was looking for. Sorry I didn't get back to you sooner. Been busy with other projects.

    Thanks again.

  8. #7
    Newbie
    Join Date
    July 28th, 2011
    Posts
    5
    Multiple datafeeds with same products...
    I have two datafeeds and they contain some products. HELP...

    How do you handle this to make one master database of products? SKUs are not the same, Manufacturer IDs are either missing or not the same, descriptions are never identical. How is this done? Hopefully someone has a better idea than I do because the only way I can see doing it so that it is 100% right is doing it manually one by one.

  9. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Pepperjam changed its datafeed structure
    By oranges in forum eBay Enterprise Affiliate Network
    Replies: 4
    Last Post: July 20th, 2014, 05:53 PM
  2. How to put CJ datafeed into database?
    By oxplode in forum Commission Junction - CJ
    Replies: 0
    Last Post: April 20th, 2014, 03:03 PM
  3. Betty Mills Datafeed Structure Notice
    By Victor Hanna in forum BettyMills
    Replies: 1
    Last Post: January 26th, 2006, 03:17 PM
  4. SAS datafeed structure (Technical)
    By NoviL in forum ShareASale - SAS
    Replies: 2
    Last Post: June 29th, 2004, 09:05 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •