Results 1 to 10 of 10
  1. #1
    Newbie
    Join Date
    March 6th, 2014
    Posts
    7
    How to handle multiple datafeeds?
    Hello,

    Now, I've set it up so commission junction sends me datafeeds for a variety of merchants. Problem is I don't know how to handle them all and create an efficient database. Should I have a different table for each merchant and call the appropriate table to lookup the product information? Should I have 1 table with absolutely everything in it? (I assume that would create a very slow website), or something else? My website needs to be able to search through the database to find the appropriate affiliate url + product thumbnail. I plan on using various affiliate networks and quite a few merchants.

    I've looked at some of the scripts but they don't really look that customizable, am I wrong? I have a website build already and just need it to convert product url's into affiliate url's using the datafeed and to also grab the associate product info.

    I've no experience using datafeeds before, so just looking for some implementation advice. I've basic SQL + PHP experience. Any help will be great!
    Last edited by oxplode; June 4th, 2014 at 11:58 AM.

  2. #2
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    If they're set up right, even very, very large tables can be quickly and easily searched. The database structure and indexes make a big difference.

    I standardize and combine into a single table, and I suspect that most others who work with datafeeds do the same.

    If you're modifying a live table, one key is to do a multi-step process to make sure you handle any "drops":

    1) Change the status on all active products in the table from that merchant to "updating".
    2) Process the feed, changing the status on each product in the feed to "updated" or "added".
    3) Remove (or flag as "removed") all of the products for that merchant that are still set to "updating".

    Another thing you need to consider is matching up identical products that are available from multiple merchants.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  3. #3
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    I am similar to Michael except I have 3 tables:

    Temp: The table I update (by scripts)
    Master: The live copy of my Temp table (Table used by my site)
    Backup: The prev version of my Master table

    So the steps I follow are:

    1) Make the updates to my Temp table
    2) Overwrite Backup with current Master table
    3) Overwrite Master with current Temp table

    Some may consider this over kill but it has saved my butt many times over the years

  4. Thanks From:

  5. #4
    Newbie
    Join Date
    March 6th, 2014
    Posts
    7
    So do you guys use cron jobs to automate all of this? I ask because it looks like I would need to update many feeds every 24h at least which would take a lot of time. I'm considering just hiring someone to implement this. How much can I expect to pay for such a job?

    Cheers for the quick replies!

  6. #5
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    I only need to update weekly, so I run ASP code (not a cron job) to update my database. My scripts are totally automated so it only take about 30 min a week to do the update.
    Last edited by kse; June 4th, 2014 at 05:34 PM.
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  7. #6
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    I make extensive use of cron jobs for automation.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  8. #7
    ...and a Pirate's heart. Convergence's Avatar
    Join Date
    June 24th, 2005
    Posts
    6,918
    Quote Originally Posted by MichaelColey View Post
    I make extensive use of cron jobs for automation.
    +1

    We run updates pretty much non-stop for about 18 hours/day. CRON jobs rock!...
    Salty kisses, Sandy toes, and a Pirate's heart...

  9. #8
    Newbie
    Join Date
    January 17th, 2014
    Posts
    18
    Quote Originally Posted by oxplode View Post
    ......

    I've no experience using datafeeds before, so just looking for some implementation advice. I've basic SQL + PHP experience. Any help will be great!
    In common : SQL + PHP experience is very very handy in our business but :

    [1] if you're interested in a lifetime working with php, keep up all the changes and risks, go ahead with programming in php 5.0,5.2 5.3 5.4 5.5 and so on

    [2] if your goal is affiliate marketing as a publisher, take for instance Pricetapestry, in Wordpress modus,
    a. as engine for managing the datafeeds and make your display by phpexec-plugin
    b. as a total plugin using standard display, using shortcodes and so on
    and concentrate on marketing

    PT needs about 300 pound - one time. I use it for years, software and - included - support.

    But yes, allways cronjobs !! daily or weekly

  10. #9
    Newbie
    Join Date
    March 6th, 2014
    Posts
    7
    You guys are so helpful, really great stuff! That pricetapestry does looks good so I've email the admin a few questions about it although it's not exactly what I want, it looks like it handles datafeeds well and hopefully I can reprogram some bits to suit.

    I've only viewed a few datafeeds but is it possible to search them by URL? e.g. a user enters a url, the database is then searched for that url and the affiliate url is then returned?

    I thought this would be possible but not 100% sure now. I'll give an example.

    My CJ datafeed for this particular product: ww.ebooks.com/100277/the-civilized-market/alexander-ivan/



    but the only similarity between the two, is the domain name and product number. Perhaps I should return results based on those 2 factors? Do all merchants use product numbers?
    Last edited by Chuck Hamrick; June 5th, 2014 at 11:54 AM. Reason: Please refrain from adding affiliate links to your posts. I will allow your datafeed link to stand.

  11. #10
    Newbie
    Join Date
    January 17th, 2014
    Posts
    18
    First, David, who 's the guy behind Pricetapestry, is always willing to customize, and there is a forum.

    If you want to search by yourself, it depends on the datafeeds you are receiving.
    As an example, my vacation-website holds 28 merchants / feeds and the products-table holds this title:

    id merchant filename name description image_url buy_url price oldprice category subcategory country region city other brand rating reviews search_name normalised_name original_name voucher_code dupe_hash

    Most double records are filtered out.

    You can search one column, or a few, full text and if necessary : buy url or image url, using for instance the plugin php-exec, or take the easy way of using shortcodes made by Pricetapestry.

    So searching on buy url .... it's possible.

    In my point of view rather important in case of shoes/clothes and vacation etc :
    using filters.

    Do you want : yellow or metallic shoes ? wich size ? do you want search on prices, more than, less than, specific merchant ...... it's mostly all possible, but take my advice, try to keep it simple.

  12. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Webgains Launches Exclusive Multiple Datafeeds Feature
    By Webgains USA in forum Webgains.com
    Replies: 0
    Last Post: December 8th, 2010, 04:17 PM
  2. Multiple products, multiple merchants
    By zendango in forum ShareASale - SAS
    Replies: 4
    Last Post: January 22nd, 2010, 07:28 AM
  3. Multiple products from multiple merchants?
    By JamesS in forum PopShops
    Replies: 3
    Last Post: October 2nd, 2007, 02:21 AM
  4. Replies: 5
    Last Post: December 6th, 2006, 02:42 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •