Results 1 to 4 of 4
  1. #1
    Affiliate sdanpo's Avatar
    Join Date
    May 4th, 2012
    Posts
    65
    Feeds update
    Hi,

    I am now implementing a feed parser.
    First step is implementing an FTP manager which is a piece of SW which communicated with the various FTP servers , retrieves the feeds and then passes them to the parser.

    I would like to know if you could share what is the best way to do it in incremental manner ?

    How did you implement it so that each time you approach the FTP - you will take only the deltas from the last time you approached ?

    My question refers to all networks (though some of them don't work with FTP , but rss or other implementations...):
    Avantlink,
    CJ,
    SAS,
    GAN,
    ShopZilla,
    LinkShare,
    PJX

    Thanks

    Dan

  2. #2
    AffiliateMan
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    19
    Quote Originally Posted by sdanpo View Post
    Hi,

    I am now implementing a feed parser.
    First step is implementing an FTP manager which is a piece of SW which communicated with the various FTP servers , retrieves the feeds and then passes them to the parser.

    I would like to know if you could share what is the best way to do it in incremental manner ?

    How did you implement it so that each time you approach the FTP - you will take only the deltas from the last time you approached ?

    My question refers to all networks (though some of them don't work with FTP , but rss or other implementations...):
    Avantlink,
    CJ,
    SAS,
    GAN,
    ShopZilla,
    LinkShare,
    PJX
    I wrote a blog article about how I Process Shareasale Data Feeds in PHP.

    The basic should work for other sources of data feeds such as the ones you've listed.

    Basically, I have my server check how old the lastest copy of the feed is on my server. Then, it uses FTP to get the update if the age of the existing file is older than a specified number of days. Of course, if pricing is important, you'd probably want to get the feed every day.

    An improvement to the method I use would be to check the timestamp of the feed on the remote server and download it only if it is newer than the one on your server. You could use PHP's ftp_mdtm function to get the last modified time of a data feed file and download it if it's newer than the one on your server.

    Some FTP servers, however, don't support returning the last modified time of a file.

    So, I'd recommend storing the size of the file as returned by the PHP ftp_size function when you download it. Then, periodically, use ftp_size to check the size on the remote server. When the size changes, download the newer version of the file and save the ftp_size value. The reason you have to save the value returned by ftp_size (instead of getting the size of the downloaded from on your server) is that the size of the file on your server may be different than the size on the remote server because it may be stored with different line endings or character encodings.

    -- Roger
    Last edited by Chuck Hamrick; December 3rd, 2012 at 03:57 PM. Reason: Approved

  3. #3
    Affiliate sdanpo's Avatar
    Join Date
    May 4th, 2012
    Posts
    65
    Roger, Thanks
    Will take your advice and implement.

    Thanks,

    Dan

  4. #4
    AffiliateMan
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    19
    "The basic should work for other sources ..."

    I meant to say ... the basic idea should work for other sources ....

    You should be able to find many other examples of how to implement FTP in PHP or whatever language you are using. Most languages provide libraries that make it pretty easy to do.

    -- Roger

  5. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Merchants, please update data feeds
    By layla in forum ShareASale - SAS
    Replies: 9
    Last Post: December 9th, 2008, 02:07 PM
  2. Replies: 6
    Last Post: August 7th, 2006, 10:15 AM
  3. Bill Bam Collectibles Product Feeds Major Update
    By Chris - AMWSO in forum ShareASale - SAS
    Replies: 0
    Last Post: August 30th, 2004, 07:52 PM
  4. Update on New Data Feeds?
    By woop01 in forum Commission Junction - CJ
    Replies: 2
    Last Post: March 25th, 2003, 01:24 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •