Page 1 of 2 12 LastLast
Results 1 to 25 of 27
  1. #1
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Question How are you updating you datafeed sites?
    I wondering how most people who have datafeed sites update their sites. I NOT looking for detail information since I already doing it but more the location of my files (code & txt files) of if I doing this all wrong.

    Here is what I do:

    1) I Ftp the zip datafeed file to my workstation from the network server or one of my host that I provided to the network.

    2) I unzip the txt file to my workstation

    3) Run code (asp or php) on my workstation that:
    a. Reads a record in the txt file (modifies it based on my rules if needed)
    b. Then it writes the record to my mysql database located on my host (Temp Table)
    c. Then reads the next record and so on until all of the records are read and written to my mysql database on my host (one by one).

    4) Then I overwrite my master table with my Temp table.

    I am assuming my process needs to be changed, so I wondering how most people handle datafeed files they get from the networks.

    Thanks
    Kevin……….
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  2. #2
    OPM and Moderator Chuck Hamrick's Avatar
    Join Date
    April 5th, 2005
    Location
    Park City Utah
    Posts
    16,646
    Kevin, I think datafeeders play it close to the cuff! I can tell you that in some of the segments I work with datafeeders are the top dogs. If your feed isn't setup well they won't use it.

  3. #3
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Thanks Chuck, My sites are working well however I think my process for updating my feeds may be a bit behind the times. I was just wondering how other affiliates updata their database.
    I sure their are some affiliates here who spend alot less time and effert updating their databases.

    Kevin.......
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  4. #4
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    Location
    Texas
    Posts
    1,268
    Hey kse, that's about what superCool does at this point too, but looking to improve. have you automated your ftp and unzip process? superCool did that recently and it helps quite a bit. nothing fancy, and superCool still submits it by hand, but one click and all files are downloaded, unzipped and ready to go (not instantly of course ). you could easily add a few lines of code to submit your own processing jobs too. that would at least make it all happen pretty much automatically while you work on other things.

    good luck

  5. #5
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Quote Originally Posted by superCool View Post
    Hey kse, that's about what superCool does at this point too, but looking to improve. have you automated your ftp and unzip process? superCool did that recently and it helps quite a bit. nothing fancy, and superCool still submits it by hand, but one click and all files are downloaded, unzipped and ready to go (not instantly of course ). you could easily add a few lines of code to submit your own processing jobs too. that would at least make it all happen pretty much automatically while you work on other things.

    good luck
    You are one (or two) steps ahead of me, I did not even know you could automate downloading and unzipping in a batch file. I guess now I have to figure out how I go about doing this.

    Thank you!!
    Kevin..........
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  6. #6
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Also superCool do you download the feed to your workstation or do you download to you server/host and update your database to their (Host)??
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  7. #7
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    Location
    Texas
    Posts
    1,268
    superCool downloads to workstation. will post some code this weekend.

  8. #8
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Quote Originally Posted by superCool View Post
    superCool downloads to workstation. will post some code this weekend.
    Thanks looking forward to see how your doing this.
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  9. #9
    Tax Paying Member
    Join Date
    November 14th, 2005
    Location
    Chapel Hill, NC
    Posts
    2,119
    Quote Originally Posted by superCool View Post
    superCool downloads to workstation. will post some code this weekend.
    Count me in....Please....
    I need all of the help that I can get.
    You must climb this mountain. There is no elevator. ---- Don't stick your finger in the liquid nitrogen.
    Carolina China

  10. #10
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    Location
    Texas
    Posts
    1,268
    here are the main parts of the script. this code pulls all the feeds out of one CJ subscription folder and unzips them. superCool has a script that will pull files from all his CJ subscriptions, plus GAN and SAS. it's basically just repeating this code several times.

    superCool adapted this code from several places on the net. some probably came from the mobilebadboy sas feed thread here on ABW. hopefully you can adapt this to meet your needs. it saves some time for superCool. he kicks off the download job and works on something else for an hour or so. then all the files are ready for processing.

    good luck - no warranty

    PHP Code:
    // connect to cj feed server
    // get into one of the subscription folders
    // loop through all files - ftp to C drive / unzip to same folder
    // 1111111 is ftp user, 22222222 is ftp password, 33333 is subscription folder name

        //** Pull CJ Files
        
    $conn ftp_connect("datatransfer.cj.com");
        
    ftp_login($conn"1111111""22222222");
        
    ftp_pasv($conntrue);
        
    ftp_chdir($conn"/outgoing/productcatalog/33333");

        
    $result ftp_nlist($conn".");

        if(
    is_array($result)) {
            foreach(
    $result as $line) {
                echo 
    '<br>FTP CJ File: ' $line;
                
    ftp_get($conn'C:\\folder1\\folder2\\' $line$lineFTP_BINARY);
                
    unZipDriver('C:\\folder1\\folder2\\'$line);
            }
        } else {
            echo 
    "No data returned.";
        }

        
    ftp_close($conn);



    //*****************************************
    function unZipDriver($dir$file) {

        
    $path_parts pathinfo($file);
        
    $ext $path_parts['extension'];
     
        if(
    $ext == 'gz') {
            
    unZipGz($dir $file);
        }
        if(
    $ext == 'zip') {
            
    unZipZip($dir $file$dir);
        }

    }

    //*****************************************
    function unZipGz($inFile) {

        global 
    $totalZips;
        
        
    // Raising this value may increase performance
        
    $buffer_size 4096// read 4kb at a time
        
    $out_inFile str_replace('.gz'''$inFile);

        
    // Open our files (in binary mode)
        
    $file gzopen($inFile'rb');
        
    $out_file fopen($out_inFile'wb');

        
    // Keep repeating until the end of the input file
        
    while(!gzeof($file)) {
            
    // Read buffer-size bytes
            // Both fwrite and gzread and binary-safe
            
    fwrite($out_filegzread($file$buffer_size));
        }

        
    // Files are done, close files
        
    fclose($out_file);
        
    gzclose($file);
        
        
    $totalZips++;
        echo 
    '<br>' $inFile ' unzipGz complete';
     
    }

    //*****************************************
    function unZipZip($inFile$outDir) {

        global 
    $totalZips;

         
    $zip = new ZipArchive;
         
    $res $zip->open($inFile);
         if (
    $res === TRUE) {
             
    $zip->extractTo($outDir);
             
    $zip->close();
             echo 
    '<br>' $inFile ' unZip complete';
             
    $totalZips++;
         } else {
             echo 
    '<br>' $inFile ' failed';
         }


    Last edited by superCool; August 13th, 2011 at 12:07 AM.

  11. Thanks From:
    kse

  12. #11
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Cool !!!

    I took your code and in 30 seconds I have downloaded & unzipped 3 of my feeds from CJ!!!!

    I may have a few questions when I set this up for GAN, SAS & LS.

    Thank you so much.

    Kevin....................
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  13. #12
    ABW Ambassador beachcom's Avatar
    Join Date
    May 11th, 2006
    Location
    Myrtle Beach SC
    Posts
    723
    Don't let Supercool's good looks fool you. He's DA MAN!

  14. #13
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Got it working for SAS & CJ so far, I guessing this will not work for LS since they use .qz.lmp ????
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  15. #14
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    Location
    Texas
    Posts
    1,268
    Glad you got it working kse. Hope it saves you some time. superCool doesn't use LS feeds, but it would probably just be a new function called from the unZipDriver function. You could probably find some code online that would drop into a new function.

    For CJ if you have several subscription folders you could also remove the subscription folder from that first folder name, then loop through folders and files to pull them all at once.

    have fun with your feeds!

  16. #15
    Tax Paying Member
    Join Date
    November 14th, 2005
    Location
    Chapel Hill, NC
    Posts
    2,119
    Quote Originally Posted by beachcom View Post
    Don't let Supercool's good looks fool you. He's DA MAN!
    Agree
    You must climb this mountain. There is no elevator. ---- Don't stick your finger in the liquid nitrogen.
    Carolina China

  17. #16
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    Location
    Texas
    Posts
    1,268
    forgot one detail - if your job keeps timing out you can add something like this to the top

    set_time_limit(10000);
    ini_set('memory_limit', '500M');

  18. #17
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Thanks again, If I get LS figured out I will post the code here as well.
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  19. #18
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    SuperCool,
    I thought I had SAS working but Its not working for me anymore can you have a look at this code (Userid/Password was changed and the remainder of the code was not changed):

    $conn = ftp_connect("datafeeds.shareasale.com");
    ftp_login($conn, "1111111", "22222222");
    ftp_pasv($conn, true);
    ftp_chdir($conn, "/3928");

    $result = ftp_nlist($conn, ".");

    if(is_array($result)) {
    foreach($result as $line) {
    echo '<br>FTP SAS File: ' . $line;
    ftp_get($conn, 'C:\\Inetpub\\wwwroot\\Datafeeds\\' . $line, $line, FTP_BINARY);
    unZipDriver('C:\\Inetpub\\wwwroot\\Datafeeds\\', $line);
    }
    } else {
    echo "No data returned.";
    }

    ftp_close($conn);

    When I run the page the output shows:

    FTP SAS File: 3928

    However no files get written to my workstation (Note GAN Datafiles get copied to this location with any problems)

    I also noticed that in the SAS Directory for /3928 their are three files: 3928.zip, 3928.txt.gz & 3928.txt

    Any ideas what going on???

    Thank you
    Kevin.........
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  20. #19
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    Location
    Texas
    Posts
    1,268
    for SAS it might be a little different since they have a folder for each program and multiple files in the folder (right? actually not that familiar with SAS feeds). Here is superCool's SAS script. He adds each merchant to this list and then the script includes the merchant name into the file name. This helps superCool who doesn't like working with mystery files like 11111.txt.gz. This script turns that into MerchantName-11111.txt.gz which is easier for poor old superCool to deal with. Maybe this code will help. It's not as nice as just pointing to a folder and saying "go get it", but adding the name is nice.

    Or you could probably use yours and just incorporate the file extension bit from the code below. you'll only need to pull one of the files.

    PHP Code:
    $merchants   = array( 
        array( 
    'merchNo'  => '1111''merchName' => 'MerchantOneName' ),
        array( 
    'merchNo'  => '2222''merchName' => 'MerchantTwoName' ),
        array( 
    'merchNo'  => '3333''merchName' => 'MerchantThreeName' ),
    );
    $file_ext ".txt.gz";

    $ftp_server "datafeeds.shareasale.com"// Do not change this
    $ftp_user_name "uuuuuuu"// Replace user_name with your SAS login name
    $ftp_user_pass "ppppppp"// Replace user_pass with your SAS password

    $conn_id ftp_connect($ftp_server);
    $login_result ftp_login($conn_id$ftp_user_name$ftp_user_pass);
    if ((!
    $conn_id) || (!$login_result)) {
         echo 
    "<br>FTP connection has failed to $ftp_server as $ftp_user_name.<br />";
         exit;
    } else {
         echo 
    "<br>Connected to $ftp_server successfully as $ftp_user_name.<br />";
    }

    ftp_pasv($conn_idtrue);

    for(
    $i $i sizeof($merchants); $i++) {
        
    $merchant  $merchants[$i];
        
    $merchNo  $merchant['merchNo'];
        
    $merchName $merchant['merchName'];

        
    $source_file "/".$merchNo."/".$merchNo.$file_ext
        
    $destination_file "C:\\folder1\\folder2\\" $merchName "-" $merchNo $file_ext;

        echo 
    '<br><br>start ' $merchName;
        
        
    $download ftp_get($conn_id$destination_file$source_fileFTP_BINARY);
        if (!
    $download) {
             echo 
    "<br>FTP download has failed for " $merchName;
        } else {
             echo 
    "<br>Downloaded " $merchName;
             
    unZipDriver("C:\\folder1\\folder2\\"$merchName "-" $merchNo $file_ext);
        }
    }

    ftp_close($conn_id); 

    echo 
    '<br><br>Done with SAS FTP'

  21. #20
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Thanks, but I still not having any luck with downloading SAS Feeds. I wondering do I need to give SAS my IP or if something on my workstation is blocking the download (but it works for CJ)????????? Anyway SuperCool thanks for all your help, but it most be something on my end. I will try it on another workstation tomorrow.

    Here is what get displayed when I run the code:

    Connected to datafeeds.shareasale.com successfully as userid
    start MyMerchant
    FTP download has failed for MyMerchant
    Done with SAS FTP


    Kevin.........
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  22. #21
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    Location
    Texas
    Posts
    1,268
    hmmm, don't know what to tell you. never had issues like that and didn't have to tell SAS anything. you do have FTP access for these merchants right? have you tried different files to make sure it's not just one bad file? have you checked out the mobilebadboy thread - http://www.abestweb.com/forums/progr...light=datafeed ? seems like there were a few people having trouble and they got it figured out.

    Good luck

  23. #22
    ABW Ambassador kse's Avatar
    Join Date
    November 29th, 2005
    Posts
    2,511
    Thanks, I will have a look at that thread. I can FTP to SAS with FileZilla and download the file without any problems so I think that ok. And I have tried three different merchants.

    Anyway its working for CJ & GAN so that 90% of my feeds anyway so you have been a great help. It just bugs me why its not working for SAS. However as I said I sure it work on Saturday at least once!!!
    MERCHANTS: Start showing your coupons directly on your site, that way your shoppers will stop leaving your site looking for them!! If not then remove your Coupon Box!!

  24. #23
    Newbie brentwz's Avatar
    Join Date
    August 21st, 2011
    Location
    Buderim, Sunshine Coast, Qld, Australia
    Posts
    11
    Question Pardon me for asking the obvious but...
    Where do you run this PHP file from?

    Run from a form on a webpage you've setup?

    Or is it a script file you access from within your FTP program, & you keep this script somewhere close at hand on your PC?

    Or is it cPanel functionality stuff that I've yet to find out about?

    Thanks, folks, looking forward to the answer!

  25. #24
    ABW Ambassador superCool's Avatar
    Join Date
    April 23rd, 2008
    Location
    Texas
    Posts
    1,268
    if you have php set up on your home computer or on a server, you can run this inside a php file - such as download.php . you wrap the code with <?php at the top and ?> at the bottom. if you're not familiar with php you can get an introduction online to see how it works. find something like a "hello world" example and get it running. then insert this code to see if you can get it working.

    good luck

  26. #25
    Newbie
    Join Date
    October 4th, 2011
    Location
    Chicago, Illinois
    Posts
    6
    I wrote this in PHP as it is what I am most fluent in. PHP is by no means the best/only way. This code is on the same server as the site it feeds. The site(s) it feeds is a spare-time project where I can play with some Ideas I get from time to time.

    I used CURL. CURL can retrieve the file via ftp but I did something a bit different just because I could.

    I used CURL to "log" into SAS and store my cookie in a file. This way it looks like I logged in from a web browser. Then I was able to directly request the zipped datafeed file by url since I already knew the url.

    I saved the zipped file to a SAS datafeed directory on the server as sort of a cache. I was a bit leery of logging into SAS this way too many times too quickly.

    I unzipped the file and looped through the lines and updated a SAS table in the database based on certain rules.

    This is the gist of what I did but there are may more particulars.

+ Reply to Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Featured: If you aren't updating your datafeed every day....
    By isellstuff in forum Merchant Best Practices Forum
    Replies: 19
    Last Post: November 20th, 2013, 03:57 PM
  2. updating a site that i built from a datafeed with webmerge
    By hungirl in forum Programming / Datafeeds / Tools
    Replies: 4
    Last Post: February 17th, 2004, 10:58 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •