[kse]

I wondering how most people who have datafeed sites update their sites. I NOT looking for detail information since I already doing it but more the location of my files (code & txt files) of if I doing this all wrong.

Here is what I do:

1) I Ftp the zip datafeed file to my workstation from the network server or one of my host that I provided to the network.

2) I unzip the txt file to my workstation

3) Run code (asp or php) on my workstation that:
a. Reads a record in the txt file (modifies it based on my rules if needed)
b. Then it writes the record to my mysql database located on my host (Temp Table)
c. Then reads the next record and so on until all of the records are read and written to my mysql database on my host (one by one).

4) Then I overwrite my master table with my Temp table.

I am assuming my process needs to be changed, so I wondering how most people handle datafeed files they get from the networks.

Thanks
Kevin……….

[Chuck Hamrick]

Kevin, I think datafeeders play it close to the cuff! I can tell you that in some of the segments I work with datafeeders are the top dogs. If your feed isn't setup well they won't use it.

[kse]

Thanks Chuck, My sites are working well however I think my process for updating my feeds may be a bit behind the times. I was just wondering how other affiliates updata their database.
I sure their are some affiliates here who spend alot less time and effert updating their databases.

Kevin.......

[superCool]

Hey kse, that's about what superCool does at this point too, but looking to improve. have you automated your ftp and unzip process? superCool did that recently and it helps quite a bit. nothing fancy, and superCool still submits it by hand, but one click and all files are downloaded, unzipped and ready to go (not instantly of course ). you could easily add a few lines of code to submit your own processing jobs too. that would at least make it all happen pretty much automatically while you work on other things.

good luck

[kse]

Originally Posted by superCool

Hey kse, that's about what superCool does at this point too, but looking to improve. have you automated your ftp and unzip process? superCool did that recently and it helps quite a bit. nothing fancy, and superCool still submits it by hand, but one click and all files are downloaded, unzipped and ready to go (not instantly of course ). you could easily add a few lines of code to submit your own processing jobs too. that would at least make it all happen pretty much automatically while you work on other things.

good luck

You are one (or two) steps ahead of me, I did not even know you could automate downloading and unzipping in a batch file. I guess now I have to figure out how I go about doing this.

Thank you!!
Kevin..........

[kse]

Also superCool do you download the feed to your workstation or do you download to you server/host and update your database to their (Host)??

[superCool]

superCool downloads to workstation. will post some code this weekend.

[kse]

Originally Posted by superCool

superCool downloads to workstation. will post some code this weekend.

Thanks looking forward to see how your doing this.

[Witzer]

Originally Posted by superCool

superCool downloads to workstation. will post some code this weekend.

Count me in....Please....
I need all of the help that I can get.

[superCool]

here are the main parts of the script. this code pulls all the feeds out of one CJ subscription folder and unzips them. superCool has a script that will pull files from all his CJ subscriptions, plus GAN and SAS. it's basically just repeating this code several times.

superCool adapted this code from several places on the net. some probably came from the mobilebadboy sas feed thread here on ABW. hopefully you can adapt this to meet your needs. it saves some time for superCool. he kicks off the download job and works on something else for an hour or so. then all the files are ready for processing.

good luck - no warranty

PHP Code:

// connect to cj feed server
// get into one of the subscription folders
// loop through all files - ftp to C drive / unzip to same folder
// 1111111 is ftp user, 22222222 is ftp password, 33333 is subscription folder name

    //** Pull CJ Files
    $conn = ftp_connect("datatransfer.cj.com");
    ftp_login($conn, "1111111", "22222222");
    ftp_pasv($conn, true);
    ftp_chdir($conn, "/outgoing/productcatalog/33333");

    $result = ftp_nlist($conn, ".");

    if(is_array($result)) {
        foreach($result as $line) {
            echo '<br>FTP CJ File: ' . $line;
            ftp_get($conn, 'C:\\folder1\\folder2\\' . $line, $line, FTP_BINARY);
            unZipDriver('C:\\folder1\\folder2\\', $line);
        }
    } else {
        echo "No data returned.";
    }

    ftp_close($conn);



//*****************************************
function unZipDriver($dir, $file) {

    $path_parts = pathinfo($file);
    $ext = $path_parts['extension'];
 
    if($ext == 'gz') {
        unZipGz($dir . $file);
    }
    if($ext == 'zip') {
        unZipZip($dir . $file, $dir);
    }

}

//*****************************************
function unZipGz($inFile) {

    global $totalZips;
    
    // Raising this value may increase performance
    $buffer_size = 4096; // read 4kb at a time
    $out_inFile = str_replace('.gz', '', $inFile);

    // Open our files (in binary mode)
    $file = gzopen($inFile, 'rb');
    $out_file = fopen($out_inFile, 'wb');

    // Keep repeating until the end of the input file
    while(!gzeof($file)) {
        // Read buffer-size bytes
        // Both fwrite and gzread and binary-safe
        fwrite($out_file, gzread($file, $buffer_size));
    }

    // Files are done, close files
    fclose($out_file);
    gzclose($file);
    
    $totalZips++;
    echo '<br>' . $inFile . ' unzipGz complete';
 
}

//*****************************************
function unZipZip($inFile, $outDir) {

    global $totalZips;

     $zip = new ZipArchive;
     $res = $zip->open($inFile);
     if ($res === TRUE) {
         $zip->extractTo($outDir);
         $zip->close();
         echo '<br>' . $inFile . ' unZip complete';
         $totalZips++;
     } else {
         echo '<br>' . $inFile . ' failed';
     }

} 


[kse]

Cool !!!

I took your code and in 30 seconds I have downloaded & unzipped 3 of my feeds from CJ!!!!

I may have a few questions when I set this up for GAN, SAS & LS.

Thank you so much.

Kevin....................

[beachcom]

Don't let Supercool's good looks fool you. He's DA MAN!

[kse]

Got it working for SAS & CJ so far, I guessing this will not work for LS since they use .qz.lmp ????

[superCool]

Glad you got it working kse. Hope it saves you some time. superCool doesn't use LS feeds, but it would probably just be a new function called from the unZipDriver function. You could probably find some code online that would drop into a new function.

For CJ if you have several subscription folders you could also remove the subscription folder from that first folder name, then loop through folders and files to pull them all at once.

have fun with your feeds!

[Witzer]

Originally Posted by beachcom

Don't let Supercool's good looks fool you. He's DA MAN!

Agree

[superCool]

forgot one detail - if your job keeps timing out you can add something like this to the top

set_time_limit(10000);
ini_set('memory_limit', '500M');

[kse]

Thanks again, If I get LS figured out I will post the code here as well.

[kse]

SuperCool,
I thought I had SAS working but Its not working for me anymore can you have a look at this code (Userid/Password was changed and the remainder of the code was not changed):

$conn = ftp_connect("datafeeds.shareasale.com");
ftp_login($conn, "1111111", "22222222");
ftp_pasv($conn, true);
ftp_chdir($conn, "/3928");

$result = ftp_nlist($conn, ".");

if(is_array($result)) {
foreach($result as $line) {
echo '<br>FTP SAS File: ' . $line;
ftp_get($conn, 'C:\\Inetpub\\wwwroot\\Datafeeds\\' . $line, $line, FTP_BINARY);
unZipDriver('C:\\Inetpub\\wwwroot\\Datafeeds\\', $line);
}
} else {
echo "No data returned.";
}

ftp_close($conn);

When I run the page the output shows:

FTP SAS File: 3928

However no files get written to my workstation (Note GAN Datafiles get copied to this location with any problems)

I also noticed that in the SAS Directory for /3928 their are three files: 3928.zip, 3928.txt.gz & 3928.txt

Any ideas what going on???

Thank you
Kevin.........

[superCool]

for SAS it might be a little different since they have a folder for each program and multiple files in the folder (right? actually not that familiar with SAS feeds). Here is superCool's SAS script. He adds each merchant to this list and then the script includes the merchant name into the file name. This helps superCool who doesn't like working with mystery files like 11111.txt.gz. This script turns that into MerchantName-11111.txt.gz which is easier for poor old superCool to deal with. Maybe this code will help. It's not as nice as just pointing to a folder and saying "go get it", but adding the name is nice.

Or you could probably use yours and just incorporate the file extension bit from the code below. you'll only need to pull one of the files.

PHP Code:

$merchants   = array( 
    array( 'merchNo'  => '1111', 'merchName' => 'MerchantOneName' ),
    array( 'merchNo'  => '2222', 'merchName' => 'MerchantTwoName' ),
    array( 'merchNo'  => '3333', 'merchName' => 'MerchantThreeName' ),
);
$file_ext = ".txt.gz";

$ftp_server = "datafeeds.shareasale.com"; // Do not change this
$ftp_user_name = "uuuuuuu"; // Replace user_name with your SAS login name
$ftp_user_pass = "ppppppp"; // Replace user_pass with your SAS password

$conn_id = ftp_connect($ftp_server);
$login_result = ftp_login($conn_id, $ftp_user_name, $ftp_user_pass);
if ((!$conn_id) || (!$login_result)) {
     echo "<br>FTP connection has failed to $ftp_server as $ftp_user_name.<br />";
     exit;
} else {
     echo "<br>Connected to $ftp_server successfully as $ftp_user_name.<br />";
}

ftp_pasv($conn_id, true);

for($i = 0 ; $i < sizeof($merchants); $i++) {
    $merchant  = $merchants[$i];
    $merchNo  = $merchant['merchNo'];
    $merchName = $merchant['merchName'];

    $source_file = "/".$merchNo."/".$merchNo.$file_ext; 
    $destination_file = "C:\\folder1\\folder2\\" . $merchName . "-" . $merchNo . $file_ext;

    echo '<br><br>start ' . $merchName;
    
    $download = ftp_get($conn_id, $destination_file, $source_file, FTP_BINARY);
    if (!$download) {
         echo "<br>FTP download has failed for " . $merchName;
    } else {
         echo "<br>Downloaded " . $merchName;
         unZipDriver("C:\\folder1\\folder2\\", $merchName . "-" . $merchNo . $file_ext);
    }
}

ftp_close($conn_id); 

echo '<br><br>Done with SAS FTP'; 


[kse]

Thanks, but I still not having any luck with downloading SAS Feeds. I wondering do I need to give SAS my IP or if something on my workstation is blocking the download (but it works for CJ)????????? Anyway SuperCool thanks for all your help, but it most be something on my end. I will try it on another workstation tomorrow.

Here is what get displayed when I run the code:

Connected to datafeeds.shareasale.com successfully as userid
start MyMerchant
FTP download has failed for MyMerchant
Done with SAS FTP


Kevin.........

[superCool]

hmmm, don't know what to tell you. never had issues like that and didn't have to tell SAS anything. you do have FTP access for these merchants right? have you tried different files to make sure it's not just one bad file? have you checked out the mobilebadboy thread - How to FTP and Import SAS datafeeds ? seems like there were a few people having trouble and they got it figured out.

Good luck

[kse]

Thanks, I will have a look at that thread. I can FTP to SAS with FileZilla and download the file without any problems so I think that ok. And I have tried three different merchants.

Anyway its working for CJ & GAN so that 90% of my feeds anyway so you have been a great help. It just bugs me why its not working for SAS. However as I said I sure it work on Saturday at least once!!!

[brentwz]

Where do you run this PHP file from?

Run from a form on a webpage you've setup?

Or is it a script file you access from within your FTP program, & you keep this script somewhere close at hand on your PC?

Or is it cPanel functionality stuff that I've yet to find out about?

Thanks, folks, looking forward to the answer!

[superCool]

if you have php set up on your home computer or on a server, you can run this inside a php file - such as download.php . you wrap the code with <?php at the top and ?> at the bottom. if you're not familiar with php you can get an introduction online to see how it works. find something like a "hello world" example and get it running. then insert this code to see if you can get it working.

good luck

[Jason - CarouselChecks]

I wrote this in PHP as it is what I am most fluent in. PHP is by no means the best/only way. This code is on the same server as the site it feeds. The site(s) it feeds is a spare-time project where I can play with some Ideas I get from time to time.

I used CURL. CURL can retrieve the file via ftp but I did something a bit different just because I could.

I used CURL to "log" into SAS and store my cookie in a file. This way it looks like I logged in from a web browser. Then I was able to directly request the zipped datafeed file by url since I already knew the url.

I saved the zipped file to a SAS datafeed directory on the server as sort of a cache. I was a bit leery of logging into SAS this way too many times too quickly.

I unzipped the file and looped through the lines and updated a SAS table in the database based on certain rules.

This is the gist of what I did but there are may more particulars.