[chrisk]

I was trying to automate the downloading of the NEW linkshare feeds using a GUI based FTP program on a windows machine. It seems like all the GUI based FTP programs would retrieve a 0 byte file instead of the data file.

I happened to have a copy of Dynu FTP sitting around (30 day trial available http://www.dynu.com/dynuftp.asp) so I wrote this script.

This script will log in using the DynuFTP component, get a directory list and download the NON-XML versions of the data files.

To use this script, downlod and install DynuFT. Then copy the code into a blank text file and give the file a .vbs extention. Next, right click the file and select "Open With Command Prompt"

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>
Set oFTP = Createobject("Dynu.FTP")
If oFTP.connect("aftp.linksynergy.com", "yourusername", "yourpassword")Then
WScript.Echo "CONNECTED"
oFTP.ExecuteCommand("NOOP")
strFileList = oFtp.GetFileList()
Dim FileArray
FileArray = Split(strFileList, VbCrLf)
iMax = ubound(FileArray)
for iCount = 0 to iMax-1
if instr(FileArray(iCount),".xml")&gt;0 then
' dont download the xml version
else
datafilename=left(FileArray(iCount),len(FileArray(iCount))-4)
Wscript.echo datafilename
i = oFtp.GetFile(cstr(datafilename),cstr("C:\" & datafilename) )
end if
next
oFTP.close()
WScript.Echo "FINISHED"
else
WScript.Echo "FAILED CONNECTION"
End If
Set oFTP = nothing
<HR></BLOCKQUOTE>

It might not be the best way, but it works well for me....

[dak142]

Interesting piece of software. Any chance you know of simular software out there that will do the same thing but in PHP rather than ASP?

[chrisk]

Most people on the Linux side write perl scripts to do it from what I understand. I am sure PHP probably has it built in also. Sorry I can't me more specific, my extertise is on the Windows side.

This script does not run from an ASP page - but it uses a component that 'can' be used on an ASP page. The script is sort of like an advanced batch file.

[micheck]

The variation for the UNIX/LINIX side of the house that works for me is as follows:

I create a file - I call it GETIT.SH containing:

#!/bin/bash

ftp -i -n &lt;&lt;HERE
open aftp.linksynergy.com
user USER_NAME PASSWORD
bin
get XXX1_YYYYYYY_mp.txt.gz
get XXX2_YYYYYYY_mp.txt.gz
get XXX3_YYYYYYY_mp.txt.gz
close
quit
HERE

I save this file in my home directory - I find that the directory structure is a bit beyond me so I use the UNIX command PWD which gives me the path to the directory where the file and I am. I also make the file an executable file by using the UNIX command CHMOD +X GETIT.SH.

I go into the CRON (timer) program in my cpanel and pick when I want the 'batch' file to run and in the blank for the 'script to run' I put the full path I got from PWD and /getit.sh

It runs for me - I hope I pointed out the gotchas so it will run for you. PM me if you'd like.

[chrisk]

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Charles_Michener:
The variation for the UNIX/LINIX side of the house that works for me is as follows:
<HR></BLOCKQUOTE>

It would be cool if someone could post a version that fetched all available files like my Windows version does. It's a pain maintaining the list


For windows folks, here is a page that tells you how to automate (free) the download if you don't mind specifying each file name in your batch file.
http://www.infoworld.com/cgi-bin/displayNew.pl?/livingst/990823bl.htm

[MichaelColey]

I was using a Unix solution called "lftp" that has a mirror option. The mirror option downloaded only the items that changed. Of course the "upgrade" broke that.

Has anyone found a Unix solution that only downloads the items that changed? I might have to dig through the Perl modules on CPAN to find an FTP module and then build something myself. I hate having to take time away from working on sites just to fix something that Linkshare "upgraded".

[chrisk]

Yeah, I used to do something similar on the windows site (mirroring) - I had FTP-Voyager on a 4am syncronization schedule... The reason I built the auto-downloader was to solve the issue with the linkshare 'upgrade'

[MichaelColey]

For anyone on a Unix platform, here's the script I developed. It was easier than I expected. It will log in, get a listing of all the files available, strip off the ".lmp" extensions, check the dates against files that have already been downloaded, download any that are new or have been changed, and set the file date to the date on the Linkshare FTP site. It relies on the ".lmp" file date being changed when a new files is available (which appears to happen). If a downloaded file is 0 length, the script skips it. You'll probably need to install "Net::FTP::Common". You might need to change the perl path in the first line. You'll need to change the user name, password, and paths. If you use the XML versions instead of the txt versions, change the match string to ".xml.gz".

#!/usr/local/bin/perl
 
use strict;
use Net::FTP::Common;
use Date::Manip;
 
my $user = "{change to your user name}";
my $password = "{change to your password}";
my $site = "aftp.linksynergy.com";
my $localdir = "/path/to/datafeed/directory/";
my $tempdir = "/path/to/temp/directory/";
my $matchstr = ".txt.gz";
 
my %ncfg = (Timeout => 120);
 
my %cfg = (
User => $user,
Pass => $password,
Host => $site,
RemoteDir => '/',
Type => 'I');
 
my $ftp = Net::FTP::Common->new(\%cfg, %ncfg);
 
$ftp->login or die "Can't login: $@";
 
my %dir = $ftp->dir;
foreach my $fn (keys %dir) {
next if $fn eq "" || $fn !~ m/$matchstr/;
my $month = $dir{$fn}{"month"};
my $day = $dir{$fn}{"day"};
my $time = $dir{$fn}{"yearOrTime"};
$fn =~ s/\.lmp$//;
my @stat = stat("$localdir$fn"); my $mydate = $stat[9];
my $lsdate = UnixDate(ParseDate("$month $day $time"),"%s");
if ($lsdate != $mydate) {
	 $ftp->get(RemoteFile => $fn, LocalDir => $tempdir, LocalFile => $fn);
	 my @stat = stat("$tempdir$fn");
	 if ($stat[7] > 0) {
		 utime $lsdate, $lsdate, "$tempdir$fn";
		 rename ("$tempdir$fn", "$localdir$fn");
	 }
}
}

Hope this helps!

[kmkimmo]

Excellent work guys!

...and great timing too! I am an ASP coder, about to convert to PHP and Unix in January, and just got signed up to LS datafeed today.

Your scripts are helpful any way I look at it.

Thank you so much for posting them! One of the most helpful posts I have seen on ABW in the last 10 months!!

[chrisk]

Awesome! Thanks for posting your script! I am sure it will help MANY people!

[ecomcity]

Hopefully Y2005 will bring the feed freaks seeking automated spam food to their knees with PRO'd sites right and left. The shoppers and SE's can only pray this happens quickly.

[chrisk]

Nevermind... I'm not being baited...
Have a nice xmas mike...

[jerastraub]

Does anyone now where to get DynuFtp, as the site listed above only have DynuCom. When you try to download DynuFtp it downloads DynuCom instead.

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by chrisk:
I was trying to automate the downloading of the NEW linkshare feeds using a GUI based FTP program on a windows machine. It seems like all the GUI based FTP programs would retrieve a 0 byte file instead of the data file.

I happened to have a copy of Dynu FTP sitting around (30 day trial available http://www.dynu.com/dynuftp.asp) so I wrote this script.

This script will log in using the DynuFTP component, get a directory list and download the NON-XML versions of the data files.

To use this script, downlod and install DynuFT. Then copy the code into a blank text file and give the file a .vbs extention. Next, right click the file and select "Open With Command Prompt"

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>
Set oFTP = Createobject("Dynu.FTP")
If oFTP.connect("aftp.linksynergy.com", "yourusername", "yourpassword")Then
WScript.Echo "CONNECTED"
oFTP.ExecuteCommand("NOOP")
strFileList = oFtp.GetFileList()
Dim FileArray
FileArray = Split(strFileList, VbCrLf)
iMax = ubound(FileArray)
for iCount = 0 to iMax-1
if instr(FileArray(iCount),".xml")&gt;0 then
' dont download the xml version
else
datafilename=left(FileArray(iCount),len(FileArray(iCount))-4)
Wscript.echo datafilename
i = oFtp.GetFile(cstr(datafilename),cstr("C:\" & datafilename) )
end if
next
oFTP.close()
WScript.Echo "FINISHED"
else
WScript.Echo "FAILED CONNECTION"
End If
Set oFTP = nothing
<HR></BLOCKQUOTE>

It might not be the best way, but it works well for me.... <HR></BLOCKQUOTE>

[jerastraub]

What do I save this as to get it to work, as I am new at this.

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by MichaelColey:
For anyone on a Unix platform, here's the script I developed. It was easier than I expected. It will log in, get a listing of all the files available, strip off the ".lmp" extensions, check the dates against files that have already been downloaded, download any that are new or have been changed, and set the file date to the date on the Linkshare FTP site. It relies on the ".lmp" file date being changed when a new files is available (which appears to happen). If a downloaded file is 0 length, the script skips it. You'll probably need to install "Net::FTP::Common". You might need to change the perl path in the first line. You'll need to change the user name, password, and paths. If you use the XML versions instead of the txt versions, change the match string to ".xml.gz".

<pre class="ip-ubbcode-code-pre">#!/usr/local/bin/perl

use strict;
use Net::FTP::Common;
use Date::Manip;

my $user = "{change to your user name}";
my $password = "{change to your password}";
my $site = "aftp.linksynergy.com";
my $localdir = "/path/to/datafeed/directory/";
my $tempdir = "/path/to/temp/directory/";
my $matchstr = ".txt.gz";

my %ncfg = (
Timeout =&gt; 120);

my %cfg = (
User =&gt; $user,
Pass =&gt; $password,
Host =&gt; $site,
RemoteDir =&gt; '/',
Type =&gt; 'I');

my $ftp = Net::FTP::Common-&gt;new(\%cfg, %ncfg);

$ftp-&gt;login or die "Can't login: $@";

my %dir = $ftp-&gt;dir;
foreach my $fn (keys %dir) {
next if $fn eq "" || $fn !~ m/$matchstr/;
my $month = $dir{$fn}{"month"};
my $day = $dir{$fn}{"day"};
my $time = $dir{$fn}{"yearOrTime"};
$fn =~ s/\.lmp$//;
my @stat = stat("$localdir$fn"); my $mydate = $stat[9];
my $lsdate = UnixDate(ParseDate("$month $day $time"),"%s");
if ($lsdate != $mydate) {
$ftp-&gt;get(RemoteFile =&gt; $fn, LocalDir =&gt; $tempdir, LocalFile =&gt; $fn);
my @stat = stat("$tempdir$fn");
if ($stat[7] &gt; 0) {
utime $lsdate, $lsdate, "$tempdir$fn";
rename ("$tempdir$fn", "$localdir$fn");
}
}
}</pre>
Hope this helps! <HR></BLOCKQUOTE>

[chrisk]

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Jairnet Online:
Does anyone now where to get DynuFtp, as the site listed above only have DynuCom. When you try to download DynuFtp it downloads DynuCom instead.

<HR></BLOCKQUOTE>

The download is the correct one. DynuFTP is a component, not a full application. Download the zip, install the component & run the script. It should work fine as long as linkshare is having a good day.

I have found that over the last month, linkshare has not been very reliable in producing the data files.

[jerastraub]

I tryied the script and thought something was wrong with it, as all the data files had 0 content. Also Chrisk do you know how to run the script above that starts with "#!/bin/bash"?
As I would like two try that open as well, as sometimes I only what to download some of the feeds.


<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by chrisk:
<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Jairnet Online:
Does anyone now where to get DynuFtp, as the site listed above only have DynuCom. When you try to download DynuFtp it downloads DynuCom instead.

<HR></BLOCKQUOTE>

The download is the correct one. DynuFTP is a component, not a full application. Download the zip, install the component & run the script. It should work fine as long as linkshare is having a good day.

I have found that over the last month, linkshare has not been very reliable in producing the data files. <HR></BLOCKQUOTE>

[chrisk]

<BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Jairnet Online:
I tryied the script and thought something was wrong with it, as all the data files had 0 content. Also Chrisk do you know how to run the script above that starts with "#!/bin/bash"?
As I would like two try that open as well, as sometimes I only what to download some of the feeds.
<HR></BLOCKQUOTE>

The software linkshare is using on THEIR side to generate the data files appears to not be reliable right now for downloading feeds. Sometimes the data generates ok, sometimes not and you get 0 byte files. Try downloading again tomorrow. If you try downloading them manualy you will get the same results.

The other script is a Linux script. It wont run under windows.

[micheck]

Since I gave up using mget *.lmp and went to get xxxxxxxxxxxx.gz - calling out each file I want I haven't had any problem receiving the data - the script is a lot longer when you call out each file but I don't have to change it often - I haven't had to make a change since LS made the lmp change - again download 10 files several times a week and have had no data burps.

Charles

[jackson992]

I am having real problems getting LS feeds still. I have the following script but can't figure what to do with it:

#!/usr/bin/perl

use Net::FTP;

$ftp = Net::FTP-&gt;new("aftp.linksynergy.com", Debug =&gt; 0);
$ftp-&gt;login("",'');
$ftp-&gt;cwd("/");
$ftp-&gt;get("1155_148860_mp.txt.gz","/domain.com/public_html/Vitacost.txt.gz");
$ftp-&gt;quit;

print "Content-type: text/plain\n\nScript completed.\n";

Please can someone help

[MichaelColey]

Here's a page with some help on getting Perl scripts to work:

http://www.itc.virginia.edu/desktop/web/perl_scripts.html

Also, here's a page about Perl modules (like Net::FTP that the script uses).

http://www.cpan.org/misc/cpan-faq.html

And lastly, here's a page with information about writing CGI with Perl, which is essentially what that script is.

http://www.cgi101.com/book/

[mattdunbar]

When I run the vbs I get a microsoft vb compilation error.

[mcalpin1]

What do these lines of code support?

RemoteDir => '/',
Type => 'I');

[MichaelColey]

RemoteDir => '/' just tells the FTP program that the downloads are in the root directory of your LinkShare FTP login.

Type => 'I' tells the FTP program to download in "Binary" mode instead of "Text" mode.

[mcalpin1]

I'm not to program savy, but I belive I revised allthe correct files. What should I save this file as? cgi?

[MichaelColey]

No, it's a perl script that you'll probably want to set up in a cron. You'll probably need someone with programming expertise if you're going to work with datafeeds.