Page 1 of 2 12 LastLast
Results 1 to 25 of 42
  1. #1
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    673
    I was trying to automate the downloading of the NEW linkshare feeds using a GUI based FTP program on a windows machine. It seems like all the GUI based FTP programs would retrieve a 0 byte file instead of the data file.

    I happened to have a copy of Dynu FTP sitting around (30 day trial available http://www.dynu.com/dynuftp.asp) so I wrote this script.

    This script will log in using the DynuFTP component, get a directory list and download the NON-XML versions of the data files.

    To use this script, downlod and install DynuFT. Then copy the code into a blank text file and give the file a .vbs extention. Next, right click the file and select "Open With Command Prompt"

    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>
    Set oFTP = Createobject("Dynu.FTP")
    If oFTP.connect("aftp.linksynergy.com", "yourusername", "yourpassword")Then
    WScript.Echo "CONNECTED"
    oFTP.ExecuteCommand("NOOP")
    strFileList = oFtp.GetFileList()
    Dim FileArray
    FileArray = Split(strFileList, VbCrLf)
    iMax = ubound(FileArray)
    for iCount = 0 to iMax-1
    if instr(FileArray(iCount),".xml")&gt;0 then
    ' dont download the xml version
    else
    datafilename=left(FileArray(iCount),len(FileArray(iCount))-4)
    Wscript.echo datafilename
    i = oFtp.GetFile(cstr(datafilename),cstr("C:\" & datafilename) )
    end if
    next
    oFTP.close()
    WScript.Echo "FINISHED"
    else
    WScript.Echo "FAILED CONNECTION"
    End If
    Set oFTP = nothing
    <HR></BLOCKQUOTE>

    It might not be the best way, but it works well for me....

  2. #2
    Full Member dak142's Avatar
    Join Date
    January 18th, 2005
    Posts
    386
    Interesting piece of software. Any chance you know of simular software out there that will do the same thing but in PHP rather than ASP?

  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    673
    Most people on the Linux side write perl scripts to do it from what I understand. I am sure PHP probably has it built in also. Sorry I can't me more specific, my extertise is on the Windows side.

    This script does not run from an ASP page - but it uses a component that 'can' be used on an ASP page. The script is sort of like an advanced batch file.

  4. #4
    Full Member
    Join Date
    January 18th, 2005
    Posts
    396
    The variation for the UNIX/LINIX side of the house that works for me is as follows:

    I create a file - I call it GETIT.SH containing:

    #!/bin/bash

    ftp -i -n &lt;&lt;HERE
    open aftp.linksynergy.com
    user USER_NAME PASSWORD
    bin
    get XXX1_YYYYYYY_mp.txt.gz
    get XXX2_YYYYYYY_mp.txt.gz
    get XXX3_YYYYYYY_mp.txt.gz
    close
    quit
    HERE

    I save this file in my home directory - I find that the directory structure is a bit beyond me so I use the UNIX command PWD which gives me the path to the directory where the file and I am. I also make the file an executable file by using the UNIX command CHMOD +X GETIT.SH.

    I go into the CRON (timer) program in my cpanel and pick when I want the 'batch' file to run and in the blank for the 'script to run' I put the full path I got from PWD and /getit.sh

    It runs for me - I hope I pointed out the gotchas so it will run for you. PM me if you'd like.

  5. #5
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    673
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Charles_Michener:
    The variation for the UNIX/LINIX side of the house that works for me is as follows:
    <HR></BLOCKQUOTE>

    It would be cool if someone could post a version that fetched all available files like my Windows version does. It's a pain maintaining the list


    For windows folks, here is a page that tells you how to automate (free) the download if you don't mind specifying each file name in your batch file.
    http://www.infoworld.com/cgi-bin/dis...t/990823bl.htm

  6. #6
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    I was using a Unix solution called "lftp" that has a mirror option. The mirror option downloaded only the items that changed. Of course the "upgrade" broke that.

    Has anyone found a Unix solution that only downloads the items that changed? I might have to dig through the Perl modules on CPAN to find an FTP module and then build something myself. I hate having to take time away from working on sites just to fix something that Linkshare "upgraded".
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    673
    Yeah, I used to do something similar on the windows site (mirroring) - I had FTP-Voyager on a 4am syncronization schedule... The reason I built the auto-downloader was to solve the issue with the linkshare 'upgrade'

  8. #8
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    For anyone on a Unix platform, here's the script I developed. It was easier than I expected. It will log in, get a listing of all the files available, strip off the ".lmp" extensions, check the dates against files that have already been downloaded, download any that are new or have been changed, and set the file date to the date on the Linkshare FTP site. It relies on the ".lmp" file date being changed when a new files is available (which appears to happen). If a downloaded file is 0 length, the script skips it. You'll probably need to install "Net::FTP::Common". You might need to change the perl path in the first line. You'll need to change the user name, password, and paths. If you use the XML versions instead of the txt versions, change the match string to ".xml.gz".

    Code:
    #!/usr/local/bin/perl
     
    use strict;
    use Net::FTP::Common;
    use Date::Manip;
     
    my $user = "{change to your user name}";
    my $password = "{change to your password}";
    my $site = "aftp.linksynergy.com";
    my $localdir = "/path/to/datafeed/directory/";
    my $tempdir = "/path/to/temp/directory/";
    my $matchstr = ".txt.gz";
     
    my %ncfg = (Timeout => 120);
     
    my %cfg = (
    User => $user,
    Pass => $password,
    Host => $site,
    RemoteDir => '/',
    Type => 'I');
     
    my $ftp = Net::FTP::Common-&gt;new(\%cfg, %ncfg);
     
    $ftp->login or die "Can't login: $@";
     
    my %dir = $ftp->dir;
    foreach my $fn (keys %dir) {
    next if $fn eq "" || $fn !~ m/$matchstr/;
    my $month = $dir{$fn}{"month"};
    my $day = $dir{$fn}{"day"};
    my $time = $dir{$fn}{"yearOrTime"};
    $fn =~ s/\.lmp$//;
    my @stat = stat("$localdir$fn"); my $mydate = $stat[9];
    my $lsdate = UnixDate(ParseDate("$month $day $time"),"%s");
    if ($lsdate != $mydate) {
    	 $ftp->get(RemoteFile => $fn, LocalDir => $tempdir, LocalFile => $fn);
    	 my @stat = stat("$tempdir$fn");
    	 if ($stat[7] > 0) {
    		 utime $lsdate, $lsdate, "$tempdir$fn";
    		 rename ("$tempdir$fn", "$localdir$fn");
    	 }
    }
    }
    Hope this helps!
    Last edited by MichaelColey; July 16th, 2005 at 05:30 PM. Reason: Updated to vBulletin "Code"
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  9. #9
    Member kmkimmo's Avatar
    Join Date
    January 18th, 2005
    Location
    Texas Coast
    Posts
    133
    Excellent work guys!

    ...and great timing too! I am an ASP coder, about to convert to PHP and Unix in January, and just got signed up to LS datafeed today.

    Your scripts are helpful any way I look at it.

    Thank you so much for posting them! One of the most helpful posts I have seen on ABW in the last 10 months!!
    - I am not young enough to know everything

  10. #10
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    673
    Awesome! Thanks for posting your script! I am sure it will help MANY people!

  11. #11
    2005 Linkshare Golden Link Award Winner  ecomcity's Avatar
    Join Date
    January 18th, 2005
    Location
    St Clair Shores MI.
    Posts
    17,328
    Hopefully Y2005 will bring the feed freaks seeking automated spam food to their knees with PRO'd sites right and left. The shoppers and SE's can only pray this happens quickly.
    Webmaster's... Mike and Charlie

    "What have you done today to put real value into a referral click...from a shoppers viewpoint!"

  12. #12
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    673
    Nevermind... I'm not being baited...
    Have a nice xmas mike...

  13. #13
    Newbie
    Join Date
    January 18th, 2005
    Posts
    15
    Does anyone now where to get DynuFtp, as the site listed above only have DynuCom. When you try to download DynuFtp it downloads DynuCom instead.

    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by chrisk:
    I was trying to automate the downloading of the NEW linkshare feeds using a GUI based FTP program on a windows machine. It seems like all the GUI based FTP programs would retrieve a 0 byte file instead of the data file.

    I happened to have a copy of Dynu FTP sitting around (30 day trial available http://www.dynu.com/dynuftp.asp) so I wrote this script.

    This script will log in using the DynuFTP component, get a directory list and download the NON-XML versions of the data files.

    To use this script, downlod and install DynuFT. Then copy the code into a blank text file and give the file a .vbs extention. Next, right click the file and select "Open With Command Prompt"

    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>
    Set oFTP = Createobject("Dynu.FTP")
    If oFTP.connect("aftp.linksynergy.com", "yourusername", "yourpassword")Then
    WScript.Echo "CONNECTED"
    oFTP.ExecuteCommand("NOOP")
    strFileList = oFtp.GetFileList()
    Dim FileArray
    FileArray = Split(strFileList, VbCrLf)
    iMax = ubound(FileArray)
    for iCount = 0 to iMax-1
    if instr(FileArray(iCount),".xml")&gt;0 then
    ' dont download the xml version
    else
    datafilename=left(FileArray(iCount),len(FileArray(iCount))-4)
    Wscript.echo datafilename
    i = oFtp.GetFile(cstr(datafilename),cstr("C:\" & datafilename) )
    end if
    next
    oFTP.close()
    WScript.Echo "FINISHED"
    else
    WScript.Echo "FAILED CONNECTION"
    End If
    Set oFTP = nothing
    <HR></BLOCKQUOTE>

    It might not be the best way, but it works well for me.... <HR></BLOCKQUOTE>

  14. #14
    Newbie
    Join Date
    January 18th, 2005
    Posts
    15
    What do I save this as to get it to work, as I am new at this.

    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by MichaelColey:
    For anyone on a Unix platform, here's the script I developed. It was easier than I expected. It will log in, get a listing of all the files available, strip off the ".lmp" extensions, check the dates against files that have already been downloaded, download any that are new or have been changed, and set the file date to the date on the Linkshare FTP site. It relies on the ".lmp" file date being changed when a new files is available (which appears to happen). If a downloaded file is 0 length, the script skips it. You'll probably need to install "Net::FTP::Common". You might need to change the perl path in the first line. You'll need to change the user name, password, and paths. If you use the XML versions instead of the txt versions, change the match string to ".xml.gz".

    <pre class="ip-ubbcode-code-pre">#!/usr/local/bin/perl

    use strict;
    use Net::FTP::Common;
    use Date::Manip;

    my $user = "{change to your user name}";
    my $password = "{change to your password}";
    my $site = "aftp.linksynergy.com";
    my $localdir = "/path/to/datafeed/directory/";
    my $tempdir = "/path/to/temp/directory/";
    my $matchstr = ".txt.gz";

    my %ncfg = (
    Timeout =&gt; 120);

    my %cfg = (
    User =&gt; $user,
    Pass =&gt; $password,
    Host =&gt; $site,
    RemoteDir =&gt; '/',
    Type =&gt; 'I');

    my $ftp = Net::FTP::Common-&gt;new(\%cfg, %ncfg);

    $ftp-&gt;login or die "Can't login: $@";

    my %dir = $ftp-&gt;dir;
    foreach my $fn (keys %dir) {
    next if $fn eq "" || $fn !~ m/$matchstr/;
    my $month = $dir{$fn}{"month"};
    my $day = $dir{$fn}{"day"};
    my $time = $dir{$fn}{"yearOrTime"};
    $fn =~ s/\.lmp$//;
    my @stat = stat("$localdir$fn"); my $mydate = $stat[9];
    my $lsdate = UnixDate(ParseDate("$month $day $time"),"%s");
    if ($lsdate != $mydate) {
    $ftp-&gt;get(RemoteFile =&gt; $fn, LocalDir =&gt; $tempdir, LocalFile =&gt; $fn);
    my @stat = stat("$tempdir$fn");
    if ($stat[7] &gt; 0) {
    utime $lsdate, $lsdate, "$tempdir$fn";
    rename ("$tempdir$fn", "$localdir$fn");
    }
    }
    }</pre>
    Hope this helps! <HR></BLOCKQUOTE>

  15. #15
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    673
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Jairnet Online:
    Does anyone now where to get DynuFtp, as the site listed above only have DynuCom. When you try to download DynuFtp it downloads DynuCom instead.

    <HR></BLOCKQUOTE>

    The download is the correct one. DynuFTP is a component, not a full application. Download the zip, install the component & run the script. It should work fine as long as linkshare is having a good day.

    I have found that over the last month, linkshare has not been very reliable in producing the data files.

  16. #16
    Newbie
    Join Date
    January 18th, 2005
    Posts
    15
    I tryied the script and thought something was wrong with it, as all the data files had 0 content. Also Chrisk do you know how to run the script above that starts with "#!/bin/bash"?
    As I would like two try that open as well, as sometimes I only what to download some of the feeds.


    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by chrisk:
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Jairnet Online:
    Does anyone now where to get DynuFtp, as the site listed above only have DynuCom. When you try to download DynuFtp it downloads DynuCom instead.

    <HR></BLOCKQUOTE>

    The download is the correct one. DynuFTP is a component, not a full application. Download the zip, install the component & run the script. It should work fine as long as linkshare is having a good day.

    I have found that over the last month, linkshare has not been very reliable in producing the data files. <HR></BLOCKQUOTE>

  17. #17
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    673
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Jairnet Online:
    I tryied the script and thought something was wrong with it, as all the data files had 0 content. Also Chrisk do you know how to run the script above that starts with "#!/bin/bash"?
    As I would like two try that open as well, as sometimes I only what to download some of the feeds.
    <HR></BLOCKQUOTE>

    The software linkshare is using on THEIR side to generate the data files appears to not be reliable right now for downloading feeds. Sometimes the data generates ok, sometimes not and you get 0 byte files. Try downloading again tomorrow. If you try downloading them manualy you will get the same results.

    The other script is a Linux script. It wont run under windows.

  18. #18
    Full Member
    Join Date
    January 18th, 2005
    Posts
    396
    Since I gave up using mget *.lmp and went to get xxxxxxxxxxxx.gz - calling out each file I want I haven't had any problem receiving the data - the script is a lot longer when you call out each file but I don't have to change it often - I haven't had to make a change since LS made the lmp change - again download 10 files several times a week and have had no data burps.

    Charles

  19. #19
    Action Jackson - King of the World
    Join Date
    January 18th, 2005
    Posts
    2,201
    I am having real problems getting LS feeds still. I have the following script but can't figure what to do with it:

    #!/usr/bin/perl

    use Net::FTP;

    $ftp = Net::FTP-&gt;new("aftp.linksynergy.com", Debug =&gt; 0);
    $ftp-&gt;login("",'');
    $ftp-&gt;cwd("/");
    $ftp-&gt;get("1155_148860_mp.txt.gz","/domain.com/public_html/Vitacost.txt.gz");
    $ftp-&gt;quit;

    print "Content-type: text/plain\n\nScript completed.\n";

    Please can someone help

  20. #20
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    Here's a page with some help on getting Perl scripts to work:

    http://www.itc.virginia.edu/desktop/...l_scripts.html

    Also, here's a page about Perl modules (like Net::FTP that the script uses).

    http://www.cpan.org/misc/cpan-faq.html

    And lastly, here's a page with information about writing CGI with Perl, which is essentially what that script is.

    http://www.cgi101.com/book/
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  21. #21
    Member
    Join Date
    February 23rd, 2005
    Posts
    77
    compilation error
    When I run the vbs I get a microsoft vb compilation error.

  22. #22
    Newbie
    Join Date
    January 18th, 2005
    Posts
    10
    What do these lines of code support?

    RemoteDir => '/',
    Type => 'I');

  23. #23
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    RemoteDir => '/' just tells the FTP program that the downloads are in the root directory of your LinkShare FTP login.

    Type => 'I' tells the FTP program to download in "Binary" mode instead of "Text" mode.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  24. #24
    Newbie
    Join Date
    January 18th, 2005
    Posts
    10
    I'm not to program savy, but I belive I revised allthe correct files. What should I save this file as? cgi?

  25. #25
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    No, it's a perl script that you'll probably want to set up in a cron. You'll probably need someone with programming expertise if you're going to work with datafeeds.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

+ Reply to Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Problems with php datafeed download script
    By steven420 in forum Programming / Datafeeds / Tools
    Replies: 11
    Last Post: November 29th, 2006, 08:17 AM
  2. Linkshare Datafeed Download Script
    By Snib in forum Programming / Datafeeds / Tools
    Replies: 28
    Last Post: August 11th, 2006, 12:05 AM
  3. DYSe script Documentation download
    By g352004 in forum Cusimano.com Scripts
    Replies: 1
    Last Post: July 24th, 2005, 02:46 PM
  4. Datafeed Download w/ script???
    By cditty in forum Commission Junction - CJ
    Replies: 12
    Last Post: October 27th, 2003, 05:58 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •