Results 1 to 14 of 14
  1. #1
    Newbie
    Join Date
    August 17th, 2007
    Location
    India
    Posts
    5
    zip file datafeed file transfer into database
    Hi i m new in this field any body help me, how i can transfer data from a zip file datafeed.What script is used

  2. #2
    Affiliate Manager PetsWarehouse.com's Avatar
    Join Date
    January 17th, 2005
    Location
    Long Island, New York
    Posts
    1,616
    You first need to unzip it.

    Use WinZip
    Bob Pets Warehouse
    Worlds Largest Pet Supply DataBase
    Join our Share-A-Sale Program [since 2003] Twitter



  3. #3
    Newbie
    Join Date
    August 17th, 2007
    Location
    India
    Posts
    5
    Quote Originally Posted by PetsWarehouse.com
    You first need to unzip it.

    Use WinZip
    thanx for responce me Pl. tell me all the steps involve behid it.

  4. #4
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    Quote Originally Posted by rahulcbr
    thanx for responce me Pl. tell me all the steps involve behid it.
    If you are using a zipped file (don't know why unless you believe it's easier - more about why in a moment) then unzip using WinZip. Rename the unzipped file to what you want to call it as each unzipped file has a different number.

    Upload the file to a location on your server on a domain in a folder or leave on your PC (server is better).

    Set up your script to map the datafeed located on your server.

    Configure your script to convert the feed into a displayable layout, specifying where the layout is to be displayed. That's basically it.

    Now why use zipped. It only creates more work. Many networks enable feed download automatically from a url so why not have a script which obtains the feed directly and then store it in MySQL instead? Alternatively download the feed from the network unzipped and store it in a folder as before using either SQL or MySQL to utilise as a database and then follow the steps to map and display. Some networks time out with big feeds but it's still better obtaining them without using zip. Just creates more work for you.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  5. #5
    Newbie
    Join Date
    February 24th, 2007
    Posts
    13
    Now why use zipped. It only creates more work.
    It's quicker, for sure. I can download a big feed zipped, like itunes, directly to my server (zipped at 500MB, unzipped at 2.5G) and unzip it on my end a helluvalot quicker than downloading that 2.5G file

  6. #6
    Newbie
    Join Date
    February 24th, 2007
    Posts
    13
    thanx for responce me Pl. tell me all the steps involve behid it.
    Do you have shell access?

  7. #7
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    Quote Originally Posted by mikebrad0927
    It's quicker, for sure. I can download a big feed zipped, like itunes, directly to my server (zipped at 500MB, unzipped at 2.5G) and unzip it on my end a helluvalot quicker than downloading that 2.5G file
    Yes I know that an individual file can be downloaded quicker but is it? You then physically unzip it and then map it. Add the two processes together and what is the actual time difference?

    Then what if you have 500 feeds to download? Are you going to get a zip physically of each? What a waste of time. Just set up a cron job and get the unzipped file straight into your database. As each feed is automatically downloaded it is checked for errors and then stored. This then leaves you plenty of time to do what you're supposed to be doing as an affiliate - promoting merchants products and services, not wasting time manually downloading feeds.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  8. #8
    Newbie
    Join Date
    February 24th, 2007
    Posts
    13
    Add the two processes together and what is the actual time difference?
    Well, you prompted me to perform a little experiment, because honestly, I've never checked to see. I ran two scripts on a smaller file as to not waste time. I used overstock.com because they, unlike linkshare, offer uncompressed feeds in addition to the compressed ones. The compressed feed downloaded and unzipped in 47 seconds and the full feed downloaded in 1 minute 22 seconds.

    Code:
    -bash-3.00$ date
    Wed Aug 22 05:58:21 PDT 2007
    -bash-3.00$ cd /home/*****/www/movies/feeds
    -bash-3.00$ /usr/bin/wget -nv --dot-style=mega ftp://<username>:<password>@ftp.overstock.com/DVD.zip
    05:58:53 URL: ftp://*username*:*password*@ftp.overstock.com/DVD.zip [16388009] -> "DVD.zip" [1]
    -bash-3.00$ unzip DVD.zip
    Archive:  DVD.zip
      inflating: DVD.csv
    -bash-3.00$ date
    Wed Aug 22 05:59:08 PDT 2007
    -bash-3.00$ date
    Wed Aug 22 05:59:40 PDT 2007
    -bash-3.00$ cd /home/*****/www/movies/feeds
    -bash-3.00$ /usr/bin/wget -nv --dot-style=mega ftp://<username>:<password>@ftp.overstock.com/DVD.csv
    06:01:02 URL: ftp://*username*:*password*@ftp.overstock.com/DVD.csv [60818995] -> "DVD.csv.1" [1]
    -bash-3.00$ date
    Wed Aug 22 06:01:02 PDT 2007
    -bash-3.00$

  9. #9
    Newbie
    Join Date
    February 24th, 2007
    Posts
    13
    Are you going to get a zip physically of each?
    If you are downloading 500 feeds, then I have to assume that you would use SSH. If this is the case it's easy, just use "gzip -df *txt.gz".

    There is NOTHING manual about downloading these feeds, just create yourself a little script, set up your cron job and viola!

    All of the feeds will be downloaded to your server (not your desktop), unzipped, and even imported into your database (if you want).

  10. #10
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    I'm not worthy!

    Leaving most of the techie stuff to my techie I'd forgotten about gzip.

    I was just running with the assumption they'd only have a simple script to read the feed hence my wondering why they bothered getting a zip file and then category map it etc.

    Still it was nice to see you enjoyed yourself with your experiment.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  11. #11
    Newbie
    Join Date
    February 24th, 2007
    Posts
    13
    Yeah, it was fun for me to do. I'm glad for this discussion because I had never really checked to see if I was really saving time with all of my late night crons. I just assumed I was and we all know what they say about assumptions.

    BTW, I looking forward to early 2008 and seeing your price comparison script.

  12. #12
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    Cheers for that. So am I. The wild thing about it all is that every customer who wants a custom job brings new ideas to the table and before you know it, you start rewriting stuff to incorporate it because it sounds like a good idea.

    It's getting to the stage though where I've got to take on additional freelance script writers on revenue share. If I get too much more work I'll have to start turning work away.

    Take a look at the affiliate window network in the UK. They've incorporated so many of our innovations. they've even just launched a widget, albeit using our old javascript format that we told them about, whereby affiliates create different sized creative from a database of products on the fly through their affiliate platform. We basically offer the same from a merchant end and also a php version for affiliates. We did away with the javascript version a bit ago.

    You can see a test widget we did for a merchant which they left up on a test page at www.deadgoodundies.com/donk.php , the networks basically used that as the concept.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  13. #13
    Newbie
    Join Date
    February 24th, 2007
    Posts
    13
    John, I'm new to this forum and therefore do not have pms avail yet

  14. #14
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    No probs. Your profile is set not to receive email so I can't email you at present either. Feel free to email me sometime and I'll tell you what it was about.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  15. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. How to FTP and Import CJ datafeeds from the CJ Zip File...
    By Mr. Sal in forum Programming / Datafeeds / Tools
    Replies: 20
    Last Post: June 10th, 2009, 04:21 AM
  2. Read Zip File
    By Uncle Rico in forum Programming / Datafeeds / Tools
    Replies: 6
    Last Post: February 9th, 2008, 05:04 PM
  3. signature not found in database file
    By qsd76 in forum Cusimano.com Scripts
    Replies: 2
    Last Post: March 8th, 2006, 12:25 PM
  4. datafeed transfer into a database
    By tsmgroup2 in forum Programming / Datafeeds / Tools
    Replies: 11
    Last Post: December 10th, 2005, 10:31 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •