Results 1 to 6 of 6
  1. #1
    Troll Killer and best Snooper!
    I decide when the pigs fly!
    Rhea's Avatar
    Join Date
    January 18th, 2005
    Location
    New York, USA
    Posts
    6,195
    Choking on a CJ datafeed file
    Hey kids, I need your help.

    I'm trying to unzip an enormous CJ txt file from the datafeed so I can open it in Excel and play with it. The problem is the zipped file is so big (600k records according to rumor) that it kills my system memory every time I try to unzip it.

    Can I chunk this file into digestible portions? How the heck do you guys deal with gigunda feeds like this? :sobbing:

    Muchos!

  2. #2
    Internet Cowboy
    Join Date
    January 18th, 2005
    Posts
    4,662
    You just about have to use a script to import it into a MySQL database and export the sections of it that you need. There might be another way, but I don't know about it if there is.
    I have used Access in the past and imported large datafeeds into it, then exporting chunks. But I don't know if Access will take a 600K item feed or if your computer will handle loading it.
    At that size, the merchant should break it up into pieces before sending it to CJ. They might be able to send you the part you need.

  3. #3
    Classic Rocker Mack's Avatar
    Join Date
    January 27th, 2007
    Location
    Lower Left Coast
    Posts
    1,167
    Are you trying to open it while it's still in the archive? Just unzipping doesn't take that much memory, it's just decompressing it as it writes it to disk.

    Once it's written to the disk you should be able to open it with Excel if you have 2007. I've opened some that large but it gets slow, even on my (3Ghz cpu/2gb ram). I did it just to see it do it.

    Mr. Scooter is right. Best to import it into a DB and just pull out what you need. I just use wamp locally to massage feeds like that. Or were you thinking of puttin the entire thing online?

  4. #4
    Troll Killer and best Snooper!
    I decide when the pigs fly!
    Rhea's Avatar
    Join Date
    January 18th, 2005
    Location
    New York, USA
    Posts
    6,195
    Thanks guys. I figured out that I was making a mistake regarding unzipping the file. So I copied the file to a new folder which caused the file to decompress, and then I opened it in Excel. It only took about a minute but Excel truncated the file at approximately 65,000 records. I guess I have to ask CJ to chunk the thing for me so I can work with it.

  5. #5
    Classic Rocker Mack's Avatar
    Join Date
    January 27th, 2007
    Location
    Lower Left Coast
    Posts
    1,167
    Not sure how much luck you'll have with CJ.

    Old versions of Excel will only open 65,535, 2007 will open 1,000,000. But 2007 is a less than eloquent solution. Your now on a path that will lead you to a multitude of options.

  6. #6
    Member Core's Avatar
    Join Date
    March 16th, 2006
    Location
    Cornfield, Illinois
    Posts
    95
    I have it set up to unzip the datafeed and insert it into a main table. After that's done it copies from the main table to databases for each website (I have a vps server with multiple websites). This makes the data easy to use for me and allows me to replace the default website id with the website for which it is being used for in the buyurl and impressionurl.

  7. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. How often datafeed file get updated?
    By smallfish in forum BettyMills
    Replies: 0
    Last Post: November 23rd, 2009, 07:35 PM
  2. Replies: 4
    Last Post: June 23rd, 2009, 07:39 PM
  3. zip file datafeed file transfer into database
    By rahulcbr in forum Programming / Datafeeds / Tools
    Replies: 13
    Last Post: August 28th, 2007, 04:59 AM
  4. How do I use a .aspx datafeed file?
    By rAd in forum Programming / Datafeeds / Tools
    Replies: 2
    Last Post: June 20th, 2005, 12:22 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •