Results 1 to 12 of 12
  1. #1
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    CJ Datafeed Processing
    I need to find a better way to automate the datafeed I get from CJ. I currently have CJ deliver a feed in the form of a few zipped files. Each ziped file then contains separate feed files from a list of merchants that I requested.

    I need to make the process of getting the files and updating the data in my database less manual. Would it be better for me to grab the files from CJ or keep it as it is now and have them ftp deliver to a directory on my host?

    Is there a way using php to process the huge zipped file and then read each of the merchant files contained within the zip?

  2. #2
    Full Member
    Join Date
    March 10th, 2006
    Posts
    466
    I need to make the process of getting the files and updating the data in my database less manual. Would it be better for me to grab the files from CJ or keep it as it is now and have them ftp deliver to a directory on my host?
    I tried a few things to automate the CJ download file. Currently it's on their server and it doesn't seem to let me use the FTP GET command. So I need to manually drag the file down. I have considered that if they put the file on my server I may be able to automate it more easily. It's once a week thing so not on top of my list, but getting there. It's around 600 MB when zipped and a pain to deal with. So actually, I can't answer your question based on my experience.

    Is there a way using php to process the huge zipped file and then read each of the merchant files contained within the zip?
    I process it into a Mysql Database on my PC and then extract the data I need. Dirk Gardner posted a bunch of code on ABW about importing to Mysql using PHP to a server. I can think of a few different ways to process the zipped file using PHP, but it depends on how the data is used later. I deliver it from a Mysql database to text files which is probably least conventional.

    Note: Using PHP, the file format .txt.gz works for me. I never did manage to automate anything using .zip extension files. That's not to say it can't be done, in fact, I'm sure it can using non-open source scripts to unzip.

  3. #3
    Full Member
    Join Date
    January 18th, 2005
    Posts
    396
    I'm not sure I'm answering your questions but ...

    I use a batch file to download the LS and CJ data to my desktop using wget.exe; it then unzips CJ using the free unzip.exe or LS with gzip.exe; I then have a couple of VBscripts that pick apart the txt files, update my data bases, create my Google... site maps and gzip/zip the resulting files. My batch process finally transfers these files to my server.

    This process takes my desktop about 6 hours once a week and is completely automatic once I start it. On the server I have a 'task schedule' that weekly (several hours after the upload process) unzips ... the new files into their domains

    my CJ autoget routine:
    wget -c -nv -T30 -N ftp://UName:Pword@datatransfer.cj.com.../*.*

    My CJ unzip command:
    unzip -o -aa *.zip

    Charles

  4. #4
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    I am trying to see if hostgator has a unzip or gunzip tool that I can call from a cron script. Support person says no, but I gotta think they do.

  5. #5
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    I've got a single zip file on CJ's FTP server that my server downloads daily. I had to request a static file name and daily updates to make this as smooth as possible. Once my server finishes downloading it uses the command line "unzip" utility to extract the files. Then my import script goes at each merchant file until they're all updated.

    It's fully automated. Once I map a feed to my database I never have to think about updating it. The only manual step in this whole process is mapping products to my category structure. It's all done via a web interface and once I've mapped a category it'll stay that way unless the merchant changes it.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  6. #6
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    I had to unzip the file manually to get all of the merchant txt files. After that, I wrote a script to read eaxh txt file in the directory and perform a "LOAD DATA LOCAL INFILE" on each file, which my last hosting company didn't support.

    Now, I just need to find an unzip binary I can call on the hostgator.

  7. #7
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by SeymourButts
    Now, I just need to find an unzip binary I can call on the hostgator.
    Yup, that's a must. If they don't offer one you might want to install one. And if you can't do that you should just get a new host.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  8. #8
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    So, my script just processed the 40 datafeed files I currently have asked for, which equates to about 25 merchants. It took 30 seconds and I now have 1,194,384 records in the DB.

    I want to add more merchants, but I wonder how many records will be too many.

  9. #9
    Full Member
    Join Date
    March 10th, 2006
    Posts
    466
    my CJ autoget routine:
    wget -c -nv -T30 -N ftp://UName:Pword@datatransfer.cj.com.../*.*
    Thanks Charles my file is "auto" downloading as I write this. I had to download a copy of Wget first, and install it in my PC.

    This will save me a ton of time in research, and manual downloads!

  10. #10
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by SeymourButts
    So, my script just processed the 40 datafeed files I currently have asked for, which equates to about 25 merchants. It took 30 seconds and I now have 1,194,384 records in the DB.

    I want to add more merchants, but I wonder how many records will be too many.
    Really depends on what sort of server resources you have and how you're querying the data. I'd say for anything over 1 million you should have at least 2gb of ram on a dedicated server for snappy pages. But that's if you're doing a lot of dynamic pages. You can do into the double digit millions if you've got multiple dedicated servers load balanced with enough ram. Just make sure you've got indexes on your tables to make the most of your resources.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  11. #11
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    Snib is the man - that is exactly how I would have solved this. Get it onto my server as one file and separate yourself from the other server ASAP. The fewer files transferred the better.
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

  12. #12
    Member
    Join Date
    October 4th, 2007
    Posts
    109
    Does it take a certain level of sales to get the datafeeds? i can sign up for their webservices and get them for free, but since the webservices are slow and klunky, I'd rather ftp at this point.

  13. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Beta Testers Required - Online Datafeed Processing Tools
    By dmorison in forum Programming / Datafeeds / Tools
    Replies: 13
    Last Post: March 3rd, 2005, 04:18 PM
  2. CC Processing
    By sagar_kar in forum Credit Card processing and other payments
    Replies: 3
    Last Post: March 1st, 2005, 09:05 AM
  3. Datafeed Management/Processing Software
    By Apollo in forum Commission Junction - CJ
    Replies: 6
    Last Post: November 30th, 2003, 09:02 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •