Results 1 to 5 of 5
  1. #1
    Full Member
    Join Date
    January 18th, 2005
    Posts
    273
    So the buy.com feed is massive. Even zipped it's at 150 meg. My server side unzip script times out before it can finish.

    So I was thinking about downloading it locally, unzipping it, breaking the text file into pieces and then zipping into smaller files and then pushing back to the server where they can be unzipped successfully. I can automate almost all of this, but I'm stuck on how to break up the text files. Any programs that will do this from the command line?

  2. #2
    ABW Ambassador buy_online's Avatar
    Join Date
    January 18th, 2005
    Location
    Richmond, VA
    Posts
    3,234
    Sounds like a lot of work.

    Have you asked your hosting provider if they can help, maybe a different decompression utility, or change default settings? Mine will allow me to start a trouble ticket, and I could get them to do it.

    You are doing this from the shell?

    Fred

    Are you sure the nurses know you're using the computer?

  3. #3
    Full Member
    Join Date
    January 18th, 2005
    Posts
    273
    Actually it looks like the problem is that the script timesout beofre I get the whole feed which obvioussly results in a corrupted zip file. I may have to investigate a non-perl way of grabbing the file.

  4. #4
    Newbie
    Join Date
    January 18th, 2005
    Posts
    13
    You can grab the file using PHP's FTP functions and set the script timeout to either unlimited or the amount of time you think it will take your server to download the feed.

    We use that method and it works for all feeds.

  5. #5
    ABW Ambassador buy_online's Avatar
    Join Date
    January 18th, 2005
    Location
    Richmond, VA
    Posts
    3,234
    This may be a little late, but Jerome's idea looks pretty good. It would be nifty to leave it in one piece if you could.

    Fred

  6. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. breaking or pulling apart data feed
    By sweetiowa in forum ShareASale - SAS
    Replies: 14
    Last Post: October 13th, 2007, 08:25 PM
  2. Handling a huge feed
    By Paul_Ward in forum WebMerge (Fourthworld.com)
    Replies: 24
    Last Post: August 28th, 2006, 02:50 PM
  3. Breaking down a feed
    By Ms Woodtick in forum WebMerge (Fourthworld.com)
    Replies: 5
    Last Post: August 5th, 2004, 07:50 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •