Results 1 to 8 of 8
  1. #1
    Roll Tide mobilebadboy's Avatar
    Join Date
    January 18th, 2005
    Location
    Mobile, Alabama
    Posts
    1,220
    Getting large file from remote server
    I'm looking for the easiest way to get a large ZIP file (200mb+) from a remote server on to my server. A script doesn't work, it simply takes too long and it eventually times out killing the download halfway through.

    I'd prefer something automatic, but since it only needs to be retrieved once a week, at best, a manual solution would suffice.

    I'm on a *nix box.

    Would appreciate ideas.

    Shawn Kerr (.com) | Disney World | SEC Football

  2. #2
    Beachy Bill's Avatar
    Join Date
    November 20th, 2005
    Posts
    8,266
    Quote Originally Posted by mobilebadboy
    I'm looking for the easiest way to get a large ZIP file (200mb+) from a remote server on to my server. A script doesn't work, it simply takes too long and it eventually times out killing the download halfway through.

    I'd prefer something automatic, but since it only needs to be retrieved once a week, at best, a manual solution would suffice.

    I'm on a *nix box.

    Would appreciate ideas.
    WGET from the command line on your server to pull it from the host server? I'm not good at all with command line stuff, but...

    ...About 3 1/2 years ago, in the infancy of Wikipedia, they made all of their data available for download. I replicated their site on one of my domains - without images - so dial-up students (at my school) wouldn't have to wait on images slowing the pages. I remember looking up some unix command stuff and implementing it - and I was able to pull over 200,000 text files (over 1.1 GB of data) from a server in Sweden to my VPS in about 15 minutes.
    Bill / Marketing Blog @ 12PM - Current project: Resurrecting my "baby" at South Baltimore..
    Cute Personal Checks and Business Checks
    If you are too busy to laugh you are too busy.

  3. #3
    ABW Ambassador bettylou's Avatar
    Join Date
    December 27th, 2005
    Location
    Indiana
    Posts
    595
    Not sure if this is relevant to your situation but the scripts that I run that take a while (written in PHP) do just fine if I add this to the beginning:

    <?php
    ini_set('max_execution_time', '0');
    ?>


    Might be worth a try.

  4. #4
    Roll Tide mobilebadboy's Avatar
    Join Date
    January 18th, 2005
    Location
    Mobile, Alabama
    Posts
    1,220
    yeah, Beachy, I'm not that good with it either.

    bettylou, thanks, but still made no difference. No matter what I try script wise it gets to about 120-130MB and times out.

    Shawn Kerr (.com) | Disney World | SEC Football

  5. #5
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Is this by chance a huge zipped up file from CJ? If so, just have them push it.

  6. #6
    Roll Tide mobilebadboy's Avatar
    Join Date
    January 18th, 2005
    Location
    Mobile, Alabama
    Posts
    1,220
    No, it's not affiliate marketing related.

    But I figured out how to get it where i want through SSH/wget.

    Shawn Kerr (.com) | Disney World | SEC Football

  7. #7
    Beachy Bill's Avatar
    Join Date
    November 20th, 2005
    Posts
    8,266
    Quote Originally Posted by mobilebadboy
    ...through SSH/wget.
    Ah-Ha! WTG. I'll bet it worked quickly, too...
    Bill / Marketing Blog @ 12PM - Current project: Resurrecting my "baby" at South Baltimore..
    Cute Personal Checks and Business Checks
    If you are too busy to laugh you are too busy.

  8. #8
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    wget is the best way for this (for http and wget) I am learning. It has options in there to manage retries and timeouts also. I even found a windows version.
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

  9. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Performing Remote Server Maintenance From Smart Phone
    By isellstuff in forum Programming / Datafeeds / Tools
    Replies: 1
    Last Post: April 7th, 2014, 04:15 PM
  2. URL file-access is disabled in the server configuration
    By mnsonline in forum Programming / Datafeeds / Tools
    Replies: 8
    Last Post: February 29th, 2008, 07:34 PM
  3. zip file datafeed file transfer into database
    By rahulcbr in forum Programming / Datafeeds / Tools
    Replies: 13
    Last Post: August 28th, 2007, 04:59 AM
  4. Cop Movie [large file inside]
    By Geno Prussakov in forum Virtual Family and Off-Topic
    Replies: 1
    Last Post: July 25th, 2007, 06:44 PM
  5. CGI - deleting file if it is too large
    By c4 in forum Programming / Datafeeds / Tools
    Replies: 1
    Last Post: March 14th, 2002, 05:31 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •