Results 1 to 9 of 9
  1. #1
    Newbie
    Join Date
    April 5th, 2006
    Posts
    3
    file sizes too big?
    I'm running into the issue that the file sizes that I'm trying to upload from a comma-separated list to a mysql generator are too big. Has anyone else encountered this? Is there a more efficient way to handle this other than physically splitting the file up?

    Thanks,
    Mike

  2. #2
    Newbie
    Join Date
    September 8th, 2005
    Posts
    3
    file too big
    build your sql file in your localhost server and then if everythings alright ask your hosting to import your database for you to your hosting account they can do it in a fly. whereas if you try to import those large sql file it returns timeout error

    regards

  3. #3
    Newbie
    Join Date
    April 2nd, 2006
    Posts
    40
    what is your mysql generator?
    CSV file size?

  4. #4
    Newbie
    Join Date
    April 5th, 2006
    Posts
    3
    PHPmill.

    189MB.

    Another issue that I run into is when I want to find/replace my affiliate ID in this huge text file Windows pretty much locks up on my trying to change them all. Any suggestions for that as well?

    Thanks,
    Mike

  5. #5
    Newbie
    Join Date
    April 2nd, 2006
    Posts
    40
    pretty big. let me think

  6. #6
    Newbie
    Join Date
    April 2nd, 2006
    Posts
    40
    did you tried: 'generate SQL file only' function?

    to find/replace my affiliate -- did you try function 'Custom Replacement'?

    i'd prefer using own perl script for this (so big files....)

  7. #7
    ABW Ambassador FFoc's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,015
    You're running out of execution time. Check the php docs for info on how to fix that:
    http://php.net/manual/en/ref.info.php
    http://php.net/set_time_limit
    “An adventure is an inconvenience rightly considered. An inconvenience is an adventure wrongly considered.” - G.K. Chesterton

  8. #8
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    198 mb = peanuts

    Skip using a generator program. MySQL itself is the uber tool.

    Switch to a setup where you can access/upload to MySQL directly via the command line (or use a language like PHP, but not with some other program in the way!), and learn the commands you need in order to manipulate it yourself (there aren't that many commands needed for basic changes to a feed--for that task, it's no harder than learning to use one of these "intermediary" programs, IMO.).

    Watch filesize-related problems disappear.

    For the find/replace, just have MySQL do that, too. It'll go through even 1 GB like butter. The only times I've had a problem with a feed due to file size is when some other program tries to tackle it.

  9. #9
    Newbie
    Join Date
    January 4th, 2006
    Location
    Berlin
    Posts
    41
    two pence
    What are you using to try and upload to your mysql, phpmyadmin or something? If so then phpmyadmin or other programs which connect to mysql will eventually time out, even if they do not most of the time your browser will actually time out before the action finishes. (if you are using a web browser).

    Seperately if you are using phpmyadmin or the like it is virtually impossible to upload massive files using a post method from a website form, which is what the browse button and upload feature is in phpmyadmin.

    198mb is actually huge for working with, although like someone else said peanuts (if you know what you are doing). Larger files need a little different way of working. What you should be doing is making the file on your local drive then uploading the sql file you made to your web server and executing the import from there, even if the browser times out, because the file is on the server the likelyhood is that the process will continue as all phpmyadmin is doing is executing commands directly to mysql which doesnt need your browser to be kept alive. Other ways do include using SSH clients and executing commands but no matter how many people say it is easy, it is a learning curve.

    To answer your find and replace problem, i presume you are using word or excel or something. It is not an efficient command, by that i mean the software was never meant to be some efficient data handling command. Now if you executed a command that told mysql to do this, then you would find like for like it would be more efficient, though even so mysql is not great at handling memory it would probably lock up for a bit but not as long as your windows machine would with a normal find and replace. Perl is a great program for managing data because it was effectively made for data, or at least evolved into being the best for it, though do not get confused with using perl to execute what is an sql command (in this case its mysql doing the work).

    So, generate the file, upload to your server first, then execute the file, either by using the same program you normally use or learning how to use ssh to execute the import.

    Hope that helps

    --------------------------------------------------------------------------
    [Removed Promotional Signature URL]
    Last edited by DesignerWiz; May 4th, 2006 at 04:37 PM. Reason: Removed Promotional Signature URL

  10. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. 10 Big companies that could file BK
    By Merchant Consultant Team in forum Midnight Cafe'
    Replies: 4
    Last Post: September 21st, 2009, 10:01 AM
  2. Big csv file, says 0 records processed. If I cut it down it processes.
    By aaelghat in forum WebMerge (Fourthworld.com)
    Replies: 6
    Last Post: July 5th, 2009, 01:12 AM
  3. Problem with big .htaccess file?
    By Affiliate Ian in forum Midnight Cafe'
    Replies: 2
    Last Post: August 20th, 2004, 02:40 PM
  4. Replies: 1
    Last Post: June 1st, 2004, 07:44 PM
  5. Rochester Big and Tall File Not Found
    By IRS in forum Commission Junction - CJ
    Replies: 3
    Last Post: May 3rd, 2004, 07:38 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •