Results 1 to 7 of 7
  1. #1
    Newbie
    Join Date
    January 16th, 2008
    Posts
    5
    Big csv file, says 0 records processed. If I cut it down it processes.
    I have a large Excel csv file that is the source file with WebMerge (it has 213,000 rows). I'm running Windows XP.

    The file looks good in the previewer (all the label and data fields look accurate). To keep things simple, I'm only building detail pages.

    If I try and process the whole thing, or even if I run it in test mode with 10 records, I get a box that says WebMerge processed 0 records in 54 seconds.

    If I cut the file down to the first few rows, it does process (WebMerge processed 8 records).

    In the debugging process I've tried changing the template file to "auto-generate", and I've turned off sorting all with the same results.

    Any hints? I can supply the files if necessary.

  2. #2
    Affiliate Manager
    Join Date
    January 18th, 2005
    Location
    Los Angeles, California
    Posts
    1,913
    There could be a few possibilities at play here.

    If you're using the sort feature, note that sorting will only work reliably when no single record in the feed exceeds 65k. That's usually many times more than anyone needs, but I've seen it come into play once over the years.

    Also, make sure the delimiters are set appropriately for the file. CSV is a notoriously inconsistent format, and while we've done our best to accommodate Microsoft's various different implementations, there's enough inconsistencies that at a minimum you may want to toggle the "Text values are in quotes" checkbox and try again.

    If those don't get it working well, you might consider re-exporting in a more consistent format like tab-delimited, and try that.

    Please keep me posted on how that works out.
    Richard Gaskin
    Developer of WebMerge: Publish any data feed on any site
    http://www.fourthworld.com

  3. #3
    Newbie
    Join Date
    January 16th, 2008
    Posts
    5
    Thanks for all the suggestions. I tried exporting in tab-delimited format, as well as togglng the "text values in quotes" field for both csv and tab-delimited files. Here's what I've found:

    • Under all formats/toggles, if I try and process the full file with 213,000 records it says 0 processed.
    • If I cut the file way down to under 100 records, it will process and I'll get html documents as I should.
    • If I cut the file down to about 100,000 records, WebMerge crashes (I get the Windows box saying WebMerge crashed, and do I want to send an error report to Microsoft).


    The behavior above is consistent with both tab and csv delimited files. Does that provide any clues? Thanks in advance.

  4. #4
    notary sojac Herb ԿԬ's Avatar
    Join Date
    January 18th, 2005
    Location
    Central/Western NY State
    Posts
    7,741
    Lightbulb
    until I upgraded my computer I monitored my memory usage while running WM.

    when the usage matched the amount of physical memory it crashed.

  5. #5
    Affiliate Manager
    Join Date
    January 18th, 2005
    Location
    Los Angeles, California
    Posts
    1,913
    What's the total size of the file, and how much RAM do you have installed?

    Yes, it does seem like a memory limitation. WebMerge can handle sources up to 4GB, but few computers have enough RAM to handle the two of the source file WebMerge needs to keep in memory during some operations.

    Can you export the feed into smaller chunks, perhaps by category or some other logical break that'll work well with your workflow?

    Where did the data originate from before being brought into Excel? Perhaps it could be exported there more easily.

    Are there fields in the data which are unused in your web production? Perhaps excluding those from the export will trim the file down to a size your machine can handle.
    Richard Gaskin
    Developer of WebMerge: Publish any data feed on any site
    http://www.fourthworld.com

  6. #6
    Newbie
    Join Date
    January 16th, 2008
    Posts
    5
    Thanks Herb & FourthWorld for the guidance...

    I think the memory limitation issue is the main possibility. I did cut the file down by eliminating columns that aren't used by WebMerge, and it got much farther. WebMerge still crashed, but I got an exception error from WebMerge which I pasted below.

    The size of the original csv file (before I cut it down) was 331 Megabytes (and 213,000 rows). I deleted columns to get the csv file down to 155 Megabytes. I have 2 GB of memory on my computer, and before I started processing the file I had 1.5 GB free.

    The data is created in Excel because I have an Excel Macro that creates the rows out of permutations of various pieces of data I have in other worksheets.

    As I was running the process on the cut-down file, I was running task manager, and I did see the available memory go down as the process was running. I had to step away, and the error happened I was gone, but based on the rate the memory was being used up, I'd imagine I did run out of memory.

    I think I can cut a few more columns out of the file, and it is possible for me to run this in two batches, so I'll try that. I do have another idea/question of running this on a Linux server, but I'll open a different thread on that so I don't hijack this one.

    Thanks again for your help, and the error message follows - I suspect it's a symptom of running out of memory?

    System info:
    System version: Win32 NT 5.1
    Display Rect: 0,0,1280,800
    QuickTime version: 7.6.2
    -----------------------------------------------------------
    Error Info:
    Handler: aborted: (Line 1164, column 1)
    try: error in statement: (Line 1150, column 1)
    if-then: error in statement: (Line 1147, column 1)
    repeat: error in statement: (Line 1123, column 1)
    Handler: error in statement: MergeTemplate (Line 1123, column 1)
    Object: button "wmCommands" of group "BS" of card id 1002 of stack "wmLibs" of stack "4W WebMerge.exe"
    Object ID: button id 1014 of group id 1007 of card id 1002 of stack "wmLibs" of stack "C:/Program Files/4W WebMerge/4W WebMerge.exe"
    -
    put: error in expression: (Line 790, column 1)
    if-then: error in statement: (Line 787, column 1)
    repeat: error in statement: (Line 722, column 1)
    Handler: error in statement: BuildAllPages (Line 722, column 1)
    Object: button "wmCommands" of group "BS" of card id 1002 of stack "wmLibs" of stack "4W WebMerge.exe"
    Object ID: button id 1014 of group id 1007 of card id 1002 of stack "wmLibs" of stack "C:/Program Files/4W WebMerge/4W WebMerge.exe"
    -
    Handler: error in statement: BuildEm (Line 119, column 1)
    Object: button "wmCommands" of group "BS" of card id 1002 of stack "wmLibs" of stack "4W WebMerge.exe"
    Object ID: button id 1014 of group id 1007 of card id 1002 of stack "wmLibs" of stack "C:/Program Files/4W WebMerge/4W WebMerge.exe"
    -
    Handler: error in statement: BuildPages (Line 79, column 1)
    Object: button "wmCommands" of group "BS" of card id 1002 of stack "wmLibs" of stack "4W WebMerge.exe"
    Object ID: button id 1014 of group id 1007 of card id 1002 of stack "wmLibs" of stack "C:/Program Files/4W WebMerge/4W WebMerge.exe"
    -
    Handler: error in statement: mouseUp (Line 3, column 1)
    Object: button "BuildPages" of card "main" of stack "4W WebMerge.exe"
    Object ID: button id 1481 of card id 1002 of stack "C:/Program Files/4W WebMerge/4W WebMerge.exe"
    -
    Thrown error:

  7. #7
    Affiliate Manager
    Join Date
    January 18th, 2005
    Location
    Los Angeles, California
    Posts
    1,913
    Could I trouble you to use the Report button on the error window to email that to me? That helps a lot for tracking issues like this. If you prefer to send that email manually, please use support@fourthworld.com

    I'm about to leave for a couple days of R&R, but I'll be back next week and look forward to working with you to get this running well as soon as I get back.

    While I'm away, you might take a look at the WM-If statements in your template to make sure they're in the right format. If you're using the mod operator in there, that issue's already been addressed and you can use one of the engines posted for testers at:
    http://www.fourthworld.com/products/webmerge/test/

    X is for Linux, exe for Win, app for Mac. Just unzip it and put it into your existing WebMerge folder to run.

    Thanks, and have a great weekend.
    Richard Gaskin
    Developer of WebMerge: Publish any data feed on any site
    http://www.fourthworld.com

  8. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. All Records not being Processed by WebMerge
    By xbsjason in forum WebMerge (Fourthworld.com)
    Replies: 5
    Last Post: September 27th, 2009, 10:02 AM
  2. Use CSV File directly for DataFeed
    By ttxxtt8 in forum Programming / Datafeeds / Tools
    Replies: 10
    Last Post: July 23rd, 2005, 02:50 PM
  3. text file processed as body of email?
    By efgroup in forum Programming / Datafeeds / Tools
    Replies: 1
    Last Post: September 20th, 2004, 06:11 PM
  4. CSV File instead of Webservices ?
    By Cosmo in forum Amazon
    Replies: 5
    Last Post: August 20th, 2004, 11:25 AM
  5. Getting data from CSV file into HTML Framed pages?
    By Happy in forum WebMerge (Fourthworld.com)
    Replies: 1
    Last Post: March 18th, 2004, 09:43 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •