Results 1 to 12 of 12
  1. #1
    Newbie
    Join Date
    January 18th, 2005
    Posts
    14
    28000 records in 30 minutes, is this normal, fast or slow?

  2. #2
    ABW Ambassador buy_online's Avatar
    Join Date
    January 18th, 2005
    Location
    Richmond, VA
    Posts
    3,234
    WebMerge is usually much faster than that, but don't panic right away. Many factors can contribute to the processing speed. Obviously, the speed of your processor, the type of OS you have, and most importantly the data feed file, its contents and your templates.

    The last items are probably the real culprit, and I am not completely sure what makes it that way. It is certainly more than just lots of information in the records that might slow things down. I have run the Betty Mills feed, and I think it took about fifteen-minutes (29,000 records), and another feed with 5,000 records takes the same amount of time - go figure.

    So short answer is yes it might be normal, but is/can be much shorter.

  3. #3
    Full Member
    Join Date
    January 18th, 2005
    Posts
    413
    The first Betty Mills datafeed that I processed on a P4 2.4GZ machine with 512 RAM took around 30 minutes (or more). So, your times may be right.

    You might try Stanley's recent suggestion. Compressing your template files might make a big difference when running a 29,000 record datafeed. Stanley wrote:

    I've found that WebMerge seems to process files faster if I compress the templates first with the added benefit that the pages load faster on the server too. Just a little performance boost for those interested...

  4. #4
    Member MarkJH's Avatar
    Join Date
    January 18th, 2005
    Posts
    68
    I think I might just be a bit slow today (having a day off work does that to me!) but can you explain what you mean by compressing the templates?
    [b][url=www.bandlink.net]Bandlink.net[/url][/b]

  5. #5
    Full Member
    Join Date
    January 18th, 2005
    Posts
    413
    Some HTML editors, like Front Page, use a formating structure that inserts extras spaces in order to make the HTML coding more readable. However, this also increases the file size of your template (and every file generated from that template). The difference may only be a couple of Kilobytes, and insignificant if your website only has a couple dozen (or hundred) webpages. BUT, when you multiple that difference by 29,0000 it becomes significant.

    Compressing the template before running WM will result in smaller webpages (saving you server space), and, as Stanley has noted, makes WM run faster. I just ran a comparative test on a datafeed with 8,000 records using it to generate a four-tiered website, i.e. with three WM settings files. Without compressing the templates the run took exactly 7 min 53 sec. After compressing the templates, the run took 7 min. 2 sec. That's a 12% saving it running time. Results may vary depending on your situation!

    Thera are various HTML compressor programs that help you compress webpages. Go to download.com and type in "html compressor."

    Two warnings:

    1) Backup your original formatted template before compressing. Depending on the editor that you are using, it may not uncompress a compressed file very well. Some compressors do this automatically for you.

    2) It has been my experience that some compressors mess up java scripts. Make sure that your test your compressed files. Some compressors have "options" to select what they should (or should not) compress.

  6. #6
    Member MarkJH's Avatar
    Join Date
    January 18th, 2005
    Posts
    68
    Ah!

    Yes, FrontPage does add rather a lot of superfluous code. I saved myself nearly 2Kb on my results template recently just taking out some <span> tags that FP had decided to add for no reason! With 8000 pages, it can really add up.

    Thanks for that, Frank.
    [b][url=www.bandlink.net]Bandlink.net[/url][/b]

  7. #7
    Affiliate Manager
    Join Date
    January 18th, 2005
    Location
    Los Angeles, California
    Posts
    1,913
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by MarkJH:
    Ah!

    Yes, FrontPage does add rather a lot of superfluous code. I saved myself nearly 2Kb on my results template recently just taking out some &lt;span&gt; tags that FP had decided to add for no reason! With 8000 pages, it can really add up. <HR></BLOCKQUOTE>

    We have customers regularly processing many more records in less time. Depending on the complexity of the templates, of course, processing speed will vary.

    How many WebMerge tags are in those templates?
    Does WebMerge note any processing errors?
    Richard Gaskin
    Developer of WebMerge: Publish any data feed on any site
    http://www.fourthworld.com

  8. #8
    Full Member
    Join Date
    January 18th, 2005
    Posts
    413
    ===================
    How many WebMerge tags are in those templates?
    Does WebMerge note any processing errors?
    ===================

    In my case I used four templates with a total of 63 WM tags (5, 19, 18 & 21).

    The database (from BeFree) has 26 fields of data and 8,095 records. I import 12 of the 26 fields using WebMerge.

    The output including index pages resulted in 8,496 webpages. This was run on a P4 1.6 GHz ThinkPad T40 with 2Gig RAM. The run time was 7 min 53 sec with uncompressed templates and 7 min 2 sec with compressed templates.

  9. #9
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    532
    This is the one I use...

    Advanced HTML Optimizer

    ...worth every penny!

  10. #10
    Member MarkJH's Avatar
    Join Date
    January 18th, 2005
    Posts
    68
    I downloaded that same piece of software yesterday.

    Makes the code extremely hard to read so I'm keeping a non-optimised version to work on but it got my template size down from 40Kb to 31Kb. That's an overall saving of nearly 80Mb!

    Richard, my main template uses 385 WM opening tags!
    [b][url=www.bandlink.net]Bandlink.net[/url][/b]

  11. #11
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    532
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by MarkJH:
    Makes the code extremely hard to read <HR></BLOCKQUOTE>

    Definitely make a copy before optimization or have the software backup the files for you (personally I make my own copy adding a 'BAK' to the filename). The plus side is it make site copying by the casual webmaster harder. Anyone using a decent HTML tool can de-optimize the code but most won't try unless your designs are really stellar

  12. #12
    Affiliate Manager
    Join Date
    January 18th, 2005
    Location
    Los Angeles, California
    Posts
    1,913
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by MarkJH:
    Richard, my main template uses 385 WM opening tags! <HR></BLOCKQUOTE>

    Well, then it's gonna take a bit. But trust me, it's working it's little behind off under the hood.

    The next version will speed up some internal operations related to the WM-Field tag by as much as 80%. So hang in there -- it only gets better and better.

    But tell me: What sort of optimization are you looking for? If it's just stripping white space and abbreviating JavaScript function and variable names I have a routine for that I could drop in as an option....
    Richard Gaskin
    Developer of WebMerge: Publish any data feed on any site
    http://www.fourthworld.com

  13. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Hi, Merchant Getting Up to Speed
    By HAF in forum Introduce Yourself
    Replies: 1
    Last Post: March 11th, 2010, 03:43 PM
  2. Speed up Firefox
    By waybar in forum Midnight Cafe'
    Replies: 7
    Last Post: January 25th, 2005, 11:12 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •