Results 1 to 4 of 4
March 23rd, 2010, 04:35 PM #1
Changing my processing from local to on the Web with PHP
- Join Date
- January 17th, 2005
I am contemplating changing my VB based preprocessing (datafeed downloading/index generation/data file formatting .../data uploading) to operate on the Web and save me the download/upload problems and time - one more step toward automating everything.
While I believe that I can translate my VB operations to PhP, I am afraid that the cpu intensive processes will bog the web server. The vb process on a fast cpu with 4 G of memory takes 28 hours now and it does bog one side of a dual Intel.
How do you folks that are handling large files online handle the 'cpu bogging' problem?
Any additional comments/thoughts?
March 23rd, 2010, 04:44 PM #2
March 23rd, 2010, 04:54 PM #3
The big thing is going to be memory usage with php, especially if the files are large enough that it takes 28 hours to process currently. PHP isn't very good at memory handling in some cases. If you can offer up more info about what it is you are doing specifically, I may be able to help.
March 23rd, 2010, 07:39 PM #4
Wouldn't converting from VB to ASP be much quicker for the code re-write?
If you are just starting out I highly recommend looking at a "cloud" solution for processing your data-feeds. You'll need to write command and control logic so you can automatically start/stop/upload processed data etc. from your web server computer. That way all your CPU/Memory usage is on another system and will not impact your (or someone else's) web server.
For example, I have a fairly small processing system, ~21-million products. Most of my processing is using on demand Amazon EC2 servers. Previously I kicked off four servers every other day for about 19 hours, running about $130/month in charges. Now I'm similar logic to run 1 EC2 spot instance almost full time (they can kill it if they have someone willing to pay more than me) that is running under $50/month (unix instance).
By bookdrawer in forum Blogging, Mobile and Social MediaReplies: 10Last Post: October 11th, 2010, 12:33 PM
By ShaneWatson in forum Midnight Cafe'Replies: 11Last Post: August 25th, 2009, 12:25 AM
By Uncle Rico in forum Programming / Datafeeds / ToolsReplies: 3Last Post: January 4th, 2009, 10:55 AM
By UncleScooter in forum Midnight Cafe'Replies: 20Last Post: July 7th, 2005, 09:44 PM