Results 1 to 4 of 4
  1. #1
    Full Member
    Join Date
    January 17th, 2005
    Considering converting my datafeed update
    At present I download my datafeeds in a zip format (using wget) - then I process them (unzipping - developing indexes - formatting the data files - loading them into several large databases - building updated site maps - zipping up the results) - I then upload the new zip files to my web site where I unzip them into the several folders that make up my site. I do this process twice a week and while it takes me only a few minutes each time - batch file scripts are nice - it does tie me to my home computer more than I like.

    I am considering rewriting my local processing scripts into php and doing the entire gathering/processing/updating on the web. Potential problems are that my processing requires a substantial computer effort and I worry that I will be guilty of trying to 'hog' a not insignificant amount of the web server resources.

    How do you folks that already do this online processing/updating handle the web server resources problem?


  2. #2
    Newbie bhavikm's Avatar
    Join Date
    July 5th, 2010
    Not being a technical guy, as well as new to the affiliate world, I'm not sure this answers your question, but I'll give it a try. Also, I have a VPS, so I'm not sure if that plays a factor in your question.

    My website is currently designed to process datafeeds on my server nightly. Using some sort of PHP script, my website pulls the datafeeds via FTP from multiple network, unzips them, puts all of the datafeeds from a network (CJ, SAS, etc) in a temporary database table for the network. Then it pulls all of these temporary tables into a master table that my website queries (after re-aligning them so the columns match).

    The original way my web designer built it, it used up a lot of resources. My website slowed to a crawl when the scripts were running, but by changing something in the PHP, he was able to get it so I didnt even notice the scripts running. I have not heard any issues with using too much resources.

  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    memory_get_usage is your friend

    In my case I allocate the maximum amount of memory that I want my application to use find out if its using

    if(memory_get_usage() > $max_memory_use){
    Dump to database

    I have found that feeds can change unexpectedly which could affect your script execution. Add some checks to see if you’re not entering garbage or wiping your existing database etc etc
    Last edited by Wayder; August 31st, 2010 at 03:18 AM. Reason: tab: enter:

  4. #4
    Full Member iolaire's Avatar
    Join Date
    October 3rd, 2006
    Arlington, VA
    I recommend using Amazon EC2 to do your processing. If you invest some (or a lot) of time in setting up the system to boot an instance, load code and run it on that instance, then get the data back to your host you will have access to almost unlimited CPU/Memory resources at very low-cost.

    I use the Medium High-CPU On-Demand Instance continually, as a spot priced request with a max price of 7 cents an hour (list price 17 cents), it rarely goes down (because the price is over 7 cents), so I get it for about $50/month - that's 1.7 megs of ram and the equivalent of 5 CPUs.
    Last edited by iolaire; September 2nd, 2010 at 08:48 AM.

  5. Newsletter Signup

+ Reply to Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts