Results 1 to 6 of 6
  1. #1
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    LinkShare, Performics, DataFeeds, Cable Modem
    Hi All,

    I would like to cut my web server out of the loop and use my home computer to grab linkshare feeds and possibly give other companies an FTP account on my home computer where they can send feeds via FTP. Right now I am driving Linkshare downloads indirectly via a web service, but it is a pain and unreliable.

    I was wondering if anyone is using a static IP at home to get this done? Is it as simple as upgrading my RoadRunner/Time Warner account to business class, getting a static IP and configuring my router/local web server?

    I'm ok with figuring out the technical details, would just like to hear a success story from someone so that I have warm fuzzy's, knowing it is possible.

    Thanks,
    Jim

  2. #2
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    I think you're going about this backwards. It's best to have the datafeeds sent to your web server so your server can handle all the database importing and price updating. If you download datafeeds to your local computer you're introducing a bottleneck. You'll need to manually send the data to your server after you receive it on your computer. It's much more efficient to take your computer out of the equation and have your server handle everything.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  3. #3
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    I'm afraid I'm doing something a bit larger than normal on a fairly tight budget. I understand what you are saying, but that solution would require me to purchase two dedicated servers due to the size of my site. I'll have to think about it...

    My datafeed files are many Gigabytes in size, and my database and lucene index are very large as well. I believe that processing would severly hamper my web site performance, and maybe not even work because of the 4 GB ram limitation on my 32 bit servers. Granted it could be 3 in the morning, but...

    I haven't priced out a symmetric 1.5 mbs cable connection yet, maybe I will change my tune???

  4. #4
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    I'm currently only running a single dedicated server for my largest datafeed site. The updates occur during the night so it doesn't hinder the shoppers during the day, but I do plan to introduce a second server soon. It's definitely better to have the updating take place outside of the shopping space. A second server is a deductable business expense, so it's definitely a worthwhile investment.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  5. #5
    Newbie Heritage's Avatar
    Join Date
    May 3rd, 2006
    Location
    Elkins Park, PA 800-550-6051 X23 robert (*) payday-loan-yes.com
    Posts
    10
    Re: Cutting the webserver out of the loop.
    I know most server plans can get up there, but more N more server plans have gone down in price. Have you explored options like Alabanza, Godaddy, or large upscale networks that offer resonable shared or dedicated hosting with good servers and even better pricing?

    Are you serving all your HTML from the database? There may be a way to cut down redundant K with SSI and conserve bandwith, drive space, and over head. Or even data indexing so you do not have to pin so much in RAM. Is it an issue of think smart, not muscle? No offense, but I would ask yourself a few more questions. I totally feel ya for needing to get control of the situation, but like other people have stated. It seems your going backwards.

    Open up some options? Do you have a planned budget? If so how much and how much drive space do you need in your database?

    All the best,

    Rob

  6. #6
    ABW Ambassador isellstuff's Avatar
    Join Date
    November 9th, 2005
    Location
    Virginia
    Posts
    1,659
    I don't want to go into too much detail on what I'm building, but I will say that a large component is a data warehouse backend that will be crunching numbers and spitting out reports 24-7 and my database for this will run into the terabytes. Obviously I can't have this thing running on my web server.

    I have about 40 front end databases x 2 which will be about (8-10 GB) x 2 INITIALLY. The front-end DB's should grow substantially. Because I have partitioned the data into many databases, I expect to be able to upload segments of it at a time. So I have a table which defines the "active" database for a particular data segment. The datawarehouse machine will query the webserver database for the active segment, then lock the mirrored (inactive) DB, compress the newly updated local DB, transfer to the webserver and overwrite the inactive DB, then "turn it on" by updating the active table row. My expectation is that I can hot swap DB's in this fashion without bringing down the server. I have automated this entire process and use ssh/scp with compression for the transfer mechanism in hopes that I will get better relability. I want this process to run unattended.

    I have to admit that I am a computer engineer and part of the fun of this thing is leveraging my ability to crunch data and solve difficult problems. I just don't want to spend too much money doing it until I have proven that it works. This is the third version of something I have been working on for awhile. The first two worked to a limited extent, now I'm trying to scale up, refine, and see what happens.

    Later,
    Jim

  7. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Performics Datafeeds
    By bestdeals in forum Google Affiliate Network - GAN
    Replies: 17
    Last Post: December 7th, 2006, 07:26 AM
  2. How to get Performics datafeeds?
    By webworker in forum Google Affiliate Network - GAN
    Replies: 3
    Last Post: October 1st, 2006, 04:13 AM
  3. Cable Modem w/router
    By Steveinid in forum Midnight Cafe'
    Replies: 8
    Last Post: October 8th, 2004, 04:26 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •