Results 1 to 3 of 3
  1. #1
    Join Date
    August 9th, 2008
    New to datafeeds and searching for some wisdom.
    New to datafeeds and searching for some wisdom.

    Hello, all. Let me preface this post by saying that I started getting involved with datafeeds about a week or two ago, and I've done my best to get a head start by reading as many sources as possible on the subject. I've also installed and have been playing around with one of the available scripting solutions. So far, this forum seems to have had the most comprehensive discussion on the topic, and I'm hoping to contribute by sharing my experience, and hopefully pick up a bit of knowledge along the way.

    My necessity for using datafeeds comes from my work in affiliate marketing and PPC. So far, I've mostly done PPC direct to merchant, and as you can imagine, I have a strong need to develop my own landing pages. Most of my current campaigns are within a retail niche, with a fine granulation of products, so datafeeds are a natural fit. While I have a reasonable amount of experience building webpages and using scripts like wordpress, I cannot code and am not interested in learning, so I'm left to look for commercially available solutions and/or custom software.

    My main needs in creating landing pages are to be able to import the items I need for my campaigns, organize them according to existing adgroups, and make sure that all the information is accurate, stays up to date, and that the site looks professional/credible. My niche is fairly narrow and highly granulated, so I don't necessarily need the feeds to be huge, but I'm still unsure as to how I can most effectively update my feeds. I've heard of cron job refreshes and aggregator services (popshops, datafeedr) but I'm still somewhat clueless in that regard. So far I've looked at Affilistore, PriceTapestry, PopShops, and Datafeedr (the later two having Wordpress integration, which is a platform I'm comfortable with).

    I've managed to try Affilistore, and there are some things I like about it so far. One of them is the navigation system, which allows me to create static pages that are populated by feed items that contain specified search strings. In other words, I can specify a certain page to show feed items containing search strings, for example "widget a," "widget b," but not "green" and not items over $100. This can end up being extremely useful to me, since most of the products I market stay essentially the same year to year, but at the same time model numbers are usually revised on an annual basis.

    Another positive is a reasonably decent default template, which, while not the most important factor in the grand scheme of things, may allow me to create a site rather quickly and cheaply. The template has certain features, like user ratings, featured items, recently browsed items, etc., which all go well with a retail shopping experience.

    I do have some concerns. I don't know if my shared hosting plan will be enough to handle my needs. Affilistore builds pages dynamically, and I'm a bit concerned about system resources and website loading times. So far I'm working with a one merchant feed containing 8k items, and if I can cull that down by removing products that I am not actively promoting it might get to be considerably less, but as long as I'm sucessfull I'll be building more datafeed based sites in the future.

    Also, the feed that I'm working with now is uncategorized, and contains many items that I am not promoting. While I can use negative keywords to keep them from showing within my basic navigation, they still manage to show up via a random products sections, as well as in a sitewide search. These items are also potentially slowing down my site and hogging CPU cycles. The problem with deleting these is two-fold: it's time consuming and potentially useless if I'm going to be updating the feed regularly. Does anybody know of a way to cull the feed and still be able to update prices and availability? And if so, are there any tools that I can use to macro delete certain items, such as deleting all lines with the word "shoes" and "pink" and all items above or below a certain price threshold?

    My final concern is with conversion rates and the question of added content. I need my site to work quickly, reliably and provide my visitors with what they are looking for along with a positive user experience. Originally I thought about creating a niche price comparison site, which would offer my visitors the added benefit of savings and potentially boost conversions. Furthermore, this kind of feature along with some original added content could allow me pursue organic SEO. While this is all still a possibility, I've scaled back my expectations to just providing an accurate and appealing selection of products within an appealing and easy to use website. With that said, I'm still unsure if Affilistore is the best option for my needs, and I'm hoping to see other's opinions and experiences with similar projects.

  2. #2
    Full Member iolaire's Avatar
    Join Date
    October 3rd, 2006
    Arlington, VA
    I don't have experience with Affilistore or another package solution.

    However I will strongly suggest that rather than working around the limitations of shared hosting you do your best to move to your own VPS like hosting solution. In my mind that is really the first hurdle an affiliate needs to pass if they are looking at data feeds - is moving from shared resources to fixed resources that you are free to use up on your own processes.

    I'm personally a fan of's OpenSolaris Accelerators since they physically limit your CPU usage when needed (but allow bursting CPU usage when it available), so when you MySQL or PHP processes are overloaded you don't get nasty emails from your hosting company... I think other cloud services like Amazon EC2 provide similar - but most VPS's, like for example the nice low cost, do not actually limit your CPU usage at the OS level - so you can still get those nasty emails.

  3. #3
    Join Date
    January 18th, 2005
    I agree that affiliates running any kind of datafeed script on the server should probably be using at least a VPS, and as noted it is important to recognize that many VPS services don't properly meter CPU usage but instead expect YOU to avoid over-utilization. (This is true at eApps, which I've used for a couple of years.) If you're planning on managing a lot of datafeeds, perhaps with millions of records, you probably need a dedicated server.

    If you use a tool that makes use of a "real" database solution (e.g. MySQL), and the database is well designed with the "right" fields properly indexed (a process that can itself generate a lot of CPU load), and you create your queries intelligently, you shouldn't have any problem with a database as small as 8,000 or 80,000 records. Keep in mind that your product-data query might not be the only database operation occuring during each pageview (most content-management solutions query a database for your page content, and other operations (such as web-analytics software) might also generate one or several database queries (read and/or write) for a variety of reasons).

    I had expected that dynamic serving of pages that include product data would be slow, and that I'd need to generate static pages -- but I was actually quite pleasantly surprised at how various solutions (including Datafeed Studio and several scripts I created) were able to deliver multiple pageviews per second with no meaningful CPU load nor page-load delays, even when the database contained more than a million product records. You should definitely do some "load testing" to see what happens when you achieve a peak traffic level (for most of us, peak traffic is likely one or two dozen pageviews per minute, but good load-testing would try to determine how your system works with higher traffic (for example, 100 dynamic page requests in 10 seconds -- perhaps while simultaneously downloading and importing updated datafeeds, if that's going to happen dynamically also).

    There's a lot of useful information about using datafeeds on ABW, including many posts I've made over the past 4 years regarding my "datafeed project" (which I've abandoned for now). I posted links to a number of these posts here.
    Last edited by markwelch; October 10th, 2009 at 07:06 PM.

  4. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Words of wisdom
    By David Carter in forum Virtual Family and Off-Topic
    Replies: 3
    Last Post: October 1st, 2007, 11:53 PM
  2. Does anybody have any words of wisdom?
    By Radarman in forum Midnight Cafe'
    Replies: 16
    Last Post: November 20th, 2006, 07:57 AM
  3. Some tips and wisdom for everyone. Welcome everyone
    By jc101 in forum Introduce Yourself
    Replies: 12
    Last Post: June 13th, 2003, 06:32 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts