Results 1 to 13 of 13
  1. #1
    Newbie
    Join Date
    May 1st, 2009
    Posts
    28
    Automating datafeed import and building price comparison website
    While searching for a script for automating affiliate datafeed import and building a price comparison website, I came across mostly with script vendors or developers based in UK. I noted nobody of them was form the U.S.

    Now, as I am not English mother tongue, I have the doubt that the sentence I Googled when searching for the script, is ok for the English spoken in the UK but not in the U.S.
    The sentence I Googled and also searched in this forum was: "price comparison script".

    Now I wonder if this sentence is not used by developers based other than in UK, and if U.S. based or international developers use something different to define this family of scripts.

    I need to know that because I would not like limiting my search to UK based software companies only.

    Thanks

  2. #2
    Tax Paying Member
    Join Date
    November 14th, 2005
    Location
    Chapel Hill, NC
    Posts
    2,119
    I think that you will find that most of this work is done in Great Britan.......Not in the USA.

    BTW....Welcome to ABW
    You must climb this mountain. There is no elevator. ---- Don't stick your finger in the liquid nitrogen.
    Carolina China

  3. #3
    Full Member
    Join Date
    October 22nd, 2006
    Posts
    200
    Most of the scripts are Uk based for for a number of reasons.

    In the US the market was dominated by LS, CJ and SAS . LS and CJ charged for feeds so the "hobbyist" affiliates could not make us of their feeds. Larger companies developed the software but kept their software in house . SAS feeds have been covered by a few good tutorials on this site so there makes no commercial sense to develop a script for SAS feeds.

    In the UK however there were many more networks all of which allowed free access to datafeeds. Of course each network had it's own datafeed format and method of delivery. So many people wrote scripts that could bring in feeds from all the networks.

    The US market has gone down the route of using the in house software such as popshops, webmerge golden can etc. The Uk affiliates tend to use their own scripts or buy one of the many available pre-written scripts.


    Regards

    Bob

  4. #4
    Newbie
    Join Date
    May 1st, 2009
    Posts
    28
    Thanks Witzer. As you suggested most of this software is form Great Britain, do you mean to say the business model of "price comparison website" (created directly from multiple Merchant product datafeeds) is ok mostly for UK market?

    I would be interested to know which markets "price comparison websites" are appealing mostly.
    Most of my current websites target mostly U.S. audience, so I wonder if it's worth investing my resources in this business -- with U.S. market in mind.

    Thanks

    Federico

  5. #5
    Newbie
    Join Date
    May 1st, 2009
    Posts
    28
    thanks a lot Donk, I was writing the reply for Witzer while you was publishing your reply, so I am just now reading your informative post.

    I am already registered with Linkshare Merchandising program (250$ fee), and with the other following networks: shareAsale, LinkConnector, Amazon, ebayPartners, and -- for my Italian websites, with TradeDoubler.

    So do you think buying this kind of software is going to make my job easier? Is it worth investing in it?
    I will have a try to popshops and golden can -- are they free?

    Thanks

  6. #6
    Full Member
    Join Date
    October 22nd, 2006
    Posts
    200
    Amazon and Ebay use their own API's so the datafeed scripts may not work with them.

    As far as I know you can get free versions of PopShops and GoldenCan but they make take the commisission for one out every four clicks. They each have their own sections on this forum so best to ask there.

    Affilistore I believe has a free version of their datafeed script you can try out but it may not work with all the networks.

    Regards

    Bob

  7. #7
    Tax Paying Member
    Join Date
    November 14th, 2005
    Location
    Chapel Hill, NC
    Posts
    2,119
    I have used datafeeds from SAS and GAN on scripts from the UK.

    The PopShops and GC are converting the feeds which they receive to an end user product which is the same process that you wish to do. I don't think that you can get direct access to their data feeds for obvious reasons. You will need to acquire datafeeds directly from the network or merchant.
    You must climb this mountain. There is no elevator. ---- Don't stick your finger in the liquid nitrogen.
    Carolina China

  8. #8
    Visual Artist & ABW Ambassador lostdeviant's Avatar
    Join Date
    September 7th, 2007
    Location
    Cuautitlán, Edo. de México
    Posts
    1,725
    just a couple notes.. Popshops charges a monthly fee and the last time i checked their system doesn't include price comparison nor is it really automated. It is designed for making tables of select products from one or several merchants and it is very good for doing that. They have been working on a search based system with an "enterprise" account and there are examples in their sub-forum here.

    Affilistore does work to import and display datafeeds, and if you only have one or two websites it is a great solution. I used it for about 8 months on a few sites. Unfortunately it was hard to set up to automatically import new feeds on a regular basis so I found it frustrating for that one weakness. At least in the last version I tried, you'd have to set up a cron job for every single imported feed on every install (every website). In the end I decided to learn php and mysql for my own scripts although I still don't have price comparison. (you either need a dedicated server or really amazing programming skills, or datafeeds with all the info in them and most datafeeds don't have matching product IDs; even name matching can be complicated since many merchants add or change a word or two for the same product)

    In the end, you can't rely on the price comparison or product catalog software since Google and the other search engines consider anything directly from a datafeed or other similar service to be duplicate content. You'll have to either do ppc or come up with related original content that visitors will want to link to.

  9. #9
    Newbie
    Join Date
    May 1st, 2009
    Posts
    28
    Thanks to all

    ...you can't rely on the price comparison or product catalog software since Google and the other search engines consider anything directly from a datafeed or other similar service to be duplicate content. You'll have to either do ppc or come up with related original content....
    I had a look at GoldenCAN and found in the faq: "...If you use the GetData, GetXML or GetRSS features then you have actual content on your website. We use many methods to make sure that we serve different content to different users such as randomizations..."

    Although, in most of my websites I plan to add price comparison sections linked from pages offering original content, I wonder if "prepending" the items name (or description) created from a datafeed with custom text, I will able to avoid SE duplicate content problem (obviously I would need of a script doing the prepending custom text job).

    ...you either need a dedicated server...
    Do you mean the process is so resources consuming that would make server down (or hosting company could ban this automated processes)?

  10. #10
    Visual Artist & ABW Ambassador lostdeviant's Avatar
    Join Date
    September 7th, 2007
    Location
    Cuautitlán, Edo. de México
    Posts
    1,725
    Quote Originally Posted by federico09
    Do you mean the process is so resources consuming that would make server down (or hosting company could ban this automated processes)?
    That's what happens when only part of a sentence is quoted.

  11. #11
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    LostDeviant wrote (and federico misinterpreted): > "In the end I decided to learn php and mysql for my own scripts although I still don't have price comparison. (you either need a dedicated server or really amazing programming skills, or datafeeds with all the info in them and most datafeeds don't have matching product IDs; even name matching can be complicated since many merchants add or change a word or two for the same product)." <

    There are several issues that impact your resource needs.

    First, if you aren't a very good programmer (and I definitely am proving that I'm not very good), then even your most basic scripts may completely max out your server resources. I've implemented a very crude method to avoid this: my import script simply "sleeps" for 1 second after every NN inserts (I keep tweaking the number).

    But I think what LostDeviant was specifically referring to was the need to do some very sophisticated searching and matching to do price comparisons properly, because the data you need to match might be in several different fields and perhaps might not be present at all; you might need to actually do "iterative" searches, first to find records that match the initial search, then to try to find "similar records" that might not include the same keywords. (For example, if the consumer searches for the "Welch Kazoo Crusher," a search might turn up several different models (AR-1, AR-2, CX-7) and other manufacturers might not include the full title (e.g. "Welch AR-1" or "AR-1 Kazoo Crusher" or "AR-1 Crusher.")

    You might deal with this problem via pre-processing (applying a very extensive set of complex filters while importing the data) or via real-time filtering (running multiple searches and filters at the time the user searches). Either way will demand a lot of CPU resources, plus a lot of disk activity.

    I'm not planning to do "price comparisons," but my current import scripts need many hours (while using 30% to 70% of available CPU resources) to import a million records (my test database now includes 2.5 million products from 600+ merchants), and for testing purposes I removed much of the filtering that I'll definitely want before implementing the "live" system on my sites. And during testing, I'm excluding the larger feeds (merchants with hundreds of thousands or millions of products in their datafeeds).

    Of course, your mileage will vary, depending on how many merchants, and which merchants. If you're just going to work with merchants and products where an ISBN or UPC is available, then your CPU requirements would be much lighter.

    I am currently using a Virtual Private Server (VPS) account (currently, the $50-per-month eApps account), but I certainly recognize that my project (even without price-comparison features) is going to need a dedicated server -- and not a $69-per-month dedicated server, but something more substantial.

  12. #12
    Newbie
    Join Date
    May 1st, 2009
    Posts
    28
    markwelch , when I introduced myself at the start of this thread I did want to make it clear first that English is not my first language (I am Italian and based in Italy).
    So, I apologize with all of you if I accidentally misinterpreted some of your sentences.

    Thanks for your patience.

  13. #13
    Visual Artist & ABW Ambassador lostdeviant's Avatar
    Join Date
    September 7th, 2007
    Location
    Cuautitlán, Edo. de México
    Posts
    1,725
    Quote Originally Posted by markwelch

    But I think what LostDeviant was specifically referring to was the need to do some very sophisticated searching and matching to do price comparisons properly, because the data you need to match might be in several different fields and perhaps might not be present at all; you might need to actually do "iterative" searches, first to find records that match the initial search, then to try to find "similar records" that might not include the same keywords. (For example, if the consumer searches for the "Welch Kazoo Crusher," a search might turn up several different models (AR-1, AR-2, CX-7) and other manufacturers might not include the full title (e.g. "Welch AR-1" or "AR-1 Kazoo Crusher" or "AR-1 Crusher.")
    That's exactly what I was thinking! I couldn't have said it better, and with my shared hosting, I probably never will implement any real price comparison. Even optimizing my mysql queries I wouldn't be surprised if one of my remaining shared hosting accounts were closed due to the extra server load caused by price comparison (when implemented).

    Even with just a regular product catalog and a basic keyword search, I actually filter quite a lot of data (on datafeed import). Most merchants really have poor quality feeds from odd characters to empty columns to prices set at $0.

  14. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Featured: FTC disclosure for price comparison website
    By aocsteve in forum Midnight Cafe'
    Replies: 1
    Last Post: March 21st, 2014, 11:42 AM
  2. Replies: 5
    Last Post: February 26th, 2012, 09:00 AM
  3. Building and Offering a Small On-Site Product-Price Comparison "Engine"
    By BeepBeep in forum Marketing Resources & Power Tools
    Replies: 7
    Last Post: March 21st, 2010, 09:24 PM
  4. Datafeed for a dutch price comparison website.
    By vistavision in forum Programming / Datafeeds / Tools
    Replies: 0
    Last Post: September 15th, 2009, 03:14 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •