[markwelch]

I'd like to offer some suggestions about functionality and features that would make "datafeed services" more useful and worth paying for. Hopefully other folks can chime in with additional feedback and suggestions.

Why don't existing services work for me? Because they require intense manual effort to define each individual page, and then recurring effort to check and update each page over time. If I create a web page that I want to include a list of product links, I don't want to be required to spend 10 minutes every month to "maintain" the product-results-list for that page. (Spending 10 minutes per month per page, for even 100 pages, is 1,000 minutes per month, or 16 hours per month. If I have 1,000 pages, that's 160 hours per month. And no, that's not unreasonable: I currently have 988 defined PopShops 'shops.' If I had the right tool, I'd have many thousands of active pages with product-results embedded.)

I recognize that no solution can completely eliminate maintenance work -- but if something can be automated, I want to see it automated.

What's wrong with the current datafeed tools available?

(1) Merchant selection: Goldencan claims to include 170 merchants in its datafeed service (including both free and fourth-click). Popshops and Datafeedr claim thousands. However, there is no easy way to "synchronize" or manage merchants as I add or drop programs.

(2) Updating: The Datafeedr service doesn't update at all; you must manually download and import from their site to yours, any time any change occurs in the data (according to the support email I received this morning). PopShops dynamically updates "somewhat" but nearly all my PopShops stores shrink and eventually empty out over a period of months as datafeed changes happen.

(3) Intense manual configuration and management of individual "pages" of results, without any workable way to dynamically generate pages.

What I Want:

(A) Automation of Merchant Management: First, I need my "datafeed tool" to automatically detect TWO status settings for each merchant relationship: First, am I approved with that merchant through this network? Second, is the merchant's affiliate program "active" (e.g. not "temporarily inactive" on SAS, or "deactivated" on CJ)? This way, no links should ever be displayed on my site unless there I have an active relationship with the merchant. (When I use the online interface, it might simply exclude merchants and products where I have no active relationship, or perhaps it could display those merchants and products "grayed out" so I'd see what I'm missing.)

(B) Selection of Merchant "Groups" in one of two ways: either "all my merchants EXCEPT..." or "only these specific merchants..."

(C) Dynamic search results: I do NOT want to be required to manually log in to an online interface (such as PopShops' excruciatingly-slow interface) to manually define each "shop" and manually select products. Instead, I want to be able to dynamically create a query to populate a "shop" or search-results page. Such queries MUST allow "required" (AND), "optional" (OR), and "negative" (NOT) keywords.

(D) I'd also like to be able to view reports that show how many "impressions" and "clicks" occurred for each page or search-query, and reports that would show changes in the search results (e.g. noting that there were 28 results for this query last month but only 7 today).

(E) I'd definitely like some filtering capabilities, ideally including some sort of "community filter" system so that if I decide that the text "FREE SHIPPING ON ORDERS OVER $25" should be removed from all product titles, I could click a box so that others could see and select that same filter in the future.

How would I use this? Quite simply: when I write a content page on my site about Disney scrapbooking, I'd like to be able to insert a snippet of code that would include a search query along with a merchant-grouping (e.g. "merchant_group=scrapbooking&query=disney_scrapbook*").

[markwelch]

In response to my support email (after I replied to the auto-reply, in order to get my support email read), PopShops support wrote back:

> "As for your question, it may be that the Data Pack API is the functionality. You create a catalog of merchants and then you or a developer creates queries. It returns XML." <

The implication is that the support person doesn't know whether the API can perform any of the functions I identified in the support query.

I know I've looked at the API page at http://www.popshops.com/faq/affiliate-xml-api before this weekend, and I've looked again yesterday and now again today -- but I still can't tell what functionality is actually offered by the API. No documentation is available unless I sign up for the API service -- in other words, I must sign up (and plunk down my credit card and authorize a $324 charge) before I can see the documentation.

From the sample code snippets, there appears to be some search functionality, but it's not clear what search functionality is provided (for example, AND, NOT, and OR functionality, as well as exact-phrase matching).

Has anyone used the PopShops API, and if so, does it provide meaningful search functionality?

[Tookings Rick]

Mark, I've read some of your older posts about attempting to have what seems to be a similar software written as well (or at least, certain aspects of it).

Say you had a piece of "magic middleware" that (a) interfaced with "most" networks + the datafeed aggregators and then (b) had some sort of front end interface so you could do the powerful/easy page drop-in feature you mentioned above. (And for (b) you could also define groups in the software...so you could manage changes either on the page side (a la "merchant_group=scrapbooking&query=disney_scrapbook*") or from the software side for easy updates.) With a good plug-in interface for the display side so you could customize as much or little as needed...friendly for 3rd party developers, etc.

Assuming it worked well...and it was a hosted service...what would you pay per month? (Do you think both a paid and "4th click" model would work, as options?)

(I'm more curious than anything...if our next project gets the go ahead, we'll be pretty swamped for a while on the dev side. But this kind of thing has always interested me personally...often surprised it hasn't already been executed well.)

[markwelch]

I don't really know how much I'd be willing to pay per month for this type of service -- certainly I don't see the current fees of $60 or $100 monthly fee as any obstacle at all, for delivery of workable search-results from affiliate-program data. I've spent hundreds of hours and many thousands of dollars over the past three years on development work, ultimately for nothing at all.

But as I noted in another discussion thread a few minutes ago, the only service which appears to come within a mile of meeting my needs is PopShops' Data Pack API, which charges $59 per month but imposes a limit of 5,000 API calls, or 1.2 cents per API call, which means I'd need to spend $1.20 eCPM just for that service. I could reduce that expense by using caching intelligently (e.g. never executing a particular query more than once in 24 hours); I also assume that the rate would drop with volume. However, any analysis of PopShops is moot , since they've suggested that I should simply stop using their service, and stop complaining.

The issue (in determining whether there might be a viable business model) isn't really "what would I pay," but what would different "types" of web publishers pay for this service, at different delivery levels. I don't know how many publishers currently pay PopShops at each pricing tier; I don't know how many publishers pay Datafeedr at each pricing tier. My suspicion is that both firms are very, very far from earning profits -- and I'm really not sure if their current business models are workable. But I haven't "run the numbers" since I don't want to sell these services; I want to buy them.

Right now, I don't see any services that would meet my "search results" requirements (even ignoring all the other functionality I'd like to have in an "affiliate datafeed solution").

[ArtDeco]

Mark

This is my first post here. I was googling for datafeeds and discovered this forum, and read several of your previous posts about your problems with feeds. I am a retired programmer (with heavy SQL database import/export experience ) and thought that surely someone would have solved this particular problem by now. Apparently not, if your experience is typical.

I just downloaded the nerds csv and the trial of datafeed studio and a similar wordpress plugin to play with, but I really don't expect much from either looking at their websites - those products look like they are designed for tens of thousands of records, not hundreds of thousands or millions.

I'm sorry that I can't help you with a Microsoft solution, but once I retired and don't need to work around their bugs anymore, I don't use any MS products at all - not even on the desktop.

Anyway this really shouldn't require all that much hardware or really advanced programming - I've imported millions of records at a run on desktop level machines many times before. I expect to cobble something together in php and mysql if I don't find something that works well out of the box.

I am too new here for PMs, but can be reached at princediscourse AT gmail.com if needed.

[thegyppo]

Hey Mark, sounds like you just haven't found the right programmer.

All of the above is very possible, just tricky since you're working with a large number of networks that do things differently.

A) Unless the Network provides this information via an API (i.e. like CJ) you'll likely need to scrape the data from the merchant info page. In your backend make sure you allocate a Merchant to a particular network.

B) You could create categories or "tags" & group merchants together, that should be pretty straight forward.

C) Pretty easy here, you'll need to pull in all the CSV's from Networks into a Database & update it daily (or perhaps more frequently). Then you can really do what you want with that data. The problem here is that you'll need to make the data consistent with REGEX/Other Rules/Filters.

D) You can track this pretty easily in Analytics, the advantage there is that you can relate it to conversions also. So for example we track outclicks to a merchant as a goal & use that data to see which merchants may get the most traction.

E) This is pretty easy also.

This idea sounds ripe for a Wordpress plugin or something similar. It's a bit too customized to develop it into a popshops alternative however that's a whole other discussion.

My advice is to hit up a Ruby on Rails Developer, makes most of the stuff above pretty easy.

[markwelch]

I've been trying to organize my thoughts about what "features and functionality" I think I'd like to see in a "datafeed intermediary." Here is a quick list of features which don't seem to be included in existing services.

Merchant Management (by Publishers)

• “Choosing merchants” – view merchant lists, filter and/or sort by criteria, select merchants for inclusion and exclusion, assign rankings or priorities for merchants. (Affiliates should be allowed to "select" merchants with whom they don't yet have an approved relationship, but of course that merchant's products and links wouldn't be shown until the relationship is approved.)
  
• “Identifying program terms & conditions” – not just during the “choosing” process but on an ongoing basis as changes occur. Payment rates are a critical component of this.
  
• “Identifying and Alerting Affiliates to Program Changes” – automatic notification (via email, IM, text, or online interface) when specific changes occur (e.g. payout rate, program offline, relationship terminated).
  
• “Monitoring merchants’ ecommerce web sites” (automated, manual) – standard evaluative data (e.g. does the site have “About Us,” “Contact Us,” and “Privacy Policy” links? Is the merchant’s home page “broken” or 404?)
  
• “Monitoring merchant relationships” (active or offline/deactivated; approved, pending, or declined) – download data from the affiliate network (or merchant’s technology) to confirm that there is an active relationship between publisher & merchant, and that the merchant’s program is active and not “temporarily offline” or “de-activated.”
  
• “Confirming affiliate-transaction tracking and reporting” – validating that tracking is working, and that transactions are being reported.
  
• “Communicating effectively with affiliate managers” – maintaining valid contact info (affiliate manager identified, with name & email address, perhaps phone number)

Datafeed Management

• “Datafeed updating” (identifying datafeed updates; downloading & adding to database; applying algorithms to imported datafeeds)
  
• “Managing (mapping, merging, splitting) datafeed fields” (during import)
  
• “Filtering and Homogenizing Datafeeds” (pre-import, during import, post-import, real-time delivery)
  
• “Categorizing Products”
  
• “Validating links”
  
• “Validating images” (removing “no image available” images and checking at random intervals for 404-not-found images) – maybe track image sizes; maybe notify affiliates on image changes.
  
• “Validating product availability” – make sure product page isn’t 404, and does not say “out of stock” or “not available” or “on backorder” (this would be very tricky to automate)

Standardization for Intermediary Access (Without distinct access, intermediaries would likely use publishers' logins and "scrape" data)

• “Datafeed access for intermediaries” – probably separate login ID/password/API key for intermediaries, to access datafeeds
  
• “Reporting access for intermediaries” – this would require individual approval by web publisher to enable access. Merchants should probably be given a “veto” power on intermediary access to reporting (but publishers would likely avoid such merchants).
  
• “Validating merchant-publisher relationships and publisher access to datafeed content.” For some networks (e.g. ShareASale), this is easy: just plug the affiliate ID into the a link from the datafeed, and see if it works properly (note that this isn’t as simple as it might ap some merchants may have many domains, and publishers might configure their “non-working” links to redirect to a separate domain. If merchants (or networks) want to limit datafeed access only to publishers whom the merchant has “approved” (e.g. granted FTP access, on ShareASale) this could be much more complex.
  
• “Authorization for datafeed content filtering/mapping/modification” – Some merchants may insist that datafeed content not be altered, or that any alterations be approved by the affiliate manager.
  
• “Access to PPC restriction info” – This is critical, not only for “direct-to-merchant PPC Affiliates” but also for other web publishers who use PPC to attract traffic to their own sites. Somehow, the affiliate must learn which keywords are restricted, and what specific restrictions apply.
  
• “Product, Category, Coupon, Text-Ad, Image-Ad Links” – Traditionally, “datafeeds” are for individual product links or for “coupons and deals.” However, it should also be possible for the intermediary to aggregate other types of links, including merchant-created image ads (banners & buttons), text links, category links, and perhaps other merchant-site content (articles, reviews, blog entries).
  
• “Access to transaction data” – This is likely to be a huge issue, as merchants, networks, and publishers may all view this data as highly proprietary. As a practical matter, this is likely to require manual approval by each publisher, and each merchant should probably be allowed to “opt out” of having this data transferred to the intermediary. Some networks might prohibit this entirely (as LinkShare prohibited AffTrack).
  
• “Access to non-transaction data” (EPC, ranking, standard payout rates, offline history) – Again, this data is likely to be perceived as proprietary by the networks and merchants, although its semi-public availability probably makes it less worrisome. The intermediary should probably honor any “opt-out” requests, but of course publishers would likely view this action by a merchant as troublesome.

[Domains]

Mark, this is a two year old thread, but I just came across it while researching for one of my project needs which is very similar to what you outline here ... and i'm wondering whether and how this space has advanced in the two years since you posted, and are there any alternatives out there today that can now do what you describe here? Thanks

[markwelch]

I am not aware of any significant changes in the past two years.

[nzcid]

Originally Posted by markwelch

I am not aware of any significant changes in the past two years.

This thread makes using datafeeds a hopeless project to work with unless you are really skilled in prgramming

[teezone]

Like any profession, some tasks require more skill than others... using datafeeds definitely requires some technical skills.

It's not hopeless, it's just a steeper learning curve.

[soda]

Originally Posted by teezone

Like any profession, some tasks require more skill than others... using datafeeds definitely requires some technical skills.

It's not hopeless, it's just a steeper learning curve.

Problem with data feeds is not the learning curve, it's the data in data feeds. Depends on the merchant, but some merchants are negligent when they create their data feeds and you have to deal with errors and incompleteness.

[teezone]

Problem with data feeds is not the learning curve, it's the data in data feeds

An unfortunate part of the learning curve is that not all datafeeds are created equal. I was trying to not be too negative

Anyone taking on a datafeed project has to address both the technology, and the (ongoing!) database maintenance. I just don't think you can address both at the same time, it would be completely overwhelming.

A good programmer should be able to create 'exception' reports that can assist in cleaning up the data.. it is a thankless (and very important!) part of the process.

[nzcid]

Originally Posted by teezone

An unfortunate part of the learning curve is that not all datafeeds are created equal. I was trying to not be too negative

Anyone taking on a datafeed project has to address both the technology, and the (ongoing!) database maintenance. I just don't think you can address both at the same time, it would be completely overwhelming.

A good programmer should be able to create 'exception' reports that can assist in cleaning up the data.. it is a thankless (and very important!) part of the process.

So which programmer should I hire to help me set up a site using a datafeed as well as showing me how to maintain the datafeeds going forward.

[BurgerBoy]

You can clean it up yourself with notepad++: Notepad++ Home

And, you can use Price Tapestry to build and maintain a datafeed site: http://www.pricetapestry.com/

That's what I do. It's really not all that hard to do.