Page 1 of 2 12 LastLast
Results 1 to 25 of 42
  1. #1
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    Datafeed Studio? Any feedback?
    I am considering testing out Datafeed Studio, which appears to provide "some" of the functionality I've been looking for -- but of course I'm not a Linux/PHP person so I am very reluctant to invest the time required to evaluate it fully in order to identify what additional work would be needed to make it fit my needs.

    Can anyone share any experiences or feedback about Datafeed Studio? I'd especially like to hear from anyone who's used it as a starting point for a larger effort. If you considered using Datafeed Studio but decided not to, I'd also like to hear why.

    Thanks for any advice.

  2. #2
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    Okay, it's starting to feel like I'm the first guinea pig for this product.

    I went ahead and plunked down my money (US$465 = 229GBP) to pay for registration (79GBP) as well as installation (100GBP) and creation of a ShareASale 'parser' (50GBP). I've set up a demo/testing site at "merchant dash product dot com" if anyone wants to poke at it.

    I've immediately identified a "problem issue," which is that the system absolutely does not like "www." prefixes on domains -- there seems to be some hard-coding that eventually redirects back to the non-www domain.

    I'm still trying to understand the core 'model' or 'metaphor' used by this system; it uses templates (apparently modified from a format similar to WordPress), but my attempts to modify the templates have failed miserably.

    The author is quite responsive, so far, but has implemented a truly dreadful trouble-ticket system which I'm simply refusing to use.

    So far, I've identified a number of issues, mostly problems that arise whenever I try to do "more than one" of something. For example, the system allows for "filters" when importing, but either it allows only ONE filter (substitution) per field, or it won't display or permit editing of multiple filters. All editing is done manually, one record at a time. When editing product data, you must manually click to open a separate window to view a product image or destination page -- so every product edit requires multiple extra clicks to do what I expect I'll need to do every time.

    When defining each datafeed, the system requires that you MUST upload a logo for each merchant.

    There is no way to upload a list of new datafeeds -- you must manually enter them one at a time (so I'll need to cycle through the same form 509 times to set up datafeeds for 509 ShareASale merchants).

  3. #3
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    Surprising performance (good)
    OK, I am pleasantly impressed.

    I've currently imported 52 ShareASale datafeeds, totalling 1,248,805 products -- and performance seems quite good when I search for products.

    Considering that I'm testing Datafeed Studio with a new $10-per-month eApps VPS account (with only 160MB memory allocated), that's very nice.

  4. #4
    ABW Ambassador 2busy's Avatar
    Join Date
    January 17th, 2005
    Location
    Tropical Mountaintop
    Posts
    5,636
    I took a quick look and it's not bad for 'just out of the box' but it needs a lot of adjusting to be useful. I'm in the market for a Garmin 305 so I asked for a price comparison. It pulled up 76 pages of 9 products, buried somewhere in those pages may be the 305 but it returned everything with Garmin anywhere in its name apparently. The template says it will do a price comparison among UK merchants, but all the prices are in USD. Things to play with in settings I would guess as well as the number of results returned and some CSS for the template. Pagination would be preferable for that number of results, but even better would be results that only pertained to the searched product specifically.
    I only did that one search, and I understand that it's only been up a matter of minutes.
    Good luck with it

  5. #5
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    Yes, indeed, the search function seems to be pretty erratic; the internal search within the admin interface is much better, so I assume the issue is that the default setting might be overbroad or fuzzy.

    But it gets worse: no matter what I do, if I search for "305" it never brings up ANY responses. And yet the Garmin 305 is actually present in the database (from Overdrive). (There appears to be a non-standard character embedded in the product description - perhaps MySQL or the script view this character as an end-of-field marker, even though the additional text is stored in the field.)

    > The template says it will do a price comparison among UK merchants, but all the prices are in USD. <

    I'm not sure which site you're looking at. My site has USD (US Dollars) as the default currency - did I neglect to edit some text somewhere that refers to GBP?

    The author's example site at Olaxi dot com shows prices in GBP (Great Britain Pounds), since he is in the UK.

  6. #6
    Analytics Dude Kevin's Avatar
    Join Date
    January 18th, 2005
    Location
    Rochester, NY
    Posts
    5,904
    And there's the rub, Mark, IMHO.

    Of all the data feed tools I've tested, only one thing becomes clear with each and every one. Merchants that do odd things in data feeds hurt their affiliates.

    I'm not saying datafeed standardization is the biggest issue this industry faces, not by a long shot, but it sure would be nice if they would just stop producing crappy ones.
    Kevin Webster
    twitter: levelanalytics

    Kayak Fishing
    Web Analytics and Affiliate Marketing

  7. #7
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    It now appears that abut one-third of all product records are NOT being imported by Datafeed Studio -- and this occurs with no error message or warning. The interface shows the "Total Products" in each merchant's datafeed (these add up to 1,783,978 from 99 merchants, in my current settings), but actually imports many fewer records (in my case, only 1,182,818 products are actually stored in the database).

    To complicate matters, the error logs are not being stored in the right place (perhaps they're not being written at all), and my Linux/Apache/MySQL/PHP skill level is best described as "clueless," so I'm unlikely to diagnose the problem myself. The author has been quickly responding to my emails but I expect that since he's in the UK, he's probably ready to call it a day.

    I definitely agree that there are a lot of issues with datafeed content, including simple corruption of data, non-standard characters, mis-translation of characters (e.g. storing product names as "Niketm," or describing a product's size as "11 4 inches" instead of "1 1/4 inches"), embedded HTML, promotional messages embedded in all product titles ("FREE SHIPPING FOR ALL ORDERS OVER $50"), and a wide range of other issues.

  8. #8
    Affiliate/AM Moonlighter dflsports's Avatar
    Join Date
    January 17th, 2005
    Posts
    874
    I wonder if the script "scrubs" the data, eliminating duplicate records? Some merchants have feeds where they have the same product listed 8 times, each for a different shoe size or shirt, etc...

  9. #9
    Analytics Dude Kevin's Avatar
    Join Date
    January 18th, 2005
    Location
    Rochester, NY
    Posts
    5,904
    DFL: BassProShops KILLS me by doing that. Each lure with 37 references for colors... but all with the same picture, description, etc. Just a different SKU

    I would welcome that filter.
    Kevin Webster
    twitter: levelanalytics

    Kayak Fishing
    Web Analytics and Affiliate Marketing

  10. #10
    Newbie
    Join Date
    May 29th, 2008
    Posts
    3
    Hi,

    I'm the author of Datafeed Studio and am happy to answer any questions here or via support@datafeedstudio.com .

    Some points I'd like to make as a result of the discussion so far :

    - www/non-www : DS supports multiple sites running off the same data, and it is recommended that for each you nominate either the www. or non-www version as the URL of a site, and let Apache redirect requests for the non-www to www (or vice-versa).

    - templates : yes, DS does provide a similar concept to Wordpress in terms of themes, and has a built-in editor. Mark, I'm still awaiting information as to what problems you're having editing templates.

    - "dreadful trouble-ticket system" : I use FogBugz, number one in Google for "bug tracking software" and generally acknowledged as one of the best out there.

    - logo for each merchant : yes, a lot of the DS themes utilise the merchant logo as a visual aid to improve conversions so that is why it is a mandatory field. You can always upload a dummy image and remove mechant logos from the theme if required.

    - list of datafeeds : correct, merchant datafeeds must be added one at a time, I think Mark's requirement to add 509 different merchants in one go is a bit extreme, but I've suggested a solution to him.

    - searching : as communicated to Mark, this looks like a problem of the 'ft_min_word_length' setting of his MySQL server, it needs to be changed to match that of DS to return correct results. Numeric searching does work, for example the 305 search works here : http://dealdrop.co.uk/?q=305&order=d...&go=Compare%21

    - USD/GBP - the currency symbol is a configurable value, it's just that the default Price Compare template says 'UK merchants'. You can of course change this.

    - one filter per field : incorrect, you can have multiple filters per field.

    - error logs, there looks like there may be a file permission problem with Mark's setup that prevents the logs being created which we need to resolve, but DS does log the import process, including any records that failed.

    I'd like to thank Mark for being the first US based "guinea pig" for this software, especially given his aversion to LAMP based solutions :-)

    He has some pretty demanding requirements for his 'perfect system' that I have to balance with ease of use for the average buyer who typically just wants to build a niche online store or price compare engine from just a dozen or so merchants.

    Ultimately Datafeed Studio is still only a few weeks old and I'm sure there's a few wrinkles to iron out, but if anyone has suggestions for what they would like to see in future releases, I'm all ears.

    Regards,

    Martin

  11. #11
    Affiliate/AM Moonlighter dflsports's Avatar
    Join Date
    January 17th, 2005
    Posts
    874
    how about some discounts for ABW members


    (Yes, I'm cheap)

  12. #12
    Analytics Dude Kevin's Avatar
    Join Date
    January 18th, 2005
    Location
    Rochester, NY
    Posts
    5,904
    I'm checking out the trial version.

    I'm waiting for a quick update to IonCube on my server though. Should be playing soon.
    Kevin Webster
    twitter: levelanalytics

    Kayak Fishing
    Web Analytics and Affiliate Marketing

  13. #13
    ABW Ambassador ladidah's Avatar
    Join Date
    October 15th, 2007
    Location
    MA
    Posts
    1,888
    mark, that is going to be a monster of a database. Is that on a dedicated server? I tried your site too and interestingly as you said, the www and the non-www versions looked very different. I did a search for a random product "sleigh bed". First without quotes then with quotes and search results turned up anything from sleigh beds to cold cream, to scrapbooking material But as martin said, maybe it's your settings. We'll see...thanks for sharing.

    I too, will check it out.

  14. #14
    Believe knight01's Avatar
    Join Date
    August 14th, 2006
    Location
    Dayton, Ohio
    Posts
    1,815
    Quote Originally Posted by markwelch
    But it gets worse: no matter what I do, if I search for "305" it never brings up ANY responses. And yet the Garmin 305 is actually present in the database (from Overdrive).
    Mark,
    It's been awhile since I looked into this, but I seem to recall mysql ignores words in queries with 3 characters or less if it is a full text search.

    I'll be starting a script tomorrow where I'll need to brush up on queries and am hoping snib and Patrick are close by. But I'd investigate what type of index is set for the fields the script is searching.
    Someday starts today
    Military Discounts

  15. #15
    Newbie
    Join Date
    May 29th, 2008
    Posts
    3
    Yes, the 'ft_min_word_length' setting of MySQL controls this.

    DS tries to use full-text searching if possible, but if one of the search terms has a length less than ft_min_word_length it does a fuzzy match instead (which is less efficient).

    Mark's searching problem is that DS expects MySQL to have a word length of 3, but his MySQL server is setup for 4.

    He can either change the MySQL setting or update the DS config file to tell it a word boundary is 4 characters - but they must be set to the same value to avoid empty results being returned.

  16. #16
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    On the issue of Datafeed Studio not importing one-third of the product data: it turned out that this is ShareASale's fault.

    In March 2007, I wrote: > "[ShareASale] did confirm that there are "some" merchants whose datafeeds don't have a unique ProductID, nor are these datafeeds processed or validated in the usual way. // This is definitely a distraction; without a unique/distinct ID which persists through updates, I can't automate the propegation of changes from the datafeed out to my web content. // I did look closely at one of these feeds, and determined that the filename part of the URL appears to be the sku, which may be unique; I'd need to spend some "quality time" with each of these feeds four feeds to confirm whether that's a workable solution.

    For the moment, I am simply excluding these four merchants [11526, 12109, 8780, 5733] from my current site-development projects. Since these four merchants do represent one-third of all product rows in my 383-merchant database, I definitely can't ignore them forever, but I am definitely not investing my time on these datafeeds before March 23.
    " < http://forum.abestweb.com/showthread.php?t=86841 (March 2007); see also http://forum.abestweb.com/showthread.php?t=93877 (August 2007)
    Since Martin created the ShareASale parser for Datafeed Studio only at my request, and since I'd forgotten about this issue, his decision to use ProductID as if it were a unique-identifier was entirely reasonable. And here we are again, forced to find some kind of work-around for what I absolutely believe is a mistake by ShareASale.

  17. #17
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    Quote Originally Posted by mwd2000
    [Mark] can either change the MySQL setting or update the DS config file to tell it a word boundary is 4 characters - but they must be set to the same value to avoid empty results being returned.
    I have absolutely NO IDEA where to look to change a setting in MySQL. I've loaded and used PHPMyAdmin, and sure enough I can SEE that the setting is 4, but I don't see any way to edit it from there, nor anywhere else. I've fished through many different directories looking for a file, and viewed a number of MySQL-related files, but I've not yet found any way to change the setting for ft_min_word_length -- and I must emphasize, this is the kind of trivial-question-which-takes-six-full-hours-of-work-to-answer that has led me, time and again, to refuse to use Linux/LAMP, because I need good documentation and/or technical support, and what you get for most Open Source products are widely scattered materials of widely varying quality.

    I search and search, and all I find are suggestions that I "change" the variable, but after many searches, I still can't find the right file, nor any useful OR trustworthy information about how to change a variable in MySQL. (See, e.g., http://dev.mysql.com/doc/refman/5.0/en/set-option.html).

    I also found many references suggesting that after I do change this variable, I will need to "rebuild my fulltext indexes," but again that's something I don't know how to do.

    I've gone searching for MySQL support; apparently Sun Microsystems offers some kind of MySQL Enterprise solution that includes technical support, but they don't post any pricing information on their web site, so I assume it's priced for large corporations -- I've left a message with their sales voice mail, bbut don't expect anything to come from that.

    I don't want to turn this into a Linux-bashing thread (just this one Linux-bashing post should be enough); I just want to move forward.

    I'm willing to pay for competent help. Any advice?
    Last edited by markwelch; June 25th, 2008 at 10:24 AM.

  18. #18
    Believe knight01's Avatar
    Join Date
    August 14th, 2006
    Location
    Dayton, Ohio
    Posts
    1,815
    Mark,
    I did not see if you are running this on a dedicated server or shared hosting of some type.

    If you have root access look for this.
    /etc/my.cnf

    Open it in pico and see if you find ft_min_word_length located in it. if it is it should say =4 based on what you've posted. Change that to 3 and restart the mysql daemon.

    If you have a shared hosting setup or don't have access to root, you'll need to ask the hosting company to do this.

    Source: http://64.233.167.104/search?q=cache...ient=firefox-a

    filename is discussed on page 4 and varible on page 7.
    Last edited by knight01; June 25th, 2008 at 11:17 AM. Reason: added source
    Someday starts today
    Military Discounts

  19. #19
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    At first, I wrote: > Thanks for the advice -- I had already checked, and no such file exists. I don't know what 'pico' is (probably a text editor?). I am using a VPS (eApps). <

    But of course I was mistaken -- there is a file, but it does not contain the field name specified, and I don't feel comfortable adding a new command that I don't know for sure how to do.

    I finally reached a MySQL sales person at Sun, and he readily acknowledged that even if I paid $2,000 to $5,000 for a support contract, their support staff would probably just point fingers back at "whoever configured the server" and would not be able to help me.
    Last edited by markwelch; June 25th, 2008 at 11:37 AM.

  20. #20
    Believe knight01's Avatar
    Join Date
    August 14th, 2006
    Location
    Dayton, Ohio
    Posts
    1,815
    Quote Originally Posted by markwelch
    Thanks for the advice -- no such file exists.
    is there a my.ini? I added a link to my source above for your review.
    Someday starts today
    Military Discounts

  21. #21
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    Wow, I just posted an ad on Craigslist looking for an "on-call" LAMP person. The first response was from a guy who had more trouble than I did finding basic information; zip. Second response: an unintelligible voice-mail message from someone who doesn't seem to speak English. Third response: an email from someone who clearly knows exactly how to do things, but can't explain (in answering one question, it starts, "Get into your terminal application" and rapidly gets more confusing from there).

    This is very, very frustrating. I've just spent three full days working with Datafeed Studio, and I am exactly where I was in March 2007 -- I have a script that can import some (but not all) datafeed files into a raw database; I can't diagnose any of the problems I'm having; I can't move forward in any meaningful way; I'll need to spend hundreds of hours to learn the core technologies to be able to use this and THEN start moving forward. The good news is that this time, I'm only out-of-pocket about $600.

  22. #22
    Believe knight01's Avatar
    Join Date
    August 14th, 2006
    Location
    Dayton, Ohio
    Posts
    1,815
    Quote Originally Posted by markwelch
    At first, I wrote: > Thanks for the advice -- I had already checked, and no such file exists. I don't know what 'pico' is (probably a text editor?). I am using a VPS (eApps). <

    But of course I was mistaken -- there is a file, but it does not contain the field name specified, and I don't feel comfortable adding a new command that I don't know for sure how to do.

    I finally reached a MySQL sales person at Sun, and he readily acknowledged that even if I paid $2,000 to $5,000 for a support contract, their support staff would probably just point fingers back at "whoever configured the server" and would not be able to help me.
    Since you have a vps you may not have access to the main mysql configuration file, although the file you found should be used to override the main configuration. If you don't feel comfortable adding the line. You may be able to ask your host to do it. I presume you submitted a support ticket to them. They will likely charge you, but it sounds like you'd be happy to pay a little to get this resolved.

    A second option if the hosting company is not willing to do it. Many years ago I had a dedicated server at a company called rackshack which became ev1servers and finally bought by theplanet. On rackshack forums there were several server admins who volunteered to help those of us who were technically challenged. Of course they also offered admin services for a fee. One, I recall, went on to start webscorpion.com and I just checked it looks like he is still doing consulting and management. If all else fails submit a request to him and see if he would be willing to help with offsite vps management. I've not talked with him in years, but he always hit me as a stand up guy.
    Someday starts today
    Military Discounts

  23. #23
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    1,670,520 products and counting
    As I have continued to play with Datafeed Studio, I've now imported about 300 ShareASale merchant datafeeds (all the "import-able" datafeeds from "my" merchants with 2,000 or more products in their datafeeds; I have excluded 898,474 products from 31 other merchants, including 14 whose datafeeds don't have unique ProductIDs, 7 whose datafeeds wouldn't load properly for other reasons, and 10 merchants whom I've excluded for various reasons, even though I'm enrolled in their programs).

    According to Datafeed Studio, the imported files contain 1,992,725 products, but Datafeed Studio's product database only contains 1,670,520 products (a loss of 322,205 products, or 16% of the total). As noted in another thread, much of this "loss" appears to be related to ShareASale merchants who do not have unique productIDs in their datafeeds.

    Even with 1.67 million product records, searches seem pretty quick to me. Earlier, I had absolutely rejected the idea of dynamic pages for my new sites, but now that I'm seeing the fast performance on the VPS I'm using (and inferring much faster speed if I switch to a dedicated server), I think I might be able to save a lot of effort by sticking with dynamic pages (though search engines would see the pages as static html).

    I am still trying to figure out how to re-index the full database after changing the ft_min_word_len setting in the my.cnf file from 4 to 3. At least a dozen people have told me to just execute a particular command using the command-line interface, but nobody will tell me how to access the command-line interface.

    I finally broke down and bought Luke Welling's book, PHP and MySQL Web Development (3rd edition, 2005), plunking down $54.36 at Border's instead of waiting until Monday or Tuesday to get the book from Amazon for $24 less. I expect to do a lot of reading in the next few days.

    Hey, Kevin: how is your DS trial going?

  24. #24
    .
    Join Date
    January 18th, 2005
    Posts
    2,973
    Ending my test of Datafeed Studio
    It appears that my testing of Datafeed Studio has just come to an end.

    My server just stopped responding (it's generating "http 500 internal server error" to all requests). And since I am using a Linux/Apache system that I don't know anything about, I cannot diagnose the problem, and the VPS provider (eApps) has responded to my last two support queries with one-sentence responses saying "please provide more information," when there is nothing more I could possibly explain.

    I have cancelled my VPS subscription for this test account, and obviously can no longer do any testing of Datafeed Studio since this installation will be wiped out.

    Datafeed Studio is a "solid but limited" application. It probably works very well for those people who are satisfied with its limitations and who either do not need to make any changes to the program or templates, or who have a much richer understanding of PHP and the template system used.

    As currently implemented, Datafeed Studio doesn't seem to provide any value or benefit that PopShops doesn't already offer, apart from the obvious benefits of a self-hosted database instead of a remote service.

    I'm sure that Martin will continue to work on the system, hopefully implementing many of the suggestions I sent him this week, and I'd expect that the program will be much more useful in a few more months.
    Last edited by markwelch; June 28th, 2008 at 03:45 PM.

  25. #25
    Full Member iolaire's Avatar
    Join Date
    October 3rd, 2006
    Location
    Arlington, VA
    Posts
    229
    Quote Originally Posted by markwelch
    (it's generating "http 500 internal server error" to all requests).
    I know it is to much to learn a new system - but just an FYI - one of the most common cause of those errors are from running out of RAM or processor resources. Maybe the system was overloaded importing data and could not serve the web pages.

+ Reply to Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Featured: Webgains New Publisher Studio
    By Ryan Gilbert in forum Webgains.com
    Replies: 0
    Last Post: April 4th, 2014, 10:08 AM
  2. Datafeed Studio - impressions
    By trienthusiast in forum Programming / Datafeeds / Tools
    Replies: 0
    Last Post: July 30th, 2010, 10:30 AM
  3. Looking for Feedback on our Datafeed
    By Link at Art.com in forum Programming / Datafeeds / Tools
    Replies: 6
    Last Post: July 15th, 2004, 01:47 PM
  4. Toledo Swords Datafeed - need feedback
    By loveleaf in forum Midnight Cafe'
    Replies: 28
    Last Post: November 4th, 2003, 05:02 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •