Results 1 to 16 of 16
  1. #1
    Schlaumeier cumbrowski's Avatar
    Join Date
    January 18th, 2005
    Location
    Fresno/CA
    Posts
    640
    We talked already a lot about issues with the new CJ Datafeed at:
    http://abw.infopop.cc/6/ubb.x?a=tpc&...9&m=6526020181

    I start a new Topic, because I believe, that Todd is not reading the other one anymore .

    I took my last comment from the old thread, but extend it now with my own experiences with the datafeed file, which I just received today for the first time successfully.

    I imported the 2.3 Gig File with 3.4 Million records into my "play" database to check it further. The first thing coming up were 876871 products in the feed from a "dead" merchant (AlphaCraze) ... nice!

    I also noticed that CJ removed the (CJ) Merchant ID from the Datafeed, which was Integer (Numeric) and also an unique identifier for a CJ Merchant.

    We save the Merchant ID for every CJ Merchant in our Database to be able to identify a CJ Merchant without any doubt and to run efficient queries, statistics etc.
    Its also faster to work with in the Database. DB Servers handle Integer values faster and more efficient (I just say: INDEXES! )

    The new "Key" to identify a merchant in the datafeed is the Merchant Name.
    A filter by Merchant NAME makes it just sloooow (because its a character field) and also causes troubles, when the merchant changes his "name" at CJ (such as from "Merchant ABC Affiliate Program" to "Merchant ABC".

    That was exactly what happened to our no.2 Merchant at CJ.
    We decided to contact CJ, which Merchants we want to include in our Datafeed and wich Merchant not. No wonder, after we received this huge file with a lot of junk in it.

    Picking the merchants is of course a decission based on product quallity/uniqueness and merchant performance (and some more of course).
    Some Stats would help, so...

    I dowloaded some merchant stats from cj and generated some nice reports by cross-linking cj's and our statistical and merchant data. Worked fine.
    I did this less than 2 weeks ago.
    I cross-linked that statistic with the Datafeed information from today (Product Count per Merchant) and removed all Merchants with no Products in the new Datafeed. The elimination was done by MERCHANT NAME.

    Sounds good so far. Problem was, that I eliminated our No.2 Merchant by doing that, because the merchant changed his name at CJ during the last 10 days.
    I just catched that, because I missed that merchant in my final statistic, but found him in the Datafeed and my old Statistic (with different names).

    Todd, that field MERCHANT ID would HELP A LOT (you had it in the old Datafeed!).
    Please, please add it to the Datafeed as separate numeric field.
    What am I saying? ... ADD THIS FIELD TO THE DATAFEED!!!!!!!!!!!!!!!!!!!
    It's not a nice to have, it's a MUST have.

    However.

    I agree to everybody about the waste of resources by providing a huge single file every day. We will pick certain merchants for future Datafeeds, but that will not solve the general issue.

    It does not matter, if you want to work with the file manually, MS Access, mySQL or a full blown multi-cpu/clustered/xx GB RAM/Super Fast xx Terrabytes HD DB Server.

    The Products at CJ get updated by the Merchant differently. From Manual Changes to manual uploads to automated daily/weekly/monthly/random datafeed. That schedule is determined by the nature of the product and the interest of the merchant in his affilate program. The volume of the Datafeed is also different from Merchant to Merchant. Some have 1-100, others thousands or millions (super affiliate Catalog City for example).

    That means for the Publishers (Affiliates), that they also do not have to update everything every day. Publishers should be able to set their own schedule for updating Content for an Advertiser.

    Remark:
    The new Product Catalog with the "dynamic" or "flexible" Categories requires anyway some manual work per advertiser for the publisher, to map useless categories to their product categories). That means, its more than ever necessary for the publisher to look at the datafeed Merchant by Merchant.

    Getting everyday and always everything is counter productive.

    - It's using too much of CJ's Bandwith everyday (I don't know, if CJ has a special deal with their provider, but WE pay for Bandwidth)
    - It's using too much of OUR Bandwidth everyday
    - It's using too much Disk Space on our Servers everyday
    - It's using too much CPU Time on our Servers to import and filter the "junk" everyday

    I would prefer to use that recources to allow more users at the same time on our servers and to keep our Website fast loading.

    Todd, talk to your management and recommend this:
    Create a screen, where the Publisher can see all his active Merchants with Products available.
    Show in that list, if the merchant updates products manually or via datafeed on a regular basis (show the schedule). The Number of products for each merchant would be appreciated as well.

    Allow Publishers to select Advertisers from whom they would like to receive a Product Datafeed.
    Give the Option to the Publisher to decide, if
    he wants to include new Merchants with Products automatically in the Datafeed or if he wants to decide manually, if he wants the products or not.

    When you create the product datafeed files for the publisher, create one file per advertiser with the file name MERCHANTID.csv (or .xml).

    You don't even have to zip the files anymore (and waste your and our cpu time for it), because you will be surprised, how picky the publishers will be when it comes to selecting merchants.
    It will probably reduce the required disk space.

    I can understand, that you do not want to offer the product catalog feature to every publisher for free. Most of the BeFree and LS Merchants also don't give their datafeeds out to everybody who comes along and asks for it.
    Make it for free for performing publishers and charge new publishers for it, to allow them to build a business.

    Carsten

    Shop-Links.net Partner
    http://www.shop-links.net

    If you can't move things, try harder!

  2. #2
    ABW Ambassador cditty's Avatar
    Join Date
    January 17th, 2005
    Location
    Memphis TN
    Posts
    1,434
    Amen. Thanks for creating the new post. I was thinking about it but I didn't want to "harass" Todd anymore.

    Chris

    ----------------------------
    Scriptsforyoursite.com - Featuring datafeed import scripts for Art.com, Coldwater Creek, Mondera.com, OverStock.com, TigerDirect and more.....

  3. #3
    Schlaumeier cumbrowski's Avatar
    Join Date
    January 18th, 2005
    Location
    Fresno/CA
    Posts
    640
    quote:
    Originally posted by cditty:
    Amen. Thanks for creating the new post. I was thinking about it but I didn't want to "harass" Todd anymore.
    Chris



    hehe.. I didn't yet. One Call, One File and 2 Posts here yet.
    That's not harassment
    I also tried to be constructive and not just "complain".
    I also spent a lot of time and thoughts to put this post together. It should be worth reading by CJ.

    Carsten

    Shop-Links.net Partner
    http://www.shop-links.net

    If you can't move things, try harder!

  4. #4
    Schlaumeier cumbrowski's Avatar
    Join Date
    January 18th, 2005
    Location
    Fresno/CA
    Posts
    640
    No feedback from anybody at CJ yet, mmpfh.

    Haiko, is there an option to setup a notification on a topic to get alerted, when certain persons read it?

    Carsten

    Shop-Links.net Partner
    http://www.shop-links.net

    If you can't move things, try harder!

  5. #5
    ABW Veteran Student Heyder's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,482
    Of course I had to get myself involved in this conversation.

    Yes there are problems but they are workaroundable.

    How about someone come up with a data clean up service and sell the cleaned up data to affiliates?

    Can I do such a thing? How much would people pay me?

  6. #6
    Schlaumeier cumbrowski's Avatar
    Join Date
    January 18th, 2005
    Location
    Fresno/CA
    Posts
    640
    Hehehehe...

    Yes, there is always a workaround.

    Such as: Want to avoid car accidents? - Don't drive ( and get hit by a car on the Crosswalk later ).

    Clean up service mmh. I guess it's cheaper to keep things clean in the first place ... anyway.

    The first thing I would do, to improve the Datafeed is to (re)add the MerchantID to it.
    I have a lot of ideas to work around the other issues, when I have that back (even though I don't like them), because I hate waste of resources.

    I have some Ideas to workaround the none existing MerchantID, but I will not be the only one that will not like it (I will not tell you what the workaround is ).

    Carsten

    Shop-Links.net Partner
    http://www.shop-links.net

    If you can't move things, try harder!

  7. #7
    ABW Ambassador webmarm's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,713
    quote:
    because I hate waste of resources


    Well, that's where you obviously differ from other posters who don't mind the waste as long as they personally can use it.

    - - - - -
    42. Yup, the answer to life, the universe, and everything.

  8. #8
    Schlaumeier cumbrowski's Avatar
    Join Date
    January 18th, 2005
    Location
    Fresno/CA
    Posts
    640
    quote:
    Originally posted by webmistress:
    42. Yup, the answer to life, the universe, and everything.


    The world would be a better (and funnier) place, if everybody would read Douglas Adams

    Carsten

    Shop-Links.net Partner
    http://www.shop-links.net

    If you can't move things, try harder!

  9. #9
    Schlaumeier cumbrowski's Avatar
    Join Date
    January 18th, 2005
    Location
    Fresno/CA
    Posts
    640
    quote:

    Well, that's where you obviously differ from other posters who don't mind the waste as long as they personally can use it.



    By the way. Two Questions.
    Do you utilize the new CJ Product Datafeed? Did you utilized the old CJ Product Datafeed?

    Also, please read the first post.

    Carsten

    Shop-Links.net Partner
    http://www.shop-links.net

    If you can't move things, try harder!

    [This message was edited by Carsten, Shop-Links on January 05, 2003 at 04:52 AM.]

  10. #10
    ABW Ambassador webmarm's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,713
    2) I did use the old feed.
    1) I have one smaller feed I can use. The big one is not worth the investment of my time currently to battle. See the other thread.

    I appreciated your post quite a bit. We who are still training to be database gurus can easily be sneered at by those with a *few* more skills. Those of you who obviously have a great deal of database experience can actually give constructive criticism beyond the "I can't use it" or the "I can use it" level.

    - - - - -
    42. Yup, the answer to life, the universe, and everything.

  11. #11
    ABW Ambassador CrazyGuy's Avatar
    Join Date
    January 17th, 2005
    Posts
    1,463
    Good constructive post, Carsten.

    I also notified CJ that lack of numeric merchant ID was a necessary fix.

    CJ should also note that while the data feed service is in this state of flux, implementing it is failing to get to the top of my "must-do" pile.

    That's frustrating, because I know there is potential there, but one of my most precious resources is my time and I hate investing it in developing something that is likely to change in the near future.

    Are you Crazy?

  12. #12
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    quote:
    You don't even have to zip the files anymore (and waste your and our cpu time for it), because you will be surprised, how picky the publishers will be when it comes to selecting merchants.


    AAAAHHH, NOOOO!!! Keep zipping it, CJ!!!

    I promote everything in the feed. If I thought the merchant's stuff was too much of a dud to bother with--their program wouldn't be in my feed in the first place.

    So please DO keep on sending it zipped--it'd take eternity to try to download an UNzipped feed that includes Crapalog City's 500,000+ SKUs! Plus, it'd use a lot more bandwidth to send it unzipped.

    (Yeah, there's not really a million things from CC in there.)

    Speaking of Catalog City, how about getting those dadblasted EXLCAMATION POINTS they have in between everything the h!e!c!k! o!u!t o!f t!h!e!r!e (the heck out of there) or, better yet, making them take them out of there...

    "Create a screen, where the Publisher can see all his active Merchants with Products available."~Carsten

    YES! And along with that--how about a Gold List of ALL the merchants at CJ that have datafeeds?

    quote:
    create one file per advertiser with the file name MERCHANTID.csv (or .xml).


    Sorry, gotta disagree. That was a PITA with the old system. I like to know who's who, and having to correlate those numbers with something meaningful is a task I don't miss a bit. And I like .txt files, too.

    As for the Merchant ID, name-changers would be the biggest problem for me. But I can see that in a lot of implementations, it would be a BIG deal, and the more merchants involved the bigger the problem.

    It's the most wonderful time of the year! ~From a "Golden Era" Christmas Song

  13. #13
    ABW Ambassador Ron Bechdolt's Avatar
    Join Date
    January 18th, 2005
    Location
    Affiliateville, USA
    Posts
    7,927
    [QUOTE]Originally posted by Carsten, Shop-Links:
    I start a new Topic, because I believe, that Todd is not reading the other one anymore .

    QUOTE]

    He does not seem to be reading this one either. CJ Support does not seem to be reading their messages either. One does start to wonder if everyone took a three week vacation over the holidays.

    Ron - 7 Days A Week Marketing

    Every day is a chance to learn something new and a opportunity to teach someone else.

  14. #14
    Newbie
    Join Date
    January 17th, 2005
    Posts
    3
    I'm just starting to look at the using the cj datafeed.

    Should I understand from this thread that every product from every merchant is in the cj datafeed, everyday?

    Please advise.
    Thanks, Phil

  15. #15
    Schlaumeier cumbrowski's Avatar
    Join Date
    January 18th, 2005
    Location
    Fresno/CA
    Posts
    640
    Quote Originally Posted by cumbrowski
    We talked already a lot about issues with the new CJ Datafeed at:
    http://abw.infopop.cc/6/ubb.x?a=tpc&...9&m=6526020181

    I start a new Topic, because I believe, that Todd is not reading the other one anymore .

    I took my last comment from the old thread, but extend it now with my own experiences with the datafeed file, which I just received today for the first time successfully.

    ....

    Btw. a bit over one year later the resolution to this post.

    The problems did not change or go away or anything like that. They are still there as up-to-date as they were a year ago, but I solved the problems myself.

    I spent dozens of hours writing my own feed importing, records filtering, merchants watching, updates tracking, internal ID mapping CJ Datafeed Manager System PRO ... plus threw a new Dell dual cpu/4 GB Ram/Raid Blade Database Server at it (which handles all the other feeds from all the other networks and in-house programs on the side as well btw.)

    Well, if there is a club to join for affiliates that were able to tame the CJ datafeed beast, let me know. There must be, because I don't think that the average affilate is able to do what I did. It must have been a test of some sort or acceptance ritual. Mmmhh.

    My remaining requests are now much more moderate and reasonable.
    See here http://forum.abestweb.com/showpost.p...9&postcount=40.

    lets continue this in 01/2007
    Carsten Cumbrowski
    Marketer & Entrepreneur @ Cumbrowski.com
    Free & no BS Internet Marketing Resources

  16. #16
    Schlaumeier cumbrowski's Avatar
    Join Date
    January 18th, 2005
    Location
    Fresno/CA
    Posts
    640
    Quote Originally Posted by phil.davis@deepdiscounts.com
    I'm just starting to look at the using the cj datafeed.

    Should I understand from this thread that every product from every merchant is in the cj datafeed, everyday?

    Please advise.
    Thanks, Phil
    Every Merchant you have an Affiliation with through the CJ Network and who has a feed.

    I get it weekly (roughly, it dies from time to time during creation on CJ's end. It's not so easy to generate a 5+ GB Delimited text file and zip it down to a 800-900 MB File every week).

    We have today feeds from 450+ Merchants in the file.

    We use 200+, don't plan to use 120+ at all and 100+ a few at the time later.

    From the 8+ Mio Products in the File are we using only 3+ Mio today. The number of products used is increasing as we go, but in the meantime are we downloading and processing a big pile of junk every week we can not control on CJ's end. We are used to wait up to a week to get the feed of a merchant program we just joined and want to use. I have no problem whatsover to have to wait the same amount of time if I decide to receive a feed from a merchant I disabled earlier.

    In addition to that are not all merchants update their feeds weekly. So you have to check the last updated date as well (you must check all products for a merchant because the "updated" date can be different for each product in the merchants feed).

    I like the auto-enable feature though. It's more cumbersome in any other network.to get access to the feed you want and in some cases even to find out if the ther merchant has a feed or not.

    I would prefer to get a separate file per merchant and some means to be able to check PRIOR downloading and processing the file, when it was changed the last time (I hate this about the Linkshare feeds, you have to download and unpack the feed first to find out that nothing changed).

    This would allow us to check frequently (daily) if a feed for a merchant we are interested in was modified or not and only download and process that individual feed if it changed since our last download.

    We do that with all our BeFree Feeds.
    Carsten Cumbrowski
    Marketer & Entrepreneur @ Cumbrowski.com
    Free & no BS Internet Marketing Resources

  17. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Replies: 4
    Last Post: June 23rd, 2009, 06:39 PM
  2. zip file datafeed file transfer into database
    By rahulcbr in forum Programming / Datafeeds / Tools
    Replies: 13
    Last Post: August 28th, 2007, 03:59 AM
  3. How do I use a .aspx datafeed file?
    By rAd in forum Programming / Datafeeds / Tools
    Replies: 2
    Last Post: June 20th, 2005, 11:22 AM
  4. How to download all product links in one file ?
    By oxigen in forum Commission Junction - CJ
    Replies: 2
    Last Post: November 3rd, 2003, 05:08 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •