Page 1 of 2 12 LastLast
Results 1 to 25 of 37
  1. #1
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Datafeed column headings?
    Hi,

    2 questions if you'll permit me!:

    1) A while ago, the datafeeds were in .csv format (which are still accessible, but not default) and had the column headings as the first line. Now, I see the datafeeds are in .txt format and have no column headings...first line is 'bam' into the products.

    Why do the datafeeds not have the first line for column headings? My script relies on knowing the name of each column, or else it'll have absolutely no idea of which database field to use.

    2) When the feeds had the column headings and the default was .csv, I started implementing Shareasale affiliates into my script. But then 1 week after, the column names changed, the order changed and everything broke my code. For all the affiliates. So I stopped promoting Shareasale affiliates. Was this changing of datafeed structure a one-off, or does it happen regularly? Order and stability is a bit essential in automated processes!

    Thanks for any insight.

  2. #2
    Member RWorld's Avatar
    Join Date
    December 18th, 2005
    Posts
    101
    I don't know the answer, but I agree that for datafeeds to be useful, the structure really needs to be formalized, and this info passed to all merchants and affiliates.

    I guess at the moment some people are using xls, some xml, some csv - without a definitive specification, it's hard to have consistency. There is a thread a about this somewhere, but I don't think any formal conclusion was reached.

  3. #3
    Lite On The Do, Heavy On The Nuts Donuts's Avatar
    Join Date
    January 18th, 2005
    Location
    Winter Park, FL
    Posts
    6,930
    Quote Originally Posted by RWorld
    I don't know the answer, but I agree that for datafeeds to be useful, the structure really needs to be formalized, and this info passed to all merchants and affiliates.

    I guess at the moment some people are using xls, some xml, some csv - without a definitive specification, it's hard to have consistency. There is a thread a about this somewhere, but I don't think any formal conclusion was reached.
    You're talking about across all merchants and networks and indies and planets and galaxies - cool idea, but getting everyone together is a tough chore.

    In the meantime, back to stability you want and question about ShareASale feed - they are standardized at SAS and the field names are in the help files, along with a few other helpful, brief, very clear items.

  4. #4
    Member RWorld's Avatar
    Join Date
    December 18th, 2005
    Posts
    101
    I was actually only referring to SAS, as I don't have experience of other networks etc - but if SAS have standardized their feed, I wonder why the OP had problems with headers and column names?

  5. #5
    Lite On The Do, Heavy On The Nuts Donuts's Avatar
    Join Date
    January 18th, 2005
    Location
    Winter Park, FL
    Posts
    6,930
    Maybe they didn't read the help files? :-)

  6. #6
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Quote Originally Posted by Donuts
    Maybe they didn't read the help files? :-)
    Maybe you didn't RTFP?
    I know what the column names are, since I can read the help files. However, my program cannot read help files so it has no idea. OK, yes, I could just add a little line in the program to tell it what the column names are.

    But then, let's say for whatever reason SAS changed the order of those fields. Or just one of them. Or just added 1 more column to the end. Well then, that would be a disaster. If they added a column, that would break the code. If they changed the order, that would populate my database with garbage because, and this is the crux, all networks use different structures for their datafeeds. So for me to have some order, my database field names MUST be identical to the column names of the feed.

    Now, if the first line of the feed had the column names in it (with whatever delimiter, it doesn't matter), then the order of those columns, for me, could chnage daily, it doesn't matter. If another column at a later date was added, it wouldn't matter, because the program would simply add the new column to the dbase table.

    Standards are great. Standards across networks are ideal, but never going to happen. The really BIG sites that use feeds dictate what the feeds should look like. We have to make do with what the networks give us. And for that it's all about stability and having a standard that is convenient for the vast majority. I read a good part of that thread discussing this, but what never came up (I only just read it, so I didn't input) was the issue of field names on the first line. IMHO, for automated scripts, that first line is essential.

    Do you guys process your feeds by hand? Would it bug you if that first line existed? Now, for now, the .csv has this first line in it, but according to the feed download area, this is just 'for now'. I can't go writing a script based on 'for now'. Moreover, the .csv files are not available via ftp. My script, while not dependent, is much more efficient in terms of regularity on ftp downloads. And as I said, the column orders have changed in the past, and nowhere in the help files does it mention that this is now fixed in stone, hence my original question.

    So, yes, I read the help files. Did you read my post?

    --edit
    in fact the column order appears to have changed 5 months ago, based on the 'last updated date in the help file. Not looking too good....

  7. #7
    Lite On The Do, Heavy On The Nuts Donuts's Avatar
    Join Date
    January 18th, 2005
    Location
    Winter Park, FL
    Posts
    6,930
    yes, i did read your posts and the ones from the other person too - who asked - "I wonder why the OP had problems with headers and column names?". That problem was wanting to know what the standard layout is now and going forward and if it's standardized and stable - that answer is yes and it's in the help files. Now you appear to be saying that your defined problem is that you want things to never ever change cuz you'd have to adjust your script and do work.

    "And as I said, the column orders have changed in the past, and nowhere in the help files does it mention that this is now fixed in stone, hence my original question"

    if you want to names, order and such to never change, make your own datafeed.

    if it changes every week or every month, i do get your point - but that's not the case here.

    and if the one change you're referring to, happened 5 months ago, and you're still asking for promises that it'll never change again, you're dreaming.

    you cannot fully automate everything via script.

    new technologies will come along, new prevailing sentiment about needed fields, structure, order, format, file types, etc will arise and merchants and networks will adjust.

    you need to do that too.

  8. #8
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    input - Being a fellow automater here is a suggestion to CYA: Check the header fields for validation before processing and start doing this with all of your site scripts. You can complain all you want about the changes but now you know that they will change so plan on dealing with it in your script. Validate the field order or other info in the feed before you process. If it is not what you expect then stop processing and have the script email you. Just makes good sense and your scripts will be that much more reliable. If it is PHP just plan on putting a validate_format() routine in every script you write.

  9. #9
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Quote Originally Posted by Donuts
    yes, i did read your posts and the ones from the other person too - who asked - "I wonder why the OP had problems with headers and column names?". That problem was wanting to know what the standard layout is now and going forward and if it's standardized and stable - that answer is yes and it's in the help files. Now you appear to be saying that your defined problem is that you want things to never ever change cuz you'd have to adjust your script and do work.

    "And as I said, the column orders have changed in the past, and nowhere in the help files does it mention that this is now fixed in stone, hence my original question"

    if you want to names, order and such to never change, make your own datafeed.

    if it changes every week or every month, i do get your point - but that's not the case here.

    and if the one change you're referring to, happened 5 months ago, and you're still asking for promises that it'll never change again, you're dreaming.

    you cannot fully automate everything via script.

    new technologies will come along, new prevailing sentiment about needed fields, structure, order, format, file types, etc will arise and merchants and networks will adjust.

    you need to do that too.
    I think you have an inability to digest what is written. As your reply was stern and abrupt, mine will be too.

    I asked
    1) Why did they drop the column headers from the datafeed. At least with column headers, one knows if the order has changed immediately, or tagged on another column at the end. Automated or not.

    2)Was this changing of datafeed structure a one-off, or does it happen regularly?

    Now, agreed, at the time I posted this, I never noticed the "Last modified on", but hey, it's not exactly flashing and dancing in the help files.

    I know things change etc, but as I said, when I signed up (sometime mid 2005), things changed within 1 week. Now I see they changed not long after that again. So, I think my question 2 is still a valid one.

    So get off your high horse and read my posts. My questions have still not been answered.

  10. #10
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Quote Originally Posted by patrick24601
    input - Being a fellow automater here is a suggestion to CYA: Check the header fields for validation before processing and start doing this with all of your site scripts. You can complain all you want about the changes but now you know that they will change so plan on dealing with it in your script. Validate the field order or other info in the feed before you process. If it is not what you expect then stop processing and have the script email you. Just makes good sense and your scripts will be that much more reliable. If it is PHP just plan on putting a validate_format() routine in every script you write.
    Hey Patrick,

    Sure, I have validation in the scripts now and the script now detects if columns have been added (creates new field in dbase) or deleted and doesn't care if the order moves*. However, the main problem is what I was asking in Question 1 and I think you iterated when you said Check the header fields for validation before processing.... The feeds don't have any header fields - only something in the help files which are not accessible from an automated script. Line 1 of the feed is straight into the product, so all I can do is check the number of columns against my dbase to make sure the column list hasn't inflated or deflated. It will never be able to check if the order has changed or if say "Short Description" swapped with "URL" or something. These "minor" changes could go undetected for a very long time in an automated process.

    I don't care if detectable changes are made every year or twice a year for eg, so long as they are just that. Detectable But how would you go about checking MySQL varchar(255) fields (for eg) against a column order swap of LongDescription and ShortDescription?


    --edit for clarity

    * Only if the feed has column descriptions on the first line

  11. #11
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Can I just add why this is so essential to me.... My site is a comparison shopping site, and for that to work identical products from different merchants need to be grouped. Now imagine - no one merchant uses the same ANYTHING for a given product w/r to other merchants. Heck even, manufacturer item numbers change! So, I have an algorithm that checks item numbers, sku numbers bar codes, product names, prices and finally description and gives a weight to each one against all the other already-grouped products to see if it belongs in one, or is in fact a "new" product.
    The thing is, once it's in the group it affects the score of other products looking to join the group, so if it is really something that shouldn't be in that group, then the rank for that group is going to drop like a brick. blah blah.

    It's not perfect, nothing is, but it's at about 90% accurate at the moment and I hope to tweak the algorithms some more to increase that. If, on a feed without header info, column order for eg short description and long description switch, then ok, the script won't think it's a new product since ever product has a unique identifier, but it will think that product needs updating and update it (and consequently that groups search ranking will drop and falsly exclude other genuine product from entering or worse, allow non-similar products to enter).

    Does this make any sense?

    All I'm asking is why the column headers were dropped. That's like car manufacturers saying, "beh, I'll think we'll drop seatbelts from our next generation model, noone uses em.".

  12. #12
    ShareASale President/CEO and ABW Veteran Brian - ShareASale's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,657
    input,

    We recognize the importance of stability in a datafeed and try not to change the format at all- though occasionally it needs to happen in order to add in certain fields or expand on what we are offering, etc...

    About the only thing that will change that I can forsee is the addition of more fields - which would be at the end of a given row so as not to change the order of the current row.

    Anything larger than that and we would make a very coordinated effort to get the word out to everyone prior to making a change.

    I apologize for the problems you had when the column field names were dropped, I think the best thing to do probably, if you want to use the FTP files is to write something into your script that defines the column values ... they won't be changing anytime soon.
    Thanks,

    Brian Littleton
    President/CEO - ShareASale.com, Inc.

  13. #13
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Thanks for clarifying Brian,

    Any chance of putting the a column header file on the ftp server also. Even if there is a reason for not incorporating the column headers in teh feed? A single file containing one line of pipe-delimited column names, (ie exactly what is in your help doc) in the ftp area could be parsed by my (and others') script so it would automatically know if columns have been appended etc and update the dbase accordingly.

  14. #14
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    Complete non-sequitor to the technical questions at hand:

    input - twice donuts has tried to help. Whether or not his answers met your questions is irrelevant. The fact is you appear to be a very rude person - one that most people would probably not want to help in the future. I don't care how good of a site or sites you have or how much money you make. Your success is going to depend alot on how you get along with other people.

    You might want to consider changing your tone when you address other people. You don't have to like their responses. But being that you are NEW here I'd think you'd want to make as many friends as possible and be as nice as possible. Your choice - but I am sure you know what will get you further here with other members and affiliate managers.

    P

  15. #15
    ShareASale President/CEO and ABW Veteran Brian - ShareASale's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,657
    input,

    the problem with adding those fields at this time is that there are a number of other users who would be troubled to see an extra field of "non-data" at the top who are used to seeing just product data...
    Thanks,

    Brian Littleton
    President/CEO - ShareASale.com, Inc.

  16. #16
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    I am very much aware of curtious and proper behaviour and it is clear how much donuts has helped others in the past from his post count. But, his first post didn't answer my questions, his second was rude and condescending and his third was nothing but flame-bait.

    Other than post #9, which was in response to his flame-bait fly-away gesture, I don't see where my tone has been improper.

  17. #17
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Quote Originally Posted by Brian - ShareASale
    input,

    the problem with adding those fields at this time is that there are a number of other users who would be troubled to see an extra field of "non-data" at the top who are used to seeing just product data...
    I understand that, which is why I made the suggestion of adding the column headers to a totally seperate file in the download area. That way those that want to use the column headers can parse the header file and then the feed file, whereas those that don't want to just parse the feed file.

    win/win situation?

  18. #18
    ShareASale President/CEO and ABW Veteran Brian - ShareASale's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,657
    ahh- not a bad idea i'll look into it...
    Thanks,

    Brian Littleton
    President/CEO - ShareASale.com, Inc.

  19. #19
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Thanks for considering it Brian.
    I don't want to be a nag here, but I just started working on implementing SAS code and I found a niggling problem with the ftp files:

    the files available are
    .txt
    .txt.gz
    .zip

    .txt.gz file is somewhat misleading as it leaves one to believe that once inflated, the file would be a .txt file. But in fact the inflated file is .csv
    So, for the time being, I have to download the non-compressed .txt file, since my script has no idea that a file extension change would occur on deflating....

    Any chance either:
    a) the .txt file is gzipped instead of the .csv
    or
    b) the gzipped .csv have the filename format .csv.gz

  20. #20
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    Why can't you download the compressed one? You know the FTP filename. You know the filename when it is extracted. The filename that is extracted is the one your program has to read. I would think both of these pieces of information would just be settings in your data massaging program.

  21. #21
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Yes Patrick, but consistency and standards means a lot. Standards in filename nomenclature is pretty much written in stone.

    I (the script) only know the filename because there is a database field telling the script what is the filename. There is also another field saying whether the file is compressed and if so, what format.

    so my entries are:
    fname (=>filename.txt)
    fname_compression (=>.gz)

    so the script knows to download filename.txt.gz then decompress it with gzip and finally read filename.txt

    now maybe I should have designed the database fields to be
    fname (=>filename.txt)
    ftp_fname (=>filename.txt.gz)

    But, you know following standards, either set of dbase fields should work. I opted for what I did because then I don't have to write more code to figure out if the file is compressed etc...since it is already in the database

    I thought we were all for standards where possible.

    Thankfully, SAS has the uncompressed file, and as the feed I'm working on hasn't been modified since April 2005, it won't be getting downloaded very often (the script only downloads if the modification time has changed since last check)!

    I could go back and modify the database structure, but with over 23 merchants safely written for, it's not going to be easy. If it were essential, I'd weigh it up, but as the uncompressed file is there and it's my coding that is following standards, I'm not going to.

    Plus how filename.csv got to called filename.txt.gz in the first place I'll never know!
    gzip filename.csv
    gives filename.csv.gz

  22. #22
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Quote Originally Posted by input
    =Plus how filename.csv got to called filename.txt.gz in the first place I'll never know!
    OK, I figured out what has happened - I was using the gzip -N flag to decompress saving the original filename and timestamp. I was using this primarily to use the timestamp from the file (to compare whether the file should be downloaded or not), but quickly changed to the timestamp on the ftp server (hence -N was nolonger needed for me). By keeping the -N flag, I was forcing decompression to the original filename (.csv).

    By leaving off the -N flag, the file decompresses to filename.txt.

    Sorry for raising this inconsistency, but the reason for it may help others.

  23. #23
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    Hey.... you discovered the issued and solved it for those that follow you. That is one the main reasons I use the internet. To see where somebody else messed up before me and figured out a fix before me.

    Thanks for sharing it.

    You keep all of your merchant information in a table? So is this a master table that is used to generate several sites or just the one site?
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

  24. #24
    Newbie
    Join Date
    February 15th, 2006
    Posts
    23
    Quote Originally Posted by patrick24601
    Hey.... you discovered the issued and solved it for those that follow you. That is one the main reasons I use the internet. To see where somebody else messed up before me and figured out a fix before me.

    Thanks for sharing it.

    You keep all of your merchant information in a table? So is this a master table that is used to generate several sites or just the one site?
    I always try to follow up with solutions to my own problems - quite often I'll be searching for a problem and find a thread somewhere where someone had the same problem and then the OP replies "Never mind I fixed it". Arrgh - what the hell did you do? Drives me nuts!

    I have all my networks and merchants (x-referenced to which network) in a table in the "main" database. Then all the merchant feed info go in a separate table which build their own tables holding all the raw feed info. These raw feed tables are used to build a second merchant table holding all the relevant (to my site) product data (the product "children") in the format standardised for my site (my categories /subcats brands etc), which is then parsed to group all the product info across all the merchants into a master product table (the product "parents") and it's this parent table which is used to display the products on the web page.

    Lots of tables, lots of data, and all rely on the feeds. Feeds are fantastic - the script can go from downloading the feed to entering 40,000 products to the raw feed table, to parsing each and every one into the main children table, and matching them into the parents database (all the while making sure there are no deleted products, and hence orphaned children) in about 30 seconds

    Right, I'm off for a weeks R&R on the piste. Have fun all.

  25. #25
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    When you get back...

    Do you always empty the tables from a given merchant and/or a given affiliate management system? Or do you keep them from week to week so you can track price trends/alerts?
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

+ Reply to Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Walmart - SKU is also in the UPC & MPN column in datafeed
    By majestique in forum Rakuten LinkShare - LS
    Replies: 2
    Last Post: October 13th, 2013, 01:40 PM
  2. Walmart - What does W or M mean in your Availability column in datafeed?
    By majestique in forum Rakuten LinkShare - LS
    Replies: 2
    Last Post: October 12th, 2013, 08:41 PM
  3. Help with Unique column in SAS datafeed
    By likemynick in forum ShareASale - SAS
    Replies: 3
    Last Post: August 16th, 2006, 06:44 AM
  4. Can I use sql to parse a datafeed column?
    By johnnyWebAffiliate in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: April 5th, 2006, 07:58 AM
  5. Sorting / Headings
    By kara1 in forum Midnight Cafe'
    Replies: 2
    Last Post: June 11th, 2002, 09:13 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •