Page 1 of 2 12 LastLast
Results 1 to 25 of 29
  1. #1
    Believe knight01's Avatar
    Join Date
    August 14th, 2006
    Location
    Dayton, Ohio
    Posts
    1,815
    Massive Datafeeds
    I've just started working more intensely with datafeeds. In the past they've been relatively small, a few thousand products.

    I'm starting to get some feeds that are larger than 65k of products and I can't open in excel to review and manipulate.

    How do you open a file this large?
    Someday starts today
    Military Discounts

  2. #2
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by knight01
    How do you open a file this large?
    PHP all the way!

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  3. #3
    Believe knight01's Avatar
    Join Date
    August 14th, 2006
    Location
    Dayton, Ohio
    Posts
    1,815
    Quote Originally Posted by Snib
    PHP all the way!

    - Scott
    Scott,
    There's a spreadsheet written in php?
    Someday starts today
    Military Discounts

  4. #4
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by knight01
    Scott,
    There's a spreadsheet written in php?
    You mean a speadsheet program? You could try Google Spreadsheets. I have my own custom interface for viewing datafeeds though.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  5. #5
    Full Member
    Join Date
    March 10th, 2006
    Posts
    466
    Look for a spreadsheet download called "CSVed" by Sam Francke. Last time I downloaded it it was free. That might do the trick. I just checked the help, and didn't find the record count limitation specifics.

    Added - I'm pretty sure it won't work with zip files, they need to be unzipped first.

  6. #6
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by mr_jones
    Look for a spreadsheet download called "CSVed" by Sam Francke.
    I've been looking for something that might be able to open/manipulate CSV files.
    I'll try this. Thanks for the tip.
    ~Rhia7 -- Remember the 7
    Twitter me

  7. #7
    Believe knight01's Avatar
    Join Date
    August 14th, 2006
    Location
    Dayton, Ohio
    Posts
    1,815
    mr_jones, thanks! This is just what I needed.
    Someday starts today
    Military Discounts

  8. #8
    http and a telephoto
    Join Date
    January 18th, 2005
    Location
    NYC
    Posts
    17,708
    Excel 2007 has unlimited records and imports from rss/xml direct from the web. Will look at what Mr Jones suggested, sounds interesting!
    Deborah Carney
    TeamLoxly.com BookGoodies.com ABCsPlus.com

  9. #9
    web dev with whiskers tn-morgen's Avatar
    Join Date
    February 15th, 2007
    Location
    USA
    Posts
    177
    Nothing for Mac?? Wah!

  10. #10
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    IMHO...

    When your datafeeds gets to be any decent size you need to STOP looking at things like Excel and Access and start looking at using databases. What is the magic number? I don't know. Personally anything over about 500 records I see as belonging in a database. I know it is a serious step up. But when you are dealing with that much data you are a serious affiliate.

    You need fast ways to look through your data, parse your data, find data, change data, etc. You can do it on your machine or do it on a remote server, but you should seriously consider using a database to deal with large amounts of data.

    If you want some help on how to get it into a database or what do with it once you get it there let me know. I (or Snib, or any other people) can help you out.

    (Disclaimer: My company does PHP and MySQL development)
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

  11. #11
    Newbie
    Join Date
    June 3rd, 2007
    Location
    the Netherlands
    Posts
    46
    Quote Originally Posted by patrick24601
    IMHO...

    When your datafeeds gets to be any decent size you need to STOP looking at things like Excel and Access

    Not a MS fan are you?

    I agree with you tho but i do see the need to inspect a datafeed before putting it into a db. Excel can be used but i prefer something like notepad or NoteTab if it gets bigger. Getting things out that will break your insert or just to replace your Aff id#. And yes you can write a function in php but especially with a new feed i want to look at it first

  12. #12
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    I don't think I stated myself clearly.

    The company I own does a huge amount of MS development work for the government. I have no problem with MS technology, or any type of technology for that matter. I look at solutions and solutions can be implemented with any type of technology. I'd suggest others do the same. Don't get hooked on MS, or Open Source, or C++, or PHP. Use the tool that best fits the solution.

    My point was - for a huge amount of data if I want to examine alot of it I would use a database and SELECT statements. On a regular basis I also use notepad++, excel, zend IDE, etc. I use whatever tool it takes to get the job done.

    If I joined a new affiliate program today and I was given of feed of 500 records I'd probably use notepad++. Probably the same with 1000 records - who knows. After that (to me) it gets kind of slow to look at that amount of data. I'd see if there was an easy way to dump it into a local database (I have mysql and sql server installed locally) for more detailed examination. It is pretty fast for me to do something like that, and I am pretty good at writing select statements to find what I want or what I DON'T want in my data. I can also use it to select out a subset of data that I do want and move that into my live database.
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

  13. #13
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by edynas
    I agree with you tho but i do see the need to inspect a datafeed before putting it into a db. Excel can be used but i prefer something like notepad or NoteTab if it gets bigger. Getting things out that will break your insert or just to replace your Aff id#. And yes you can write a function in php but especially with a new feed i want to look at it first
    What I do is pull the information up with PHP so I can get a good look at it. I basically see one product per page and I'll check several pages of products. This is all part of my datafeed mapping mechanism. I use it to map the fields to my database fields. Something I'd like to do is build a mechanism for displaying X number of values from a particular field so I can get a broader idea of what's in there. I just need to make sure it's a random sample across the entire datafeed.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  14. #14
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by Snib
    What I do is pull the information up with PHP so I can get a good look at it.
    - Scott
    Do you use some type of program such as Excel or CSVed to look at the information? How should information be examined?
    ~Rhia7 -- Remember the 7
    Twitter me

  15. #15
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by Rhia7
    Do you use some type of program such as Excel or CSVed to look at the information? How should information be examined?
    No, like I said I use PHP to read the datafeeds and display the information through my custom datafeed admin program.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  16. #16
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by Snib
    No, like I said I use PHP to read the datafeeds and display the information through my custom datafeed admin program.

    - Scott
    Okay, do you have any advice for a beginner like me on how to read datafeeds?
    ~Rhia7 -- Remember the 7
    Twitter me

  17. #17
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by Rhia7
    Okay, do you have any advice for a beginner like me on how to read datafeeds?
    Probably Popshops or GoldenCan.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  18. #18
    Moderator Nabz's Avatar
    Join Date
    August 24th, 2005
    Location
    VY Canis Majoris
    Posts
    1,751
    I don't know, but I find these datafeeds a lot complex, I'm not a programmer... and about GoldenCan, why would I give every 4th click to them ?

  19. #19
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by Nabz
    I don't know, but I find these datafeeds a lot complex, I'm not a programmer... and about GoldenCan, why would I give every 4th click to them ?
    I'm with Nabz: I'm not about to give up the 4th click nor am I going to pay $5 per month.

    I'd just like to know: what's the best way to open/examine CSV files?
    What can I do with them after I open them?

    Is there a way to manually create links from the given data? I don't use webmerge nor do I use MysQL & PHP (yet).
    ~Rhia7 -- Remember the 7
    Twitter me

  20. #20
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Los Angeles
    Posts
    4,053
    Note: I have steadfastly refused to upgrade from Office 97 professional because of the garbage XML later versions put out. And I am not willing to pay a programmer to do a dynamic site for me and pay to maintain or fix things if needed. If I want to learn to program, I'd have to do it in my spare time while flipping burgers for a living. We have to work with what we have and what we know.

    Repeat: some of us are NOT programmers and don't know how and it wouldn't be cost-effective or wise to pay in many cases, especially with small niche sites or blogs.

    Okay, do you have any advice for a beginner like me on how to read datafeeds?
    Rhia7, yes. There is an Open Source spreadsheet program that can open feeds very easily - and SAS feeds are especially easy to work with. If a feed is too big there is a program that splits the files up for you before you even try to open in the spreadsheet program.

    I've never gotten further than Carolyn's tutorial for WebMerge, and I've been using it ever since she posted that, as well as a slightly modified version of the template she posted - which is like her "storefront" pages with just a select number of products. Works like a charm, and from what I've gathered you do some product pages the same way as I do.

    I'll try to dig out what the program is that splits up the feeds, it even works on the HUGE feeds that PFX has for some merchants. It *might* be called "file splitter" or a search for that or similar terms might locate something.

  21. #21
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by webworker

    Rhia7, yes. There is an Open Source spreadsheet program that can open feeds very easily
    Great. What is the name of the Open Source Spreadsheet that you have in mind?

    I'll give CSVed a try when I get a chance -- otherwise it's copy & paste
    ~Rhia7 -- Remember the 7
    Twitter me

  22. #22
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Los Angeles
    Posts
    4,053
    Open Source Spreadsheet
    Open Office (there may be another, I can't recall the name)

  23. #23
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    Rhia,

    The first question should be: Why do you want to open them ? Do you want to generate web pages from them? Do you just want to see what is in them ?

    What is your goal with using a datafeed and I (others) can tell you best how to use them or want to use to open them.

    Webworker: And I am not willing to pay a programmer to do a dynamic site for me and pay to maintain or fix things if needed. If I want to learn to program, I'd have to do it in my spare time while flipping burgers for a living. We have to work with what we have and what we know.
    WebWorker: What is the point of this statement?

    It is very cost effective to pay for a service - any service - if you don't know how to do it yourself and you want it done a certain way. People have paid me for 30 years to program. I pay my carpet guy to carpet because I want it done right and I don't want to be a carpet guy. Just because I don't know how to carpet does not mean I don't want to walk on it !

    P.S. Some of us ARE programmers and are willing to help those that are not
    Last edited by PatrickAllmond; November 23rd, 2007 at 12:51 AM.
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

  24. #24
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by patrick24601
    Rhia,

    The first question should be: Why do you want to open them ? Do you want to generate web pages from them? Do you just want to see what is in them ?

    What is your goal with using a datafeed and I (others) can tell you best how to use them or want to use to open them.
    I merely want to see what is contained in the datafeeds.
    I'd like to create links/product links from the datafeeds.
    I generate web pages manually.


    Quote Originally Posted by patrick24601
    What is the point of this statement?
    Please edit your quote box because that statement you quoted came from Webworker not me. You do not address Webworker at all in your post and should because readers who read the thread quickly might attribute the quote to me and I do not want to be associated with that quote.
    ~Rhia7 -- Remember the 7
    Twitter me

  25. #25
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    OK. Then all you really need is Notepad (Windows) or something equivalent for Mac/Linux. I use notepad all of time just to open the file and look at it. When it comes to large datafeeds I use notepad++ because it handles larger files better. Just my opinion.

    All of the other programs mentioned let you look at it in a more organized way. Excel/Openoffice let you open it and separate the columns of data so you can see them more clearly.

    A datafeed is just a list of products. It has several rows, one row per product. Each row has several columns. You could open a datafeed file in any of the programs mentioned and then copy/paste the text into your HTML editor to generate your pages manually. This is a good simple way to use the datafeed data for what you mentioned.

    Here is an example for you - here is a single row from one of the SAS datafeeds:
    453670861|Instinct|6719|MagazinesQuick.com|http://www.shareasale.com/m-pr.cfm?merchantID=6719&userID=YOURUSERID&productID=453670861|http://www.magazinesquick.com/xcart/files/images/thumbs/t_40965_01.jpg||19.95|54.45|Books/Reading|Magazines|Instinct Magazine is a clever mix of "cosmo" and "maxim" for gay men. The magazine is filled with funny, sexy, and smart features.|Lifestyles & Cultures|12|30|||2007-06-01 11:42:08.040|instock|||||||

    Each | separates the different parts of information about this product. SKU, Product title, description, etc. You can copy and paste those into your own pages and you would be well on your way.

    Does this help?
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

+ Reply to Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Will the datafeeds in the merchants datafeeds thread track my commissions?
    By john9245 in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: March 29th, 2005, 09:42 AM
  2. You have a massive infestation!
    By SSanf in forum Commission Junction - CJ
    Replies: 6
    Last Post: May 27th, 2003, 12:17 PM
  3. eBay: Massive, massive problems
    By Ralph Slate in forum Commission Junction - CJ
    Replies: 4
    Last Post: January 8th, 2003, 11:17 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •