Page 1 of 2 12 LastLast
Results 1 to 25 of 26
  1. #1
    Affiliate/AM Moonlighter dflsports's Avatar
    Join Date
    January 17th, 2005
    Posts
    874
    splitting datafeeds into smaller feeds
    Hello!

    I know just enough about php and mysql to break my sites pretty regularly I'm using scripts that do most the work importing datafeeds and displaying products.

    Some feeds are huge, 100mb or so, and I really do not want all the data, especially on a niche type site.

    I thought about installing a webserver on my local machine, importing all the data into a mysql database and then running some queries to extract the data I want.

    I was hoping to find a program. I tried one today, datafeed extractor or something like that, I did not like it.

    Does anyone have any suggestions?

  2. #2
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    I can't self promote a product but feel free to contact me.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Los Angeles
    Posts
    4,053
    dflsports, there's a free utility out there that splits files, either text or zipped I believe. Do a search for file splitter.

  4. #4
    Affiliate/AM Moonlighter dflsports's Avatar
    Join Date
    January 17th, 2005
    Posts
    874
    Quote Originally Posted by webworker
    dflsports, there's a free utility out there that splits files, either text or zipped I believe. Do a search for file splitter.
    I did search for file splitters and there are plenty of free ones. Thanks for the suggestion, one of those might come in handy some day. Only thing is I need to extract data by category and most datafeeds are not in order by category. Even if I did sort them myself, then figuring out where to split would be a crap shoot at best.

  5. #5
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    If you don't have a local web server install one!! And don't just do it for this, but for the overall convenience it creates. You can build your sites and do all your PHP/mySQL without needing FTP to test anything. It's a HUGE time saver and essential for any PHP development.

    As for splitting the datafeed I suggest building a PHP script that simply ignores the fields you don't want and imports the data you want. This can all happen on your web server once you build it and test it locally. Then you'll never need to run any splitting software and you won't need to download datafeeds to your computer.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  6. #6
    Affiliate/AM Moonlighter dflsports's Avatar
    Join Date
    January 17th, 2005
    Posts
    874
    Quote Originally Posted by Snib
    If you don't have a local web server install one!! And don't just do it for this, but for the overall convenience it creates. You can build your sites and do all your PHP/mySQL without needing FTP to test anything. It's a HUGE time saver and essential for any PHP development.

    As for splitting the datafeed I suggest building a PHP script that simply ignores the fields you don't want and imports the data you want. This can all happen on your web server once you build it and test it locally. Then you'll never need to run any splitting software and you won't need to download datafeeds to your computer.

    - Scott
    Hahaha, that sounds easy enough

    I guess installing the webserver is the way to go, just use the scripts I have now and then query the database to extract the data I want. This should be fun, NOT!

  7. #7
    ABW Ambassador PatrickAllmond's Avatar
    Join Date
    September 20th, 2005
    Location
    OKC
    Posts
    1,219
    Installing a local web server is amazingly easy and you will kick yourself for not doing it earlier. Google xampp. Download it. Install it. Done. You now have a local webserver running along with a local mysql server and a local FTP server. It will probably take you longer to read this and reply than to get your local webserver up my friend.

    Snib is the shizznit!
    ---
    This response was masterly crafted via the fingers of Patrick Allmond who believe you should StopDoingNothing starting today.
    ---
    Focus Consulting is where I roll | Follow @patrickallmond on Twitter
    Search Engine Marketing | Search Engine Optimization | Social Media | Online Video

  8. #8
    Troll Killer and best Snooper!
    I decide when the pigs fly!
    Rhea's Avatar
    Join Date
    January 18th, 2005
    Location
    New York, USA
    Posts
    6,195
    While we're on the subject of splitting datafeeds...sorry dfl if I'm hijacking your thread...does anyone have any advice on how to split a ginormous CJ datafeed into multiple files? It's too big to fit in Excel (it gets truncated) and I don't have Access.

    Muchos!

  9. #9
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by Rhea
    While we're on the subject of splitting datafeeds...sorry dfl if I'm hijacking your thread...does anyone have any advice on how to split a ginormous CJ datafeed into multiple files? It's too big to fit in Excel (it gets truncated) and I don't have Access.

    Muchos!
    You can request that CJ split up your feeds by merchant. Do all your merchants come in a single file?

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  10. #10
    Troll Killer and best Snooper!
    I decide when the pigs fly!
    Rhea's Avatar
    Join Date
    January 18th, 2005
    Location
    New York, USA
    Posts
    6,195
    Quote Originally Posted by Snib
    You can request that CJ split up your feeds by merchant. Do all your merchants come in a single file?
    Hey Scott, thanks for responding.

    My CJ merchants each have their own feed, although they're bundled together in a single download. But that's not the problem. The problem is that one merchant has a feed that is about five times the maximum records Excel can handle (~65K) so when I open that feed in Excel I end up with only the first 65K records and the remainder are "lost."

    What I'd like to do is have a method that allows me to open that file and have it split into as many separate Excel files as is necessary to capture all 300K+ records.

    Got any magic that can do that?

  11. #11
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by Rhea
    What I'd like to do is have a method that allows me to open that file and have it split into as many separate Excel files as is necessary to capture all 300K+ records.

    Got any magic that can do that?
    Nothing off-hand, but why not try Google Excel or Open Office? They might not have the same limits as Microsoft. The ideal solution is not to use Excel at all and do something like the Bullet Proof Datafeed solution here on ABW.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  12. #12
    Affiliate/AM Moonlighter dflsports's Avatar
    Join Date
    January 17th, 2005
    Posts
    874
    Rhea, try try that file splitter search. It seemed like those programs will split the files into smaller more manageable files and there were at least 2 that are freeware.

  13. #13
    Troll Killer and best Snooper!
    I decide when the pigs fly!
    Rhea's Avatar
    Join Date
    January 18th, 2005
    Location
    New York, USA
    Posts
    6,195
    Will do. Thanks, guys.

    It's not a high priority on my list of 8 bazillion things I want to do by next Tuesday, but I'll bump it up a bit and see if anything lands on me.

  14. #14
    Full Member iolaire's Avatar
    Join Date
    October 3rd, 2006
    Location
    Arlington, VA
    Posts
    229
    Quote Originally Posted by Rhea
    The problem is that one merchant has a feed that is about five times the maximum records Excel can handle (~65K)
    Excel 2007 doesn't have that limitation, but it comes with some drawbacks like no longer opening foxpro data files and such. It still opens tab, csv, etc...

    I'm not a fan of dealing with datafeeds manually, but if you already use excel an upgrade to 2007might be easiest for you.

  15. #15
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    Are you looking to say extract like a single category from a feed of about oh I don't know, say 250,000 products? Then display that single category and discard the rest?
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  16. #16
    Troll Killer and best Snooper!
    I decide when the pigs fly!
    Rhea's Avatar
    Join Date
    January 18th, 2005
    Location
    New York, USA
    Posts
    6,195
    John, extracting a single category from the feed would be ideal, as long as the records output from that category don't exceed the record limit of Excel. I doubt they would though.

    Thanks iolaire, I'll have to check out the features in Excel 2007 and see if that solves my problem.

    This really should be something that the merchant is working on fixing but their affiliate manager hasn't been heard from in months.

  17. #17
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    [QUOTE=Rhea]John, extracting a single category from the feed would be ideal, as long as the records output from that category don't exceed the record limit of Excel. I doubt they would though.[QUOTE]



    Sorry Rhea but for a minute think of me as having a mental block.

    Why is it important NOT to exceed the limits of excel? The reason I ask is that nobody has ANY need to open an excel file to check consistency of a feed if they use the right script.

    Whether using direct ftp or if in a zip, extracting and renaming the csv and uploading the file to a fixed location, you don't open the file in excel, merely rename it. Then when category mapping just use your filter option in the script (if you have one) to just display a single category or to delete a single (or more) category.




    The above shows how to filter a feed to replace sub sets etc


    The above shows how to delete categories or rename categories so only specific categories display. All done in MySQL.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  18. #18
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    The reason I ask why exceeding the limit is so crucial is I am displaying on 1 site a feed of in excess of 140,000 products and I could reduce this without opening excel (having just renamed the file and uploaded) to a category of say 3,000 products without any problem at all.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  19. #19
    Troll Killer and best Snooper!
    I decide when the pigs fly!
    Rhea's Avatar
    Join Date
    January 18th, 2005
    Location
    New York, USA
    Posts
    6,195
    John, I don't work with datafeeds that way. I don't upload everything in a file to my sites and my sites aren't automated. I hand pick the products to put on my sites and which pages they'll go on. So any script that automates the process is useless to me. I have to be able to open the Excel file and manipulate the data.

  20. #20
    Influencer Marketing GravityFed's Avatar
    Join Date
    January 18th, 2005
    Location
    Ithaca & Park City
    Posts
    3,338
    Just saw this thread..

    dflsports, you may be aware already but with all your merchants in AvantLink you have the ability to customize each of your datafeed subscriptions.

    ...by category, sub-cat, product group or brand.. and from data format to column structure. no software needed, just an AvantLink Affiliate account

  21. #21
    Affiliate/AM Moonlighter dflsports's Avatar
    Join Date
    January 17th, 2005
    Posts
    874
    Quote Originally Posted by Gary-AvantLink
    Just saw this thread..

    dflsports, you may be aware already but with all your merchants in AvantLink you have the ability to customize each of your datafeed subscriptions.

    ...by category, sub-cat, product group or brand.. and from data format to column structure. no software needed, just an AvantLink Affiliate account
    Yeah Avantlink has the best tools for datafeeds and creating links, rss, etc... They are light years ahead of anyone else. Well popshops is close but not quite there yet. I am looking for ways to get more traffic to avantlink merchants utilizing those great tools (I was not paid for that endorsement )

    I did try install xampp. Installation was smooth until I hit some error. Searched google, found one solution but it was the most annoying error ever and took way too much time to solve. Ruined my night But I tried another apache server, Wamp, installation went much smoother, just one hiccup getting phpmyadmin to work.

  22. #22
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    I shall reply in a bit with a huge rant on that. When I get home.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  23. #23
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    Actually can't be bothered right now except to say everything tech wise available from any US based firm is three years behind Western Europe and in Europe it's generally free and not monetised. Everything being offered by all the firms in the US is basically open source anyhow with a slight revamp and is generally available free elsewhere.

    As for newer tech, well the US is "light years" behind everyone else.

    Don't believe me....well let me say one final thing. I supply scripts to a $16 Billion corporation and they have used all the so called affiliate tech leaders in the US cos for the last year they've been testing the affiliate market as an affiliate prior to their launching as a major merchant with an affiliate program.

    Lets just say they were not impressed with US led innovation and came to Europe for tech savvy scripts.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

  24. #24
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by John Jupp
    Why is it important NOT to exceed the limits of excel? The reason I ask is that nobody has ANY need to open an excel file to check consistency of a feed if they use the right script.

    The above shows how to delete categories or rename categories so only specific categories display. All done in MySQL.
    Problem is most people don't have the technical or monetary resources to purchase or create "the right script".

    Quote Originally Posted by John Jupp
    Actually can't be bothered right now except to say everything tech wise available from any US based firm is three years behind Western Europe and in Europe it's generally free and not monetised. Everything being offered by all the firms in the US is basically open source anyhow with a slight revamp and is generally available free elsewhere.

    As for newer tech, well the US is "light years" behind everyone else.
    Why the sudden US bashing?

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  25. #25
    Outsourced Program Manager John Jupp's Avatar
    Join Date
    January 23rd, 2005
    Location
    England
    Posts
    1,502
    Nah not US bashing Scott. It'd just been a very long day from 4am to 9pm at the keyboard and, I don't know, reading yet another "pay $$$$$ for this" type system when if people read forums they'd find all the resources listed given for free.

    So apologies to any whom I may have offended. Was not my intent.
    Last edited by John Jupp; February 24th, 2008 at 03:54 AM.
    Flambi Media Limited - USA/UK/EU Affiliate Management Expertise

+ Reply to Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. My Checks Have Been Getting Smaller and Smaller!
    By Cheesehead in forum Virtual Family and Off-Topic
    Replies: 25
    Last Post: June 23rd, 2010, 11:08 AM
  2. Splitting the Value of a Page in the SERPs
    By Kevin in forum Midnight Cafe'
    Replies: 1
    Last Post: April 7th, 2009, 06:19 PM
  3. Datafeeds for Products... but Coupon Feeds?
    By trustful88 in forum Commission Junction - CJ
    Replies: 1
    Last Post: June 22nd, 2008, 06:52 PM
  4. Splitting the Fields in Overstock Feed
    By Rogi in forum Programming / Datafeeds / Tools
    Replies: 9
    Last Post: September 2nd, 2005, 02:59 PM
  5. Splitting up big feeds
    By websmith in forum Programming / Datafeeds / Tools
    Replies: 9
    Last Post: February 16th, 2004, 09:58 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •