Results 1 to 15 of 15
  1. #1
    Member Bob_A's Avatar
    Join Date
    January 18th, 2005
    Posts
    73
    Question Need help with large data feed
    I received this large data feed it is about 200Mb. What is the best way to manage this? Is there a way I can chop this up in to smaller pieces and if so what is the best software to use?


    Bob_A

  2. #2
    ABW Ambassador AddHandler's Avatar
    Join Date
    January 19th, 2005
    Posts
    1,270
    Feeds that big are usually garbage any way..
    80% of the 200MB is probably DUPLICATE PRODUCTS..

  3. #3
    Member SeanW's Avatar
    Join Date
    July 28th, 2005
    Posts
    69
    Quote Originally Posted by Bob_A
    I received this large data feed it is about 200Mb. What is the best way to manage this? Is there a way I can chop this up in to smaller pieces and if so what is the best software to use?
    Bob_A
    It depends on what format it's in, and how you plan to use it. For instance if it's CSV you can easily chop it up or just filter on what you want:

    type feed.csv | find "Television" > tv.csv

    Or, depending on how you're parsing it, you may not need to read it in all at once.

    Sean

  4. #4
    Newbie
    Join Date
    February 6th, 2005
    Location
    North Carolina
    Posts
    16
    Split Datafeed into small parts
    I can split your datafeed into 20 or more seperate csv files
    can I see a sample of your datafeed

    Thanks Tommy

  5. #5
    Member Bob_A's Avatar
    Join Date
    January 18th, 2005
    Posts
    73
    Tommy
    Do you have some special software that does this?

  6. #6
    Crazy like a fox suzigeek's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,096
    I use access on large files and then apply filters to get the data I want.

    If your somewhat familiar with microsoft access and have it, it is not to difficult to do.

    I can give more detailed instructions if you go that route.
    Suz~~GearGirl~~

  7. #7
    Newbie
    Join Date
    February 6th, 2005
    Location
    North Carolina
    Posts
    16
    Split Datafeed into small parts
    Quote Originally Posted by Bob_A
    Tommy
    Do you have some special software that does this?
    affiliate datafeed csv splitter

    Im working on a datafeed tool site. It is not finished so please over look.
    But the datafeed csv splitter works great and is free. You can split your affiliate datafeed here http://data-feed.net/csv-split/index.php

    if you have any questions or help please email me tommy@scriptfeed.com

  8. #8
    Member Bob_A's Avatar
    Join Date
    January 18th, 2005
    Posts
    73
    Hi Suzigeek
    Thanks for the offer. I sent you a PM.

    Scriptfeed thanks for the link.

    Bob A

  9. #9
    Newbie
    Join Date
    February 6th, 2005
    Location
    North Carolina
    Posts
    16
    Quote Originally Posted by Bob_A
    Hi Suzigeek
    Thanks for the offer. I sent you a PM.

    Scriptfeed thanks for the link.

    Bob A
    Bob Your welcome
    Did the tool work ok for you?

  10. #10
    Newbie
    Join Date
    February 10th, 2005
    Posts
    8
    If you are using linux/unix, you can use the split command. Type "man split" for details.

    If you are on windows, you can get the split command via the cywin tools (http://www.cygwin.com/).

    You can specify how big each chunk will be...

  11. #11
    Member
    Join Date
    October 23rd, 2005
    Location
    NYC
    Posts
    65
    if you are using windows and just want to view it...try http://www.textpad.com/ which will keep on kicking until it freezes ;-)

  12. #12
    Full Member heisje's Avatar
    Join Date
    January 18th, 2005
    Posts
    314
    .

    ms access, as mentioned before - simple, really . . .

    heisje

    .

  13. #13
    Member
    Join Date
    October 23rd, 2005
    Location
    NYC
    Posts
    65
    awww...everything is simple for you ;-)

  14. #14
    Full Member heisje's Avatar
    Join Date
    January 18th, 2005
    Posts
    314
    .

    ahh, the complexities of simplicity . . . .


    heisje


    .

  15. #15
    Member
    Join Date
    October 23rd, 2005
    Location
    NYC
    Posts
    65
    haha

  16. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Extract Data from a Data Feed
    By TalhaA in forum Programming / Datafeeds / Tools
    Replies: 6
    Last Post: February 25th, 2012, 05:40 PM
  2. Data Feed - Invalid Compressed Data in .gz files?
    By Bleddyn in forum Rakuten LinkShare - LS
    Replies: 7
    Last Post: November 29th, 2010, 09:30 AM
  3. Data feed is too large!
    By SSanf in forum BettyMills
    Replies: 9
    Last Post: October 26th, 2003, 08:24 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •