Results 1 to 16 of 16
  1. #1
    ABW Ambassador
    Join Date
    November 1st, 2005
    Posts
    2,064
    Way to Get Large Datafeeds Compressed Like Sears etc?
    I would like to know if anyone has come across a program or website that will reduce large datafeeds for affiliates. I know Avantlink have a great way of setting up datafeeds as you can filiter them to only get what you want. As far as I know they are the only network that offer this essential part of getting datafeeds but the problem is I need to also do this with other merchants on other networks too. Goldencan let you filter out a datafeed but the problem is they turn it into a premade store where all I want is the raw datafeed file to use. In Goldencan I can get Sears datafeed into subcategories etc so all I get is a small part of their catalog. If I was to download the CJ Sears datafeed it would be huge. I'm only using Sears as an example, their are lots of merchants that have datafeeds that are too large for the everyday affiliate. So what options are out there?
    Hosting Discounts from Professional Rates Hosts - Deals and Coupons on Domain Names from GoDadday, Namecheap, Domain.com and more top registers.

  2. #2
    ABW Ambassador Bob Lawrence's Avatar
    Join Date
    July 2nd, 2007
    Posts
    1,090
    Don't know of any service like that but on Linkshare there's away to pull certain parts of the feed. There's a text file in each folder that has the numbers of the category and how many products are in that category.
    If you have done so already, download their merchandiser pdf file and it explains it better than I can off the top of my head.
    Where's the Great Life of Affiliate Marketing Hiding?


  3. #3
    ABW Ambassador
    Join Date
    January 4th, 2006
    Location
    USA
    Posts
    2,477
    Large datafeeds are truly pain in the butt, especially for people who are not building coverall comparison sites.

    Take Hayneedle as an example, they used to have dozens of separate sites with much smaller feeds when they were with PJ/EEN. After they moved to CJ, their feed become a monster one. I only use a specific part of their giant feed. I've asked CJ support to see if they can split the feed since they already done so with PJ, they said they will look into it, and it was three months ago...

  4. #4
    Pimp Duck popdawg's Avatar
    Join Date
    January 18th, 2005
    Location
    Take off eh?
    Posts
    3,249
    Could always make your own.
    A simple MySQL database where you would have a list of merchants and categories. Then a script that opens the file, extracts what categories you want and dumps the data into a separate table or file depending on how you want to access it.
    I've worked with feeds as small as a few rows and as large as, well sears and walmart and ... wait for it ... Overstock.com. Yes this goes back a ways, but I am also doing a version of this on a larger scale today as I reboot my affiliate marketing career.
    And yes, it's an extra step that the merchant/network could eliminate, but I have always looked at it like this, give me all the data and let me use it the way I see fit for my business to both our advantage.
    I have worked with merchants who only gave a "manageable" chunk of data and many times there was lots left out based on what they thought was important to have included.
    I use MySql & PHP personally, have done it with MSSql, MS Access, heck a hundred years ago I did it with some tool that split text files up into chunks small enough for Excel to handle ... that sucked.
    You already handle feeds, more than likely the time investment wouldn't be huge to whip something like this out.
    Just a thought ... I'll go climb back in my fossil now.
    ================================================================
    Been away, now I'm back. Not as much, but I'm back & starting from scratch. Where I was, was fantastic. Where I am now, less so. Things have changed, become harder. So have I. Game ON!!!
    ================================================================

  5. Thanks From:

  6. #5
    The affiliate formerly known as ojmoo
    Join Date
    January 18th, 2005
    Posts
    1,466
    Glittered, popdawg is right. BUt the question is what is the level of your programming sophistication. If you know what your doing, then the size of the datafeed doesn't matter. Once the feed is downloaded, no matter what the size, you still have to go through it line by line and put what you want into your database and ignore the rest.

    The problem with very large datafeeds isn't the size of the the feed its that the information is spread out. Here is what I mena in my case. I need dresses, on a 'small' each dress is listed separately and it contains the color/size information in one feild each no matter if there are 5 colors and 12 sizes. This is what i need. But on large datafeeds (and even some small ones) the reason they are large is because they have the same dress listed several times for each color and each size. This is why they are huge, you have the same silly dress listed 20-100 times. With the same picture regardless of the size/color. This is where it gets tricky.

    But even if you have something that breaks the feeds in parts it won't help. But this being said what does it matter if its 100 records or 1,000,000. It takes no time to ignore the products you don't want when going through the 1,000,000. I assure you it takes my processing program longer to take 1,000 products out of a 2,000 feed (this time has to do with modifying/deleting the data in the database) that it does to take no (or one or 2) products out of a million product feed.

    If you are a php novice PM me and I can write you something to cut a datafeed down to size.
    Expert who says Moo

    a.k.a. OJMOO

    Cow Dance


  7. #6
    The affiliate formerly known as ojmoo
    Join Date
    January 18th, 2005
    Posts
    1,466
    OK it was simple enough to write. Here is a program I call cull.php. If you type in
    cull.php?term=TERM&datafeed=DATAFEED.txt where you replace TERM with a word or phrase you are looking for and DATAFEED with the name of the datafeed you want, it'll create a subset of that datafeed called DATAFEED_small.txt that contains the records with that keyword or phrase somewhere in that line. It works pretty quick when I tested it on the target datafeed. If you need any more assistance PM me.

    <?php
    ini_set('max_execution_time', 120);
    ini_set('display_errors', 1);
    error_reporting (E_ALL & ~E_NOTICE);
    chdir("/home/ojmoo/public_html/uploads");
    ?>
    <html>

    <head>

    <meta http-equiv="Content-Language" content="en-us">
    <meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
    <?php
    print "start<br>\r\n";
    $term = $_GET['term'];
    $datafeed = $_GET['datafeed'];
    $newdatafeed = str_replace(".txt","",$datafeed) . "_sml.txt";
    $cnt = count($datafeed);
    $reccnt = 0;
    $ttlrec = 0;
    $gfp = fopen($newdatafeed,"w");
    if ($fp = fopen($datafeed, 'r')) {
    $thedata = fgets($fp,65536);
    while (!feof($fp)) {
    $ttlrec++;
    if ( strpos(strtolower(" $thedata"),strtolower($term)) > 0 ) {
    fwrite($gfp,"$datafeed $thedata");
    $reccnt++;
    }
    $thedata = fgets($fp,65536);
    }
    fclose($fp);
    print "Datafeed $newdatafeed created with $reccnt records<br> Original datafeed $datafeed had a total of $ttlrec records<br>\r\n";
    }
    fclose($gfp);
    print "done<br>\r\n";

    ?>
    </body>
    </html>
    Expert who says Moo

    a.k.a. OJMOO

    Cow Dance


  8. Thanks From:

  9. #7
    The affiliate formerly known as ojmoo
    Join Date
    January 18th, 2005
    Posts
    1,466
    Unhappy
    It removed the indentation of the program in the quotes. I'm sorry.
    Expert who says Moo

    a.k.a. OJMOO

    Cow Dance


  10. #8
    Pimp Duck popdawg's Avatar
    Join Date
    January 18th, 2005
    Location
    Take off eh?
    Posts
    3,249
    Change the chdir path as well to match your particular path.
    Who says ABW is dead? Nice Moo
    ================================================================
    Been away, now I'm back. Not as much, but I'm back & starting from scratch. Where I was, was fantastic. Where I am now, less so. Things have changed, become harder. So have I. Game ON!!!
    ================================================================

  11. Thanks From:

  12. #9
    Pimp Duck popdawg's Avatar
    Join Date
    January 18th, 2005
    Location
    Take off eh?
    Posts
    3,249
    sorry ... oranges
    ================================================================
    Been away, now I'm back. Not as much, but I'm back & starting from scratch. Where I was, was fantastic. Where I am now, less so. Things have changed, become harder. So have I. Game ON!!!
    ================================================================

  13. #10
    The affiliate formerly known as ojmoo
    Join Date
    January 18th, 2005
    Posts
    1,466
    Sorry about the chdir, I had modified a program I had coded b4 and it slipped through. You don't need the chdir at all.

    I answer to both MOO and Oranges as you can tell from my directory structure ;-)
    Expert who says Moo

    a.k.a. OJMOO

    Cow Dance


  14. #11
    ABW Ambassador
    Join Date
    November 1st, 2005
    Posts
    2,064
    hey thanks everyone for the feedback and Oranges I will have a look at this now and see if I can work with it... I will report back later..

    I also should mention how I work generally is that I like to download a merchants datafeed first up and then import it into an access database or even view it in Excel. I like to take a good look at it visually and see how its setup and look especially at the categories etc and see how I can work them into my site.

    Once I have what I want in the feed then I Import it into my site with MySQL and some plugins etc. My hosting is not dedicated so I have some limits and I don't want to bog down the server with loading datafeeds like Sears over several gigs just to find about 20 specific products. I will filter the data out on import but it still loads the server up if its a huge file I believe. I will manually import the feeds and play around with them for sometime and then eventually it will be put on a cron job once its working perfectly.

    I'm not a programmer by birth, I just hack away and get things done as I need to lol...

    Anyway I will take a look at the above now....
    Hosting Discounts from Professional Rates Hosts - Deals and Coupons on Domain Names from GoDadday, Namecheap, Domain.com and more top registers.

  15. Thanks From:

  16. #12
    OPM/Moderator Hectic GHC's Avatar
    Join Date
    July 13th, 2006
    Location
    Roswell, GA
    Posts
    2,643
    As the manager for Sears PartsDirect, I'm pushing them to decrease the size of their datafeed, excluding products with no images. Most OEM for parts don't include pictures. I'd rather see a datafeed of 20,000 than 3 million. But, it's a long-term process to get it updated.
    Greg Hoffman
    Affiliate Marketing Advocate of the Year 2016; Best OPM/Agency - 2014; Best OPM/Agency, Five Years in a Row - ABestWeb.
    Visit Greg Hoffman Consulting

  17. Thanks From:

  18. #13
    ABW Ambassador
    Join Date
    November 1st, 2005
    Posts
    2,064
    I just downloaded Cafepress datafeed and it is actually over 1 gig when unzipped or 143 megs zipped. I only wanted to find a few select products in the feed including my own designs so I can take a look and see if they are in there. I would then work out how to filter them out or delete the other products in the text file. But I can't even open it in Excel, I get the following error:

    This message can appear due to one of the following:
    The file contains more than 1,048,576 rows or 16,384 columns. To fix this problem, open the source file in a text editor such as Microsoft Word. Save the source file as several smaller files that conform to this row and column limit, and then open the smaller files in Microsoft Excel. If the source data cannot be opened in a text editor, try importing the data into Microsoft Access, and then exporting subsets of the data from Access to Excel.
    The area that you are trying to paste the tab-delineated data into is too small. To fix this problem, select an area in the worksheet large enough to accommodate every delimited item.
    Notes
    Excel cannot exceed the limit of 1,048,576 rows and 16,384 columns.
    By default, Excel places three worksheets in a workbook file. Each worksheet can contain 1,048,576 rows and 16,384 columns of data, and workbooks can contain more than three worksheets if your computer has enough memory to support the additional data.
    Can anyone recommend a way for me to be able to open this feed so I can the products I want?
    Hosting Discounts from Professional Rates Hosts - Deals and Coupons on Domain Names from GoDadday, Namecheap, Domain.com and more top registers.

  19. #14
    ...and a Pirate's heart. Convergence's Avatar
    Join Date
    June 24th, 2005
    Posts
    6,918
    You'll want a file parser.

    Found this on the forum, don't know if it's still available:

    http://www.abestweb.com/forums/inner...am-136663.html
    Salty kisses, Sandy toes, and a Pirate's heart...

  20. Thanks From:

  21. #15
    ABW Ambassador
    Join Date
    November 1st, 2005
    Posts
    2,064
    thanks I will check it out over the weekend
    Hosting Discounts from Professional Rates Hosts - Deals and Coupons on Domain Names from GoDadday, Namecheap, Domain.com and more top registers.

  22. #16
    ABW Ambassador
    Join Date
    October 22nd, 2006
    Posts
    1,065
    Notepad++ is an open src text editor. You can open and edit large files with it.

  23. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Sears Affiliate Program (large screeshot)
    By knight01 in forum Google Affiliate Network - GAN
    Replies: 0
    Last Post: January 11th, 2010, 04:42 PM
  2. Product Feeds XML-Compressed
    By QiSoftware in forum Affiliate Window - AW
    Replies: 1
    Last Post: March 29th, 2009, 10:31 PM
  3. Running through large datafeeds
    By Snib in forum Programming / Datafeeds / Tools
    Replies: 25
    Last Post: January 9th, 2007, 01:45 PM
  4. Split up the directories when working with large datafeeds?
    By Nintendo in forum WebMerge (Fourthworld.com)
    Replies: 1
    Last Post: August 15th, 2004, 05:32 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •