Results 1 to 12 of 12
  1. #1
    Newbie
    Join Date
    December 22nd, 2013
    Posts
    3
    Creating database of all products in country with images , desc and UPC.
    Hi

    I'd like to hear your opinion about building database
    of all products in country (starting with US) with their
    images, descriptions and UPC.

    Lets assume that i could code what is required (data parsers, sql etc.)
    but the problem is with getting the data.

    How or where should i get datafeeds from?
    What are costs associates or are there free solutions?

  2. #2
    Newbie
    Join Date
    December 22nd, 2013
    Posts
    3
    Any one was doing such project?

  3. #3
    Member Prosperent's Avatar
    Join Date
    November 29th, 2009
    Location
    CO, USA
    Posts
    820
    Easier said than done. We have done what you describe, but it's with a team of developers, and 4+ years of work. Getting the data is the easy part. It's everything that goes into it after that.

  4. Thanks From:

  5. #4
    Newbie
    Join Date
    December 22nd, 2013
    Posts
    3
    I've read about problems on this forum - 4+ years is a lot. Thank you for the insight.

  6. #5
    Member Prosperent's Avatar
    Join Date
    November 29th, 2009
    Location
    CO, USA
    Posts
    820
    The biggest issue is how fast the data updates. You can't just load the data and call it a day. You need to update thousands of feeds and millions of products on a daily basis. You also need the infrastructure to parse, serve, and store the vast amounts of data. Granted, we serve up over 8 billion queries a month, but it takes 32 high end 16-32 core servers to do so.

  7. #6
    OPM and Moderator Chuck Hamrick's Avatar
    Join Date
    April 5th, 2005
    Location
    Park City Utah
    Posts
    16,646
    There are several successful datafeed affiliates on the forum who have worked for years to develop systems with huge numbers of products. They all run their own server farms.

  8. Thanks From:

  9. #7
    Full Member
    Join Date
    November 21st, 2010
    Posts
    230
    I can't argue if this has actually been done, but I have to take exception to the idea that gathering the data is easy. I have yet to personally see any single database with every UPC code that's being used. Most products I see are more crowd-sourced than anything else and are often substantially lacking.

    Now of course it's possible to combine all datafeeds and extract information that IS available, but it is a far cry from all products to be sure. Something like popshops has done for example. But even that is no simple feat as mentioned. When you consider some of the larger retailers can have hundreds of thousands of SKUs. Walmart's datafeed for example is impossibly huge to deal with under most circumstances (myself, I have to rely on the SOAP/API and breaking keywords down into several categories to even attempt working with it). I can only imagine Amazon's being even tougher to deal with, except that Amazon doesn't really expose a lot and thus has its own challenges to deal with.

  10. #8
    Member Prosperent's Avatar
    Join Date
    November 29th, 2009
    Location
    CO, USA
    Posts
    820
    UPC data is a mess. Most don't provide it, and when they do, it's sometimes wrong, improperly formatted, etc.

  11. Thanks From:

  12. #9
    Newbie
    Join Date
    February 18th, 2014
    Location
    Karachi, Pakistan
    Posts
    4
    So the question still remains there, is there a way that we can get data in a proper structure with UPC & other details so that we can upload it on the website & make some bucks??

  13. #10
    ...and a Pirate's heart. Convergence's Avatar
    Join Date
    June 24th, 2005
    Posts
    6,918
    The answer remains the same - there is no consistency from feed to feed and from network to network. Have to bust your ass to get usable data...
    Salty kisses, Sandy toes, and a Pirate's heart...

  14. #11
    OPM and Moderator Chuck Hamrick's Avatar
    Join Date
    April 5th, 2005
    Location
    Park City Utah
    Posts
    16,646
    I hate to say it but most merchants create a Google feed then try to morph that into a network feed. So if networks standardized on Google feeds you would have their standard which opens a whole other can of worms.

  15. #12
    Affiliate Manager
    Join Date
    January 18th, 2005
    Location
    Los Angeles, California
    Posts
    1,913
    Quote Originally Posted by Chuck Hamrick View Post
    I hate to say it but most merchants create a Google feed then try to morph that into a network feed. So if networks standardized on Google feeds you would have their standard which opens a whole other can of worms.
    https://xkcd.com/927/

    Richard Gaskin
    Developer of WebMerge: Publish any data feed on any site
    http://www.fourthworld.com

  16. Thanks From:

  17. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. How to import so many products into database?
    By hyp959 in forum Programming / Datafeeds / Tools
    Replies: 1
    Last Post: January 8th, 2010, 11:54 AM
  2. Country Home Products - Link Error & Payments
    By spawnua in forum Google Affiliate Network - GAN
    Replies: 5
    Last Post: June 1st, 2009, 01:57 PM
  3. Creating 3 columns for products
    By jackson992 in forum Programming / Datafeeds / Tools
    Replies: 12
    Last Post: September 11th, 2006, 12:01 AM
  4. New to creating a database for datafeeds
    By tsmgroup2 in forum Midnight Cafe'
    Replies: 5
    Last Post: July 10th, 2005, 05:45 PM
  5. Creating a Database...help
    By ricochet in forum Midnight Cafe'
    Replies: 2
    Last Post: December 23rd, 2003, 11:12 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •