Results 1 to 6 of 6
  1. #1
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    I'm very new to Amazon and have only just begun using their XML service. It's pretty neat, but there are definitely trade-offs. The biggest I've found is the 10 product per query limit. No UPC search on most categories either. I wouldn't mind dynamically calling their server as one of my users views an Amazon product, but without a UPC search there's no clean method to do a price comparison. Because of this I need to download and cache their entire database, 10 products at a time. It's an interesting challenge and I've got some good ideas to deal with it, but is this the only way? In the long run I'll need to dedicate a server just to this process. We'll see what it's worth.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  2. #2
    AffiliateMan
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    19
    You don't need to download all the products and your really don't want to try. Amazon only allows you to cache information for 24 hours, so you'd have to download information about millions of products each day - kind of defeating the purpose!

    Caching your AWS queries is pretty simple. You can see examples of how to it in the PHP source for my scripts at:

    http://www.browsenodes.com/proxy/
    http://www.browsenodes.com/contextads/

    -- Roger

  3. #3
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    I realize I need to download a million products a day, but I'm looking for a more efficient way to do it. As it stands now I can only get 10 products at a time from Amazon's XML server. I know I'm going to need a dedicated server just for this process, but sometimes you just have to do what you have to do.

    I looked at your scripts and I think you're only caching the results that are manually pulled, correct? I need to get *all* of Amazon's data if I'm going to use it effectively.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  4. #4
    AffiliateMan
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    19
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Snib:
    I realize I need to download a million products a day, but I'm looking for a more efficient way to do it. <HR></BLOCKQUOTE>

    You're only allowed to make, on the average, one AWS call per second. That means a maximum of 86,400 calls per day. Each call returns, at most, 10 products. So the most products you can download/update per day is 864,000. You're not going to be able to get around that according to Amazon's Terms of Service.

    So, instead, you need to think about how to use AWS as it's intended. Call it when you need the information from Amazon.com and cache the calls. It'll look to your visitors as if you've downloaded all the products (see store.jadebox.com for an example).

    -- Roger

  5. #5
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    I realize the 864,000 product limit that Amazon's terms infers but they need to take into account a little bit of leniency. Even if I cache products as people request them, eventually I'll have all of their products cached. I'll still need to update my cache every day and in the long run it'll be over 864k products in the cache.

    There's also the overhead of hitting Amazon's AWS server every time a user makes a query on my server. This uses up precious resources on my user interface server. This server is already working hard so I definitely need to move this processing to another server.

    Additionally, Amazon's AWS tool does not allow UPC searches on most categories. It only works for Books if I remember correctly. This being the case, I'm going to need to have all of the data available with UPC codes if I'm going to be able to search it. The only way to do this is to cache it all because the heavy results contain UPC's. The way I see it, the only way to get a searchable database of UPC's is to make my own.

    Thanks for the responses, I'm glad to have somebody to talk to about this. I know it's a complicated system and many people just don't know how or aren't willing to deal with it.
    Hatred stirs up strife, But love covers all transgressions.

  6. #6
    AffiliateMan
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    19
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>There's also the overhead of hitting Amazon's AWS server every time a user makes a query on my server. This uses up precious resources on my user interface server. This server is already working hard so I definitely need to move this processing to another server.
    <HR></BLOCKQUOTE>

    Just think what the load would be if your server were downloading product information 86,400 times a day. :-)

    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>The way I see it, the only way to get a searchable database of UPC's is to make my own. <HR></BLOCKQUOTE>

    I haven't heard one way of the other but it's reasonable to suspect that AWS 4, which is coming soon (maybe), *might* support UPC searches for all modes.

    Until then you could store a table of UPC/ASIN relationships. That would be much smaller and easier to maintain than trying to store all the information for every product. Then call AWS to fill in the details when you need to display a product.

    -- Roger
    http://www.browsenodes.com/

  7. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. WordPress Caching PlugIns comparison
    By Lanny in forum Blogging, Mobile and Social Media
    Replies: 0
    Last Post: February 11th, 2011, 11:04 AM
  2. Caching thumbnails on the fly -- Scott (Snib)
    By micheck in forum Programming / Datafeeds / Tools
    Replies: 9
    Last Post: February 22nd, 2010, 04:08 PM
  3. Speed up site (caching?)
    By hovesh07 in forum PopShops
    Replies: 0
    Last Post: December 8th, 2009, 03:05 PM
  4. Geo Caching
    By Doug247 in forum Midnight Cafe'
    Replies: 3
    Last Post: September 5th, 2007, 12:10 AM
  5. Replies: 2
    Last Post: May 25th, 2004, 08:57 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •