Page 1 of 2 12 LastLast
Results 1 to 25 of 27
  1. #1
    Super Sh!t Stirrer SSanf's Avatar
    Join Date
    January 18th, 2005
    Posts
    9,944
    Scraping a site for making my own data feed
    I have a couple of merchants too lazy or stupid to offer a data feed.

    All I want is a list of the product pages and the relevant images. Anyway, how to get this information easily?

    I see free site scraper tools but I am not sure if these will help extract the information or what. Ideas?
    Comments are opinion unless otherwise noted. Remember, pillage first. Then burn. Half of all people in the world have IQs under 100. You best learn to trust ol' SSanf!

  2. #2
    Member
    Join Date
    January 18th, 2005
    Location
    Choctaw, Oklahoma
    Posts
    114
    Ssanf,

    This requires a script. Here is a url:

    http://www.greggdev.com/web/articles.php?id=5

    "This article will attempt to teach you both how to use an affiliate data feed and how to create one from a site that does not offer one."

    "Having trouble on where to begin or how to write the scripts? We can help. We have created scripts to do just what this article describes for dozens of popular websites. The best part is, if you only have a little experience in programming in PHP, you can have us setup the scripts for one site and use them as a model for additional sites that you might want to create a feed from. The scripts we write are fairly portable and easy to follow. Our rates are as low as $30 an hour."

    For a long time I looked and looked for an application I could buy that would do just what you've asked for. This is the only thing I found, but as they say on their website, they do it the first time and show you how to do it yourself for other websites.

    I haven't used them, so this is not a personal recommendation - just something I found.
    Blessed Be,
    White Wolf

  3. #3
    Member
    Join Date
    March 10th, 2005
    Posts
    155
    Could a site scraper be used to copy other sites..... Could this be why we dont see any??

    Would developers/companies shy away from creating it because of possible legal battles and copywrite infringment ???

    Just a thought..


    Kend

  4. #4
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    I've done that before. It's really only effective if the merchant has a small number of items or if you're just doing a one-time scrape. You also have to have pretty extensive error-checking, because a change in the page layout can really mess up your script.

    I've also used similar techniques to supplement datafeeds that had incomplete information. I store the data in a database, and when I run across a new product, I scrape the page to get the missing information.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  5. #5
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by Kend
    Could a site scraper be used to copy other sites..... Could this be why we dont see any??

    Would developers/companies shy away from creating it because of possible legal battles and copywrite infringment ???

    Just a thought..


    Kend
    I think that could happen. An affiliate is much better off contacting the merchant -- after all, the affiliate wants the commission, right? Codes from scraping don't always convert -- am I right?

  6. #6
    Member Chocolate_Chicken's Avatar
    Join Date
    January 19th, 2005
    Location
    The Hen House
    Posts
    1,227
    If it's only a few hundred products, it can be worthwhile to just set up an empty database file and cut/paste the info in manually.

  7. #7
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by Chocolate_Chicken
    If it's only a few hundred products, it can be worthwhile to just set up an empty database file and cut/paste the info in manually.
    Shouldn't you ask the merchant?

    Btw, how would you like it if someone decided to "scrape" a hundred little "factors/style/whatever can be scraped" off of your site? Is that okay?

    If an affiliate would like a direct link to an "amazing widget" or a hundred widgets and the details are coming off of a merchant site, just clear it with the merchant. When in doubt, ask the owner of the site/program/merchant -- you can't go wrong with permission.

  8. #8
    The Eternal Optimist zimmy's Avatar
    Join Date
    January 18th, 2005
    Location
    NC
    Posts
    696
    Shouldn't you ask the merchant?
    It is a good idea to check with the merchant, but remember you aren't stealing from them. They have an affiliate program in place so that we can sell their products.



    ~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*

    The big shots are only the little shots who keep shooting.
    -Christopher Morley

  9. #9
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by zimmy
    It is a good idea to check with the merchant, but remember you aren't stealing from them. They have an affiliate program in place so that we can sell their products.
    They might want to have some things exclusive to themselves or designated other parties.

    Just because you affiliate with them doesn't mean you have free access and/or own everything listed on their site -- but on the other hand, any affiliate who would like to get into legal entanglements --> go right ahead. Go for it baby and then see if there is any problems with the intellectual property stuff.

    Don't blame me, I warned you

  10. #10
    Lite On The Do, Heavy On The Nuts Donuts's Avatar
    Join Date
    January 18th, 2005
    Location
    Winter Park, FL
    Posts
    6,930
    I have seen merchants that forbid this in their terms. I think they worry about duplicate content affecting their own seo efforts (among other things). So on the one's that don't forbid it specficially in their terms, I'd definitely ask first. On the ones that do forbid it, I'd try to discuss it with them why it'd be worth their time to develop a feed for affiliates that recognizes and avoids the legit concerns they may have.

  11. #11
    Action Jackson - King of the World
    Join Date
    January 18th, 2005
    Posts
    2,201
    If I'm an affiliate I have the right to sell anything on their site. Makes no sense that merchants wouldn't want this.

  12. #12
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Nunya, Business
    Posts
    23,684
    I think it's a no-brainer that you would get their permission first.

  13. #13
    Super Sh!t Stirrer SSanf's Avatar
    Join Date
    January 18th, 2005
    Posts
    9,944
    It is quite a few thousand products for each merchant. I don't worry about taking the information because I have used both merchants this way for several years and they pay OK and have never complained.

    I have done it by hand for both several times. But, the update takes days and I am sick of it. I am just looking for an easier way.

    Unfortunately, they never even respond when I ask them about it. However, they are money makers for me so I need to do this.

    Really want to find an answer to this.

    Blessed Be, White Wolf. I am kind of uncomfortable getting int a $30 an hour contract with someone unless I can be in the same room and watch them work!
    Last edited by SSanf; February 2nd, 2006 at 02:19 PM.
    Comments are opinion unless otherwise noted. Remember, pillage first. Then burn. Half of all people in the world have IQs under 100. You best learn to trust ol' SSanf!

  14. #14
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by jackson992
    If I'm an affiliate I have the right to sell anything on their site. Makes no sense that merchants wouldn't want this.
    Are you joking?

    Your rights in terms of access to their products (which often have trademarked trade names) and their intellectual property/content are narrowly defined by the company. You don't have broad rights.

    As Donuts mentioned, you should read the TOS --that's the agreement you click to.

    The company might want a product to be exclusive to their own site or to a designated site of their choice.

    Here is an example of what they can do if you start to take their products/content without explicit permission:

    • They can terminate the Merchant-Affiliate agreement
    • They could contact your host and say that you are hosting copyrighted/trademarked material without permission -- this could possibly result in the termination of your site

    • You could get into an ugly fight with an attorney


    There are other possible outcomes -- do some research.
    Would you like to be informed about intellectual property issues?
    A good start would be to take a look at the Chicago-Kent Journal of Intellectual Property online here:

    http://jip.kentlaw.edu/

    What's the big deal about getting permission so that you can avoid related intellectual property disputes?

  15. #15
    http and a telephoto
    Join Date
    January 18th, 2005
    Location
    NYC
    Posts
    17,708
    Quote Originally Posted by jackson992
    If I'm an affiliate I have the right to sell anything on their site. Makes no sense that merchants wouldn't want this.
    Not always Jack. Lots of merchants have restrictions on some items that are not available for sale or that are non-commissionable. They also have restrictions on items they sell that you do make commission on, but that you are not allowed to have images on your site for. I can name several right off the top of my head, and you are in at least one of the programs! But they also have a datafeed.

    So you DO need to be careful, and just because the merchant has permission to use images, the affiliate may not have the right to!
    Deborah Carney
    TeamLoxly.com BookGoodies.com ABCsPlus.com

  16. #16
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Bottom line: if you want a particular product/image link on your site and/or in the datafeed that is displayed on your site instead of taking it (and at the very least risk annoying the Merchant Affiliate) without permission contact the Merchant and say:

    "I am the webmaster of XYZ.html and I have a section of widgets over here and I would really like to put that purple one of yours into my feed or I would like a product link" or whatever your business goal is.

    Tell the Merchant what you want & where you'll put it or what your intended use is and make a nice brief request.

    The merchant might say yes, the merchant might say no.

    You asked. You did the right thing in this business relationship.

    If the merchant says no -- you can try to appeal at another time. The "NO" might be absolute -- "tough" -- "no can do."

    Work with what you are allotted and/or find similar merchants who might give you more materials and/or who might be more flexible.

    Don't just take it because a merchant could get angry and based upon how much you were taking or how you were using it (i.e. don't showcase the widget in a page devoted to how to make nuclear bombs -- chances are the merchant will turn you down) the merchant could complain to your host and/or initiate some type of legal action (that just gets ugly in these scenerios).

  17. #17
    Super Sh!t Stirrer SSanf's Avatar
    Join Date
    January 18th, 2005
    Posts
    9,944
    I checked, now 3, merchants details pages and no special terms exist beyond the SAS agreement. I used the linking methods for linking to merchant product pages provided in the SAS forum on ABW, here. http://forum.abestweb.com/showthread.php?t=48550

    SAS and SAS merchants are fully aware we can and do create links this way as shown in their forum. SAS has even provided the method. There is no restriction what-so-ever on how many links you can create this way.

    These links should not even be an issue. You should know, by now, that I do not suggest untoward methods.

    I just need a fast way to do them.

    I guess no one here knows.

    So, if I get a scrapper, will it be able to give me a list of product page links and the relevant images? Does anyone know the answer to that question? I am not sure what scrapper programs do exactly and what they can do. However, I am sure there are some legitimate uses for them as well as bad. Has anyone seen these programs at work? The descriptions I have read don't tell enough. They assume previous knowledge.
    Last edited by SSanf; February 2nd, 2006 at 05:39 PM.
    Comments are opinion unless otherwise noted. Remember, pillage first. Then burn. Half of all people in the world have IQs under 100. You best learn to trust ol' SSanf!

  18. #18
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    I doubt you could find a program that would do that "out of the box". You'll need to have someone develop one for you.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  19. #19
    Action Jackson - King of the World
    Join Date
    January 18th, 2005
    Posts
    2,201
    Sorry but I don't hold with the you can only support some products and not others. A merchant who does that doesn't have any business being a merchant.

  20. #20
    Super Sh!t Stirrer SSanf's Avatar
    Join Date
    January 18th, 2005
    Posts
    9,944
    Thanks, Mike.

    That is what I really wanted to know.
    Comments are opinion unless otherwise noted. Remember, pillage first. Then burn. Half of all people in the world have IQs under 100. You best learn to trust ol' SSanf!

  21. #21
    Super Sh!t Stirrer SSanf's Avatar
    Join Date
    January 18th, 2005
    Posts
    9,944
    If anyone wants to write me such a program really, really cheap, please PM me.

    I don't need a whole data feed. Just the URLs to the product buy pages and the related product image URLs and the product name. I can convert the links to affiliate links on my own.

    But, it must work on any site I send it to. So, I guess, the trick is to go to the site and find the pages with buy buttons. Grab the URL of the buy page and grab the product image URL and name then spit them out on a spread sheet.

    I am not even trying to get the merchant's product descriptions. But, to sell it to anyone but me, I guess, you would need to grab that, too. There you could have problems with the merchants, I guess. Because, then you would be copying their sites.

    I am looking for the bare basic minimum to get people to their sites from mine which resembles their sites in no way what-so-ever other than showing their products and leading people to where they can be purchased which is what I am supposed to do.

    If someone does this, I sure think there is a market other than just me.

    Thanks.
    Last edited by SSanf; February 3rd, 2006 at 11:20 AM.
    Comments are opinion unless otherwise noted. Remember, pillage first. Then burn. Half of all people in the world have IQs under 100. You best learn to trust ol' SSanf!

  22. #22
    Newbie
    Join Date
    March 25th, 2006
    Posts
    8
    Ssanf,
    Did you find anyone who can write such a program for you? Thanx.

  23. #23
    Super Sh!t Stirrer SSanf's Avatar
    Join Date
    January 18th, 2005
    Posts
    9,944
    Not yet.

    You want to write one?
    Comments are opinion unless otherwise noted. Remember, pillage first. Then burn. Half of all people in the world have IQs under 100. You best learn to trust ol' SSanf!

  24. #24
    Newbie
    Join Date
    March 25th, 2006
    Posts
    8
    No. I am looking for one too. There is one guy I found in one of the other forums mentioned he can do it. I am exploring it. thanx.

  25. #25
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    You'll never find one that will work on any arbitrary merchant's site and produce accurate results. Just look at Froogle. That's exactly what they try to do. Google has hundreds (thousands?) of PhD's working for them, and they can't get it right. Most of their results come from merchant datafeeds, but they also spider and scrape to get results. Some of those spidered/scraped results are absolutely awful. They'll get the wrong thumbnails, the wrong prices, the wrong product names, etc.

    This is something that you would have to build separately for each merchant, if you want it done right.

    (No, I'm not interested in building it.)
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

Closed Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. The Ethics Vs Site Scraping
    By John Jupp in forum Midnight Cafe'
    Replies: 11
    Last Post: November 18th, 2007, 07:08 PM
  2. Data Feed - Movie Site
    By jbmac in forum Midnight Cafe'
    Replies: 1
    Last Post: February 16th, 2006, 02:13 AM
  3. Making WebPages Using your Data feed. Alternative to Web Merger
    By Indianabuy.Com in forum Programming / Datafeeds / Tools
    Replies: 2
    Last Post: February 1st, 2006, 12:45 PM
  4. Scraping a site?
    By ks11 in forum WebMerge (Fourthworld.com)
    Replies: 24
    Last Post: December 6th, 2003, 07:36 AM
  5. Data Feed for PHP Site?
    By gearguy in forum Programming / Datafeeds / Tools
    Replies: 3
    Last Post: May 26th, 2003, 07:47 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •