Results 1 to 15 of 15
  1. #1
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Creating an Accurate Product Search Engine
    We haven't had any good design and development threads lately so I thought I'd start a thread about building a solid search engine. To this point I've been using a fulltext index on a few key fields like the product name, manufactuter, model number, and category, but I'd like to improve it. What I'm thinking is I'll break the results down by category. Sort of like Shopping.com and Pricegrabber do it. Then to dig deeper my visitors can narrow the search by clicking the resulting categories. I can also show a broad match ignoring the categories like I do now. The only problem with this is the products need to be categorized to appear at the top of the results. In my case it's very difficult to categorize all of my products. I've got a fair chunk mapped to categories, but there are tons more that aren't.

    I was also thinking about sorting my results by a combination of product clicks and fulltext relevance. This way the popular products will appear closer to the top, but that causes a problem if the products being searched for have never been found. I think this solution works best for sites with thousands and thousands of daily visitors, but it may not work too well for a lower traffic site.

    One problem I've found with a fulltext search is products are given authority over other products just because they use a particular keyword more times than once. For example the search query might be "apple ipod" and the top result will be "apple case for ipod that fits any apple ipod" while the actual apple ipod shows up further down the list. I'm having a hard time with this because obviously the actual apple ipod should appear first, but fulltext index doesn't know that. The category solution will work to combat this, but I'm still trying to find more solutions.

    Any comments or feedback? This is hugely important to me because a solid search engine can make or break a site.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  2. #2
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    Giving precedence to items people have chosen in the past with the same search is very, very effective. I take it one step further and if it's a heavily searched keyword where the #1 choice is like 20x as chosen as the #2 choice, I just redirect straight to the product. (Eventually, I'll add an option for them to see the actual search results in that case, but I haven't had any complaints.)

    To handle searches like your "apple ipod" search, I weight the shortest matches higher than the longer ones. Someone searching for "apple ipod" is probably more interested in an "apple ipod" than an "apple ipod cover" or an "apple case for ipod that fits any apple ipod".

    Fulltext is good, but it misses the mark in many areas. Another area it's not that helpful in is misspellings or naming variations. Product numbers can be done so many different ways "name-123" or "name 123" or "name123" or "Name123" or "NAME123", etc. Sometimes it's best to strip out everything but the alphanumeric portion and search on that.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  3. #3
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Those are some great ideas Michael! I like the idea of following the search history. Do you do that with sessions or do you embed the search query into the search results links? I'd probably build a table that lists a search and product that was clicked upon that search. Might be able to take it a step further and break the searches up by keyword. So if you have a large keyword search you know that each of those keywords have something to do with the product clicked. Then you don't need a perfect match from search to search.

    The only problem with this is it won't work very well for new products. Something that was just released won't be at the top of the results because it doesn't have much of a history. I guess to balance this you can show your results based on the past month or two of searches. Probably don't want to be digging up results for items that people were searching for 6 months ago that are no longer in demand. This really depends on the types of products you're selling.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  4. #4
    2005 Linkshare Golden Link Award Winner  ecomcity's Avatar
    Join Date
    January 18th, 2005
    Location
    St Clair Shores MI.
    Posts
    17,328
    I fooled with some search applications for clients that give weight to search terms based upon popularity. Color preferences and other keyword buying terms seemed to work well with very fast full text search results. Try it live at bottom of www.suspenders.com

    The search terms even show up in the server logs aiding in PPCSE bidding. Keyword terms automatically get bolded.
    Webmaster's... Mike and Charlie

    "What have you done today to put real value into a referral click...from a shoppers viewpoint!"

  5. #5
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by ecomcity
    I fooled with some search applications for clients that give weight to search terms based upon popularity. Color preferences and other keyword buying terms seemed to work well with very fast full text search results. Try it live at bottom of www.suspenders.com
    This just looks like a traditional fulltext search. How do you judge popularity? The number of times the keyword appears in the product name doesn't necessarily mean this product is more popular for the keywords than another product.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  6. #6
    2005 Linkshare Golden Link Award Winner  ecomcity's Avatar
    Join Date
    January 18th, 2005
    Location
    St Clair Shores MI.
    Posts
    17,328
    Quote Originally Posted by Snib
    This just looks like a traditional fulltext search. How do you judge popularity? The number of times the keyword appears in the product name doesn't necessarily mean this product is more popular for the keywords than another product.

    - Scott
    The premium script had options for enduser ranking, but I didn't think shoppers would pause to give meaningful rankings. The best solution to what your looking for come integrated into the higher priced shopping carts which rank search keyword results based upon actual top sellers.
    Webmaster's... Mike and Charlie

    "What have you done today to put real value into a referral click...from a shoppers viewpoint!"

  7. #7
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by ecomcity
    The premium script had options for enduser ranking, but I didn't think shoppers would pause to give meaningful rankings. The best solution to what your looking for come integrated into the higher priced shopping carts which rank search keyword results based upon actual top sellers.
    But you still need a certain amount of keyword relevance ranking even when you include top sellers. As affiliates we can track clicks to merchants so that can give us a general idea of the top sellers. What I'm trying to do is leverage that information with the relevance based fulltext search. One thing is for sure, there isn't a single solution to this. The most successful search engines all add their own twist to provide better results.

    Maybe a good solution is to use Michael's idea of history ranked results as a top tier search, then when the results are depleted you can add a combination of relevancy and click popularity. How about amending the fulltext formula slightly. Take the resulting relevancy score and round it down to group more products into the same score. Then sort those products by clicks. And when something is without a doubt the product being searched for send them directly to the product page.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  8. #8
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by Snib
    ...This is hugely important to me because a solid search engine can make or break a site.
    - Scott
    Are you keen on creating your own search engine? What do you think of a "custom" (i.e. search yourdomain.com) Google search for the site?
    ~Rhia7 -- Remember the 7
    Twitter me

  9. #9
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by Rhia7
    Are you keen on creating your own search engine? What do you think of a "custom" (i.e. search yourdomain.com) Google search for the site?
    Problem with the Google search is they control the search data. If you put it in their hands they can dish it out any way they'd like. They also control what pages to index and what pages to exclude. I don't recommend it at all if you want to accurately portray the information on your site. It's always best to have full control over your data and content.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  10. #10
    What's the word? Rhia7's Avatar
    Join Date
    January 13th, 2006
    Posts
    9,578
    Quote Originally Posted by Snib
    It's always best to have full control over your data and content.

    - Scott
    Yes, that would be a plus

    Do you have any opinions on the script engines there are available on script sites?

    Not every affiliate is so advanced with the abilitity to create a search engine
    ~Rhia7 -- Remember the 7
    Twitter me

  11. #11
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    Quote Originally Posted by Snib
    I like the idea of following the search history. Do you do that with sessions or do you embed the search query into the search results links?
    Basically, I do the logging on the product pages. If the referrer page is my search page, I extract the keywords, standardize them (lower case, trim extra spaces, etc.), and log the standardized search term and the product that the user clicked on.

    Then, in the search page, I pass the search terms to the same subroutine to standardize them, select the most common products that people clicked through on for that keyword, and weight those products somewhat higher. Other factors are still considered, so new products will still appear, but over time it helps put the most relevant products first.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  12. #12
    ABW Ambassador MoneyBusiness's Avatar
    Join Date
    March 14th, 2006
    Posts
    2,051
    Good thread guys. Following along with a fine-toothed comb.

    Have you implemented search engines like this on most of your product/datafeed sites?

    Also, are we talking about a fully customized search script? Or was it built off of an existing application?
    Follow my dog, Maya, and I, as we fumble around the affiliate marketing world: www.MarketingMaya.com
    Recent Project: Cheap Personal Checks (using DigitalQuill)

  13. #13
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    For me, it was fully home-grown.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  14. #14
    Newbie TheHoff's Avatar
    Join Date
    March 5th, 2006
    Posts
    260
    One method I use to pull the proper items and combat misspellings is the levenshtein function. If a search result set has 0 items, I'll run the search term against item titles with the levenshtein difference and see if anything is close enough to be a match.

  15. #15
    ABW Ambassador Snib's Avatar
    Join Date
    January 18th, 2005
    Location
    Virginia
    Posts
    5,303
    Quote Originally Posted by TheHoff
    One method I use to pull the proper items and combat misspellings is the levenshtein function. If a search result set has 0 items, I'll run the search term against item titles with the levenshtein difference and see if anything is close enough to be a match.
    That's a very useful function, but the problem with the PHP function is it's not available in SQL queries. I think a better solution is to compile this function into mySQL's runtime library so you can run an SQL query with it. For example:

    Code:
    select * from products where levenshtein(name, '$search_query') <= 1
    This is something I've considered doing for quite some time now, but I just haven't gotten around to it. Rather than using it to search it might be a good idea to use it against previous searches to check the validity of the search query. It's also a very CPU intensive function so you can't go too crazy with it.

    - Scott
    Hatred stirs up strife, But love covers all transgressions.

  16. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Google Search Engine Upgrade For Christmas - Product Searches
    By Trust in forum Search Engine Optimization
    Replies: 16
    Last Post: September 26th, 2006, 10:21 AM
  2. In Search Of A Site Search Engine Spider
    By Heidi in forum Midnight Cafe'
    Replies: 2
    Last Post: September 19th, 2002, 07:13 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •