Results 1 to 3 of 3
  1. #1
    Join Date
    January 17th, 2005
    I'm using the Amazon script on one of my sites and various bots are coming through a little too good.

    Instead of spidering what's there, the bots running a marathon and wandering off into areas unrelated to the site.

    How can I let the bots in so I get indexed but not have them jumping around into tools, dvds, specific author and actor searches etc. that don't relate?

    I'm using the store template with the tabs and re-assigned the tabs to different shop/type_browse/mode_numbers except for the final tab that goes to shop/template_stores

    Any thoughts on how to allow the bot (teoma & msn are mostly running wild) in but not so dang deep?


  2. #2
    ABW Ambassador cusimano's Avatar
    Join Date
    January 18th, 2005
    Toronto, Canada
    In Associate Engine v4.11.17-beta, you can redirect specified nodes/stores to those nodes/stores at the website. For example, adding browse.remote.dvd = yes to ae.ini would cause all pages of the entire DVD store to be redirected to the corresponding page at See AE beta release notes, AE download directory, and list of stores.

    Currently there is no way to limit the depth ("depth" is the number of sub-directories in the navbar path, e.g.: "DVD / Genres / Action & Adventure / Action Directors / Steven Spielberg" has a depth of 5). One problem with depth is that product detail pages always have a depth of 2 (e.g.: "DVD / SOME_MOVIE_NAME").

    Yours truly,
    Cusimano.Com Corporation
    per: David Cusimano
    Affiliate Tools: Datafeed Merge

  3. #3
    Join Date
    January 17th, 2005
    Thanks David, that should help me a lot.

  4. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Blocking bots from spidering
    By beggers in forum Scripts
    Replies: 7
    Last Post: September 19th, 2006, 11:17 AM
  2. Inktomi Spidering finally
    By cjk in forum Search Engine Optimization
    Replies: 1
    Last Post: January 24th, 2004, 08:14 AM
  3. Google Spidering...
    By eggerda in forum Search Engine Optimization
    Replies: 14
    Last Post: September 15th, 2003, 12:44 PM
  4. mega google spidering going on
    By Heyder in forum Search Engine Optimization
    Replies: 8
    Last Post: December 10th, 2002, 11:15 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts