Results 1 to 4 of 4
  1. #1
    Full Member
    Join Date
    January 17th, 2005
    So I got some datafeeds up and running and decided I needed a good way to search the ENTIRE site.

    I realized my folly of custom tables per merchant, so now I can't really use mysql fulltext with my current database schema reliably. It seemed like a good idea at the time because my import script took only a few seconds per datafeed. (relatively small ones like 2000-5000 products each)

    I've tried a few scripts that had a spider to index my whole domain... it's still going but a this rate it won't be done for maybe a day. Is there any fast search scripts available with fast site indexing or are all spider type scripts quite slow?

  2. #2
    ABW Ambassador DesignerWiz's Avatar
    Join Date
    January 18th, 2005
    Hello Canadian eh, I think that you have selected the right type of "search site" script function that will collect your entire site data from an internal server crawl for your need.

    Unfortunately, when you get over 20k pages it does seem like it takes an eternity but is the small price you pay for "internal page crawl search scripts".

    I don't know how many pages your crawling, but the average time we see per 1k pages is about 3-4 minutes.
    Ray Thomas
    Webmaster Resources:
    ABW Board Category: Programming / Coding

  3. #3
    Full Member
    Join Date
    January 17th, 2005
    1k pages every 3-4 minutes? this a custom script or downloadable one?

    The one I used took something like 30+ hours for approximately 8000-8500 pages. Although it searched like a browser following links and the vast majority of those pages are dynamically generated.

    One of the tables it uses has over 1.5 million rows now.

    Today I added another few thousand pages and I'm hesitant to update the search database since it might start from the beginning :P

  4. #4
    Full Member c4's Avatar
    Join Date
    January 17th, 2005
    I sent you a PM Canadian eh
    [URL=]Help desk software[/URL]

  5. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Search for AE Script
    By bandi in forum Scripts
    Replies: 1
    Last Post: October 13th, 2009, 05:04 PM
  2. Search Script Help
    By PPCguy in forum Programming / Datafeeds / Tools
    Replies: 2
    Last Post: March 13th, 2006, 01:35 PM
  3. Good search script
    By Wayder in forum Programming / Datafeeds / Tools
    Replies: 0
    Last Post: November 29th, 2002, 04:17 PM
  4. Perl Script Search
    By andyjsim in forum Programming / Datafeeds / Tools
    Replies: 11
    Last Post: December 1st, 2001, 01:33 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts