Results 1 to 5 of 5
  1. #1
    Full Member
    Join Date
    January 18th, 2005
    Posts
    315
    Anyone know where to get a search package with a spider (not just one that spiders urls you enter, but follows all the links to spider all the websites it can find on the net) , script for pulling results, automatically adding results from spider to database .etc. all html customisable for free or very cheap.

    How much cpu usage would spider take up? And how much would be the av. disk space required for storage of say 1million results?

    If theres no full search packages around, does anyone know (or use) any metasearch scripts that allow full search results customisation and search a good couple of engines (ie. about 5 or so)

    Thanks,
    John

  2. #2
    ABW Veteran Student Heyder's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,482
    Jq,

    You've probably already checked into the FD search engine but what most people don't know is that you can get a mod to allow it to automatically crawl. Also with just the default set up it can keep going until it runs out of links.

    I don't have cpu stats for you except to say that it will be extremely high. As far as storage space goes it depends on how you would set up you database. By default FD is a text file so it is rather large.

    A rule of thumb for guessing webpage size is to remember that each character of text equals 8 bits and you can either cache the whole page or set the limit to the first (x-amount) of text.

  3. #3
    Full Member
    Join Date
    January 18th, 2005
    Posts
    315
    Hi,

    Thanks for the info.

    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>You've probably already checked into the FD search engine but what most people don't know is that you can get a mod to allow it to automatically crawl.<HR></BLOCKQUOTE>

    What is the FD search engine. Ive looked around but havent come across this yet? Any idea how much it is or where i can find it and the mod required?

  4. #4
    ABW Veteran Student Heyder's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,482
    Fluid Dynamics

    It's free with their copyrights or something like 30 bucks to remove it.

    The mod is free too you just have to do it youself. Read this before you decide to go with it. It is my impression that you can set it up on a unix server to spider on it's own but I could be wrong.

    actually now that I went to go get the link I'm not so sure that it's a true auto crawler in the sense you were speaking but you can check it out by going here http://www.xav.com/scripts/search/help/1087.html

  5. #5
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    Sorry to jq...according to this
    http://www.xav.com/scripts/search/features.html

    it only searches the sites you tell it to.

    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>FDSE is different than Google or Altavista, which search the entire Internet. FDSE only searches the sites that you tell it to. It can handle about 10,000 documents in all, which is plenty for one site but much fewer than the total number of documents on the Internet.~From the page I linked to<HR></BLOCKQUOTE>

    But THANKS HEYDER, because that is EXACTLY what I have been wanting for my own sites!!!

    "Shop at MySites.com, you can find anything you want, just enter your term HERE!"

    Yesss...

    [ 04-04-2002: Message edited by: Leader ]

  6. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Accounts package?
    By perfectG in forum Virtual Family and Off-Topic
    Replies: 4
    Last Post: March 9th, 2003, 04:11 AM
  2. whats up with this package??
    By jq in forum Domains & Hosting
    Replies: 1
    Last Post: January 2nd, 2002, 05:59 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •