Results 1 to 7 of 7
  1. #1
    Affiliate Addict Robert484848's Avatar
    Join Date
    January 18th, 2005
    Posts
    334
    My whole family does AFM and we would like to have a site just for us to use as scratch paper. That way we would be able to post our most current link pages and ideas. What I dont want is search engiens to think we are spaming links. Is there a way to block all spiders?
    Thanks all in advance
    Bob

    "I did'nt get where I'm at today by worrying about how I'm going to feel tomorrow."

  2. #2
    Full Member
    Join Date
    January 18th, 2005
    Posts
    315
    Use robots.txt and block all spiders ...but this will not stop the rougue spiders though

    "The Future belongs to those who believe in the beauty of their Dreams."

  3. #3
    ABW Ambassador DesignerWiz's Avatar
    Join Date
    January 18th, 2005
    Location
    U.S.A
    Posts
    2,777
    Here's how to block the spiders. Create a file called "robots. txt" that includes the following code:

    user-agent: * Disallow: /*

    The first line specifies the agents, browsers or spiders that should read this file and adhere to the instructions in the following lines of code. The second line stipulates which files or directories the spider or browser should not read or index. The example above uses the "/*" which means the agent should not read or index anything as the asterisks denotes "everything."

    The robots. txt file must be placed in the root directory of your Web site.

    And the following specific page:

    If you are hosting a Web site on America Online's server or one of the other free or subdirectory Web site services and you can't place a robots. txt file in their root directory, you can use a META tag that talks to some of the spiders:
    META NAME=" ROBOTS" CONTENT=" NOINDEX"

    You will need this META tag on every page in your Web site that you don't want indexed.

    Ray Thomas
    DesignerWiz.com CEO
    Development Resource & Javascript Public Archive Center
    http://DesignerWiz.com
    ABW Board: Category: Programming / Coding

    [This message was edited by DesignerWiz.com Team on October 15, 2003 at 05:02 PM.]

  4. #4
    Affiliate Addict Robert484848's Avatar
    Join Date
    January 18th, 2005
    Posts
    334
    Thanx guys
    I owe you one

    "I did'nt get where I'm at today by worrying about how I'm going to feel tomorrow."

  5. #5
    Full Member
    Join Date
    January 18th, 2005
    Posts
    256
    Put a password on it.

  6. #6
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    It should be:

    User-agent: *
    Disallow: /

    That second * Ray has is unnecessary and - I think - incorrect.

    Search Engine Positioning - 1 Design 4 Life

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,663
    Have had developmental pages blocked by robots.txt show up in Google. Neither Google nor I could explain how it happened. Your best bet is to block the site with global.asa or a .htaccess file depending on your server type.

    Wayne

  8. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. site not indexing in bing
    By specusa in forum Search Engine Optimization
    Replies: 1
    Last Post: May 8th, 2013, 09:48 AM
  2. Google Indexing My Site Search Results
    By John Powell in forum Search Engine Optimization
    Replies: 12
    Last Post: November 6th, 2009, 08:53 PM
  3. Site is being ripped how do i stop it
    By KPR in forum Business & Legal Issues
    Replies: 9
    Last Post: December 24th, 2008, 11:16 PM
  4. Spiders and site copy
    By caranamoore in forum Blocking Tips/Advice/scripts
    Replies: 0
    Last Post: September 24th, 2003, 11:27 PM
  5. $1 to stop parasites on your site?
    By Haiko de Poel, Jr. in forum Midnight Cafe'
    Replies: 107
    Last Post: January 10th, 2003, 05:48 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •