Results 1 to 3 of 3
  1. #1
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Madrid
    Posts
    758
    If I want a specific spider to not visit a directory I would use
    User-agent: spidername
    Disallow: /directory name

    but what if the directory was:

    http://www.domainname.com/index.php/cPath

    Would it be:

    User-agent: spidername
    Disallow: /index.php/cPath

    or

    User-agent: spidername
    Disallow: /index.php

    or

    User-agent: spidername
    Disallow: /cPath

    It was easier with HTML
    Javi

    -------------
    <font size="1">Ladies and Gentleman you are being robbed every day, you know it and you're not doing anything about it. When is enough, enough? </font>

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    Well, the first one would disallow just the cPath directory, so if that's what you want to do, then that's the one to use.

    Search Engine Positioning - 1 Design 4 Life

  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Madrid
    Posts
    758
    Thanks!

    -------------
    <font size="1">Ladies and Gentleman you are being robbed every day, you know it and you're not doing anything about it. When is enough, enough? </font>

  4. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Blocking Certain Forum Pages with robots.txt
    By Rob_BAM in forum Search Engine Optimization
    Replies: 3
    Last Post: May 29th, 2012, 08:58 AM
  2. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  3. Do you use a robots.txt?
    By Mr. Sal in forum Voting Booth
    Replies: 11
    Last Post: November 12th, 2003, 07:29 PM
  4. robots txt
    By reflections in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: December 26th, 2002, 06:22 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •