Results 1 to 2 of 2
  1. #1
    Full Member
    Join Date
    January 18th, 2005
    Posts
    218
    Can someone give a quick lesson about Robot.txt? I thought it was a file that was used to prevent robots from crawling a website. Now I heard that it can help.

  2. #2
    Full Member
    Join Date
    January 18th, 2005
    Posts
    462
    Seaslug44,

    Check out this site...

    Hope it helps.

    Core

  3. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Google Robot Searching for robot.txt
    By ahmar in forum Search Engine Optimization
    Replies: 4
    Last Post: December 26th, 2004, 01:26 PM
  2. Robot.txt example...
    By eggerda in forum Search Engine Optimization
    Replies: 7
    Last Post: September 18th, 2003, 11:56 AM
  3. Robot.txt versus Amazon.PL
    By beggers in forum Cusimano.com Scripts
    Replies: 12
    Last Post: March 18th, 2003, 04:16 PM
  4. Robot txt
    By mousejockey in forum Programming / Datafeeds / Tools
    Replies: 12
    Last Post: January 14th, 2002, 05:05 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •