This article from search engine land offers lots of information on Robots.txt:

A Deeper Look At Robots.txt