I did a blogpost about robots.txt files here. You can also generate it automatically this days with help from Google via your Google webmaster tools

Have you found that using the robots.txt file with disallow effectively works to stop your pages from showing in a search index?

Have you ever used noindex in the robots.txt file? I hear it is only accepted by Google.

Robots.txt is a W3C standard.... it applied to all good crawlers out there so its now used by Google only

Basically, robots.txt file will not do anything to improve your search engine positioning but it indicate robots which files you will not allow to be crawled and indexed in the search engines. Whenever a robot crawls your site it looks for the robots.txt file. If it doesn't find one it assumes automatically that it may crawl and index the entire site.

can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!

Originally Posted by xhan

can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!

No, you can only stop the crawlers from finding pages and therefore they will not be indexed.

Originally Posted by xhan

can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!

In the meta tags for that page you could use noarchive. That tells the SE not to cache your web page. I don't think it can be targeted to a single phrase though.

HTML Code:

<META content="NOARCHIVE" name="ROBOTS">

@surreypcsupport
@azseoguy

cheers guys, i've no archived it - I hope it helps!! :S