Robots.txt file is a simple text file uploaded to the root of a website to control the activities of robots. You can use a robots.txt file to allow or deny access to certain parts of you website. Most of the time webmaster make the mistake of calling robots.txt file robot.txt file, if you should name you robots.txt file incorrectly robots will not find it hence it you will be unable to control them.
A simple robots.txt file that allows all robots to crawl your website can be found below, just copy it into a text file and upload it to the root of you website.
-----------------------------------------------------------
# robots.txt from www.ukwebmasterworld.com
User-agent: *
Disallow:
------------------------------------------------------------
Some popular search engine robots as follows:
Google Googlebot
Inktomi Slurp
MSN MSNBOT |
