Thread: Robot.txt file

question no 1

if we want to include all the search engine to index our web site:

is these how the Robot.txt file going to look like

User-agent: *
Disallow:

Question no 2

Is this code simply mean only ia_archiver will not be able to crawl our site and the rest of
the robot can?

User-agent: ia_archiver
Disallow: /

Or do we need to add two more lines like these

User-agent: ia_archiver
Disallow: /

User-agent: *
Disallow:

so we can get the other robot to access our web site...

...and finally, what is the type of file in our website site that we usually don't want the robot to read, so I can put this code like...Disallow:/something/ in...Thanks for the help guys...

Basically, the main function of the robots.txt file is to instruct robots not to crawl and index certain files most particularly password protected folders and folders which contain only images.

I did a blogpost about robots.txt files here. You can also generate it automatically this days with help from Google via your Google webmaster tools

Have you found that using the robots.txt file with disallow effectively works to stop your pages from showing in a search index?

Have you ever used noindex in the robots.txt file? I hear it is only accepted by Google.

Robots.txt is a W3C standard.... it applied to all good crawlers out there so its now used by Google only

Basically, robots.txt file will not do anything to improve your search engine positioning but it indicate robots which files you will not allow to be crawled and indexed in the search engines. Whenever a robot crawls your site it looks for the robots.txt file. If it doesn't find one it assumes automatically that it may crawl and index the entire site.

can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!

Originally Posted by xhan

can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!

No, you can only stop the crawlers from finding pages and therefore they will not be indexed.

Originally Posted by xhan

can you use robots.txt to stop google caching certain phrases? I dont want my website to be found by googling my email address!

In the meta tags for that page you could use noarchive. That tells the SE not to cache your web page. I don't think it can be targeted to a single phrase though.

HTML Code:

<META content="NOARCHIVE" name="ROBOTS">

@surreypcsupport
@azseoguy

cheers guys, i've no archived it - I hope it helps!! :S