Summary

A robots.txt file is a simple file that can be uploaded to your website to aid search engine crawlers, to tell them which pages they can and cannot access. 

The minimum needed for a valid robots.txt file is as follows:

User-agent: *  
Disallow

To any search engine crawlers that find this file, it says every user-agent (the * character) is allowed to crawl every single page on your site (as 'disallow' has nothing specified)

Disallowed Pages

You may wish to add pages that you don't want search engines crawling as they offer no intrinsic value to a search - for example, a login, basket or checkout page. TO do this you can use the disallow tag (one per line) to specify specific pages that are disallowed, e.g.:

User-agent: *
Disallow: /basket
Disallow: /login

Uploading a robots.txt file

The robots.txt file needs to be accessible at the root of your website, e.g. www.example.com/robots.txt
Once you have created the file you will need to upload it to the root of your site's filecache:

You can then point search engines to this file. 

Sitemap

You can also add the location of your sitemap to the robots.txt file which will allow some search engine crawlers to automatically pick up the file. You would need to upload your sitemap to the root of the filecache (the sitemap.xml file shown above ) then add a line to your robots.txt file pointing to this file:

User-agent: *  
Disallow
:
Sitemap: www.example.com/sitemap.xml

Did this answer your question?