A robots.txt file is a simple file that can be uploaded to your website to aid search engine crawlers, to tell them which pages they can and cannot access. 

The minimum required for a valid robots.txt file is as follows:

User-agent: *  
Disallow: 

To any search engine crawlers that find this file, it says every user-agent (the * character) is allowed to crawl every single page on your site (as 'disallow' has nothing specified).

You may wish to add pages that you don't want search engines crawling as they offer no intrinsic value to a search. For example, a login, basket or checkout page. To do this, you can use the disallow tag (one per line) to specify specific pages that are disallowed, e.g.:

User-agent: *
Disallow: /basket
Disallow: /login

The robots.txt file needs to be accessible at the root of your website, e.g. www.example.com/robots.txt. Once you've created the file, you'll need to upload it to the root of your site's file cache.

You can then point search engines to this file. 

You can also add the location of your sitemap to the robots.txt file which will allow some search engine crawlers to automatically pick up the file. You'd need to upload your sitemap to the root of the file cache (the sitemap.xml file shown above) then add a line to your robots.txt file pointing to this file:
​
​User-agent: *  
Disallow:
​Sitemap: www.example.com/sitemap.xml
​