All Collections
eCommerce
The robots.txt file (SEO)
The robots.txt file (SEO)

This article will explain what a robots.txt file is and how to create one.

Paul Holland avatar
Written by Paul Holland
Updated over a week ago

A robots.txt file is a simple file that can be uploaded to your website to aid search engine crawlers, to tell them which pages they can and cannot access. 

The minimum required for a valid robots.txt file is as follows:

User-agent: *  
Disallow

To any search engine crawlers that find this file, it says every user-agent (the * character) is allowed to crawl every single page on your site (as 'disallow' has nothing specified).

Disallowed Pages

You may wish to add pages that you don't want search engines crawling as they offer no intrinsic value to a search. For example, a login, basket or checkout page. To do this, you can use the disallow tag (one per line) to specify specific pages that are disallowed, e.g.:

User-agent: *
Disallow: /basket
Disallow: /login

Uploading a robots.txt file

The robots.txt file needs to be accessible at the root of your website, e.g. www.example.com/robots.txt. Once you've created the file, you'll need to upload it to the root of your site's file cache.

You can then point search engines to this file. 

Sitemap

You can also add the location of your sitemap to the robots.txt file which will allow some search engine crawlers to automatically pick up the file. You'd need to upload your sitemap to the root of the file cache (the sitemap.xml file shown above) then add a line to your robots.txt file pointing to this file:

User-agent: *  
Disallow
:
Sitemap: www.example.com/sitemap.xml

Did this answer your question?