SEO : Robots.txt

Every site needs a robot.txt file located in its root folder. This simple text file tells the search engine spider what content it is allowed to index, and what it is not.

If your website doesn't have one you could have real problems, as it is the first thing a spider looks for on its arrival.

Check if your site has a robots.txt file

Type in www.your-site-name.com/robots.txt (where 'your-site-name' is, of course, your domain name) and you should see some text that starts with "User-agent". If not you don't have one.

How to create a robots.txt file

1. Using notepad or something similar create a new file
   
   
2. Put the following (which basically translates as "let all spiders index my site and don't hide anything")
   
   
   
   User-agent: *
   Disallow: 
   
   
   
3. Save the file with the name robots.txt
   
   
4. Upload it to the root folder of your website
   
   
5. Browse to www.your-site-name.com/robots.txt - you should see the user agent text (if not you've done something wrong)
   

You would be surprised how many professional web design companies forget this simple step!

How to prevent spiders indexing certain files or folders

Let's suppose our site has an admin folder, as well as a members folder with sub folders that we don't want to show up in Google searches.

Simply update your robots.txt file to read:

User-agent: *
Disallow: /admin
Disallow: /members

And if we wanted to disallow access to a specific page as well (eg. login.html) then
we put:

User-agent: *
Disallow: /admin
Disallow: /members
Disallow: login.html

Stopping certain search engines

Quite why you would want to do this we're not sure, as we all want more traffic... but a good example of how to block certain engines and allow others through can be found at: www.robotstxt.org/robots.txt

	Do... test your robots.txt file using Google Webmaster Tools
	Don't... forget to upload the file to the root of your website!

www.robotstxt.org - pretty much everything you need to know about writing a robots.txt file Google Webmaster Tools - test your robots.txt file to check it is working correctly