Robots Meta tags and SEO
A lot of sites online will not include a robots tag these days, yours may be one but you should always include a robots Meta tag, it is good practice. If you have a well thought out SEO strategy you will almost certainly need a robots Meta tag in order to exclude non relevant pages from getting indexed. On static websites that don’t work off template files but simply work on different pages with the header built in, implementing as robots Meta tag is fairly straight forward, just place one in each file with appropriate rules.
<meta name="robots" content="index,follow"> // A basic robots tag with index and follow rules
Robots Meta tag rules
The robots Meta tag has two major rules you should learn first and these are the index and follow rules. The index rule specifies whether search engines should cache / save the page and in turn allow it to show in search results such as on Google search. This concept is extremely important and if you don’t understand how indexing works you should read a bit on it. The follow rule specifies whether search engines should follow the links on the page. Configuring the rule to nofollow essentially tells search engines this page is a dead end and they should not follow any links on the page. You can have a combination of rules such as index, follow (index the page and follow the links), noindex,follow (don’t index the page but follow the links to other pages) or index,nofollow(index the page but do not follow the links to other pages). Proper configuration of the robots Meta tag can increase how effectively search engines crawl your site, increase page rank distribution between pages, and exclude non relevant pages from search results.
Creating a robots Meta tag
If your site is a dynamically generated one you can build a simple PHP function in order to generate a robots tag. If you use a template file to generate a lot of different pages and you need to exclude certain ones it can be a bit trickier to implement. In certain cases you can look at building a database column with your index rules and pull them dynamically depending on the page. Of course you would need to implement this functionality into your admin system but if you are an experienced programmer it should be no problem.
When to use a robots Meta tag
If you have an admin system for your website always include a robots tag with noindex,nofollow rules just in case a situation ever arises whereby your admin system pages might be accidentally crawled. If you ever configure your server authentication or similar wrongly you could end up allowing search engines to crawl your admin system and leave a clear route online to where your admin system is and you could even leave yourself open to exposing sensitive information. Although you can use a robots text file to block routes to your admin pages and the pages themselves, adding a robots Meta tag will ensure you have an extra layer of security just in case your robots file is ever misconfigured. Always consider everything that can go wrong may go wrong so protect yourself early to avoid headaches later on.