How to Use the Robots Meta Tag
The Robots meta tag has been around for ages, but some new features introduced in 2006 and 2007 have made it a lot more useful than it used to be. Here’s the 4-1-1.
Robots meta tag basics
The robots meta tag is a part of a web page that communicates information to search engine robots. The robots meta tag is written within the <head> of an HTML file and looks something like this:
<meta name=”robots” content=”noindex, nofollow, noodp, noydir, nosnippet”>
Here is what you can put inside the “content” area:
- “noindex”– this will prevent search engines from indexing the page
- “nofollow” — this will prevent search engines from following any link on the page
- “noarchive” — this will prevent search engines from keeping a cached version of the page in search results
- “nosnippet” — this will prevent a description from appearing on your site’s listing on search engine results pages
- “noodp” — this will prevent search engines from displaying the Open Directory title or description on your site’s listing
- “noydir” — this will prevent Yahoo! from displaying the Yahoo! Directory title or description on your sites listing
- “none” – this is the equivalent of “noindex, nofollow”
- “noimageindex” – this will prevent search engines from indexing images on the page
Excluding a page from indexing with NOINDEX
To exclude a page from indexing, use the following format: <meta name=”robots” content=”noindex”> .
This type of exclusion is an alternative to excluding with the robots.txt file. Differences between the two methods of exclusion can be seen in the following table: (note, this table applies to Google only).
Table: Google Robots Exclusion with the Robots Meta Tag
| Feature | Robots.txt exclusion | Robots meta tag exclusion |
|---|---|---|
| Page accumulates PageRank? | yes | yes |
| Page listed in Google? | the URL may be displayed in listings | no |
| Page passes PageRank through outbound links? | no | yes! |
Improving Search Engine Listings with NOODP and NOYDIR
Some websites are saddled with outdated titles and descriptions from the Yahoo! Directory or the Open Directory. This is a very simple problem to solve, using the “noodp” and “noydir” meta tags. These tags will prevent the directory titles and descriptions from displaying for your site. Instead, your own HTML titles will display, along with snippets that are generated from your page content (or, sometimes, your Meta Descripton tag).