Q: Hello! I am a consultant in web analytics, and I’m looking for a tool that simply counts how many pages the website has, so I can calculate the inclusion ratio, or the percentage of pages indexed in the search engine. It seems like such a basic thing, yet I’ve been literally searching for hours and can’t find what I’m looking for. Can you help?
A: What a great question. Most people just do a site: search in Google to find the number of pages on their website (for example, you could search for <site:www.yourseoplan.com> and look at the number of pages returned in Google – currently 282 pages). But if you want to find out what portion of your total pages have been indexed by Google, you’ll need another way of counting the total number of pages on your site so you can compare it against Google’s index count.
We suggest using XML sitemaps along with Google Webmaster Tools in order to calculate the inclusion ratio of pages indexed in Google. If you cannot create an XML sitemap internally, you may want to look into using one of the sitemap tools that are available, below are some resources:
- For a small site under 500 pages, you could use the online version at http://www.xml-sitemaps.com/ .
- For larger sites, you can pay for the standalone software available on that site, or review other sitemap generators at: http://code.google.com/sm_thirdparty.html
When you have a sitemap created, sign into Google Webmaster tools and follow the steps below (if you don’t have an account, create one!):
- When logged into Webmaster Tools, go to the Sitemaps section (Crawl > Sitemaps).
- From here, add the sitemap that you created for your website.
- If all goes well, Google will show the URLs you have submitted (see below):
- Google does not show Indexed count yet, but give it a day (or so) and come back and Google will now give you the pages it has included in the index.
With the Submitted and Index counts, you can then calculate an inclusion ratio. See image below:
For larger sites, you may want to break up sitemaps by category to provide better diagnosis for any potential indexing problems your site may have. These sitemap categories can then be included into a sitemap index file, and you can submit just that sitemap index file to Google Webmaster Tools. More information on a sitemap index file can be found here.
When you submit a sitemap index file to Google Webmaster Tools, you’ll get an overall number of Submitted and Indexed pages, and you can click down to the individual sitemaps to get a sense of the category index rates.