Q: Hello! I am a consultant in web analytics, and I’m looking for a tool that simply counts how many pages the website has, so I can calculate the inclusion ratio, or the percentage of pages indexed in the search engine. It seems like such a basic thing, yet I’ve been literally searching for hours and can’t find what I’m looking for. Can you help?

A: What a great question. Although we don’t know of a tool that is designed to do specifically what you’re describing, we suggest you try an XML Sitemap generator. These tools are designed to create, basically, a list of all the pages on a site, so you could get the total number as a fringe benefit.

For a small site under 500 pages, you could use the online version at http://www.xml-sitemaps.com/ . For larger sites, you can pay for the standalone software available on that site, or review other sitemap generators at: http://code.google.com/sm_thirdparty.html

Another option would be an application such as SiteCrawler, designed to download and crawl an entire website.

Be warned: any problems that Google is encountering while trying to index your website (such as pages accessible only through javascript form submittals) may also be encountered by these other software programs. Getting an accurate number using any of these tools will be almost impossible. You might be better off using a trend as your metric – rather than total percentage indexed, you can report on the (+) or (-) change in number of pages indexed over time.

A website redesign is a time for celebration! But if you recently redesigned your website without thinking about the effect on its search engine presence, you may be in for a rude awakening. Follow a few simple guidelines to be sure that your fabulous new site isn’t going incognito.

One question that we are asked over and over again is this: “I just redesigned my website.  How can I make sure that I don’t lose my search engine rankings?”

If you just launched a redesigned website, or you are about to go through a revamp, we’ve made this checklist for you.  Follow it, and your grand debut won’t become a search engine flop.

You Never Know What You’ve Got Until It’s Gone

How did visitors get to your old site? Knowing this will help you know where to focus your efforts when reclaiming lost traffic.  For a meaningful baseline, dig up this information about your old site:

• Conversions. Do you have any data on sales, leads, or other performance of your old website?  Make a note of it so that you can compare it against the new site.
• Search Engine Rankings. Did you track your old site’s search engine ranks? Is your new website targeting the same keywords? If so, you’ll want to keep a record of your old site’s ranks as a baseline. You can learn the best way to track search engine ranks in chapter 6 of our book, SEO: An Hour Day, and you can record your rankings on our downloadable SEO rank tracking worksheet.
• URL “Hot List.” What were the most visited pages on your old website? And, what were the most common entry pages (the pages that your audience come to first)? These pages will be your highest priorities for “cleanup time” tasks (we’ll explain below).
• Inbound Links. It’s important to know how many other sites are linking to your old website – especially if you’re changing URLs. While the previous tasks in this list require some recordkeeping before the new site is live, this is one you can do just as well after launch.  See our handy search shortcuts page to learn how to find out who’s linking to your site.

On the Day of Launch: Handle With Care

With key background info in hand, you’ll be in a good position to manage your site redesign with care. Here are some site launch best practices for a website redesign:

• Avoid Changing URLs. Keep as many of your old URLs as possible. Don’t change your domain name if you can avoid it!
• Page Redirects – Must happen concurrent with launch. In an ideal world, every URL from your old website would redirect to an appropriate page on your new website.  But we know that this can be hard to achieve in reality.  So try this on for size: using a server-side 301 setting (your IT people will know what this is) redirect the following pages, in order of priority:

1. Your Website Home Page (be sure you get all the variations – index.asp, index.php, or whatever you have)
2. Any Pages with Special Status (i.e,. Customer Support) That Make Them Important to your Business
3. Your Top Entry Pages
4. Your Top Most Popular Pages
5. Pages on the Path to Conversion (the pages that visitors often visited on their journey from entering the site to converting)
6. Any Other Pages on the Website.

Don’t delay this step – you need the redirects in place before the search engine robots come back and visit your site. And be sure, when setting up the redirect, that each page from your old site goes to a well-chosen landing page on your new site – not just the home page!

P.S.: Looking to go techie? Here’s a link to an excellent guide to redirects in all sorts of languages (PHP, .htaccess, Ruby on Rails and so on).

• Server Downtime. It probably goes without saying that server downtime is to be avoided; however, most reports we’ve read say that Google will come back again if a page fails once. Here are some tips if you’re moving your website to a different server.
• Internal Links. No website is immune to broken links – even your own. After a redesign be sure to run a link validator on your website to be sure that those internal links have all been updated properly. These are available online or using website development software (in Adobe Dreamweaver, for example, select Site > Check Links Sitewide).
• File Not Found Page. As a result of all the potential broken links listed above, your audience will probably see the “File Not Found” Page more often than usual after a site redesign.  Be sure it’s well written and explains the situation. (“To better serve our customers, we have redesigned our website” is good; “404 Error!  The URL you have typed is incorrect!” is not.)  Better yet, include links to those most-popular pages you figured out earlier.

After Launch

• Monitor 404 errors. After your new site launches, keep an eye on your website stats, so you can see if there are a large number of “File Not Found” errors showing up for a particular URL. You can also see a list of broken links at Google Webmaster Tools. Any pages with a large number of errors should recieve 301 redirects.
• Inbound Links. There may be dozens or hundreds of links all over the web linking to non-existent URLs on your website.  Each of these should receive a polite request for an update – but don’t hold your breath.  In our experience, less than half of these requests result in an update. If you’ve got analytics in place, you can review which links are sending the most traffic and pester in order of importance.
• Directory Listings. Really just a special case of “inbound links,” directory listings deserve a little special attention.  Take the time to submit whatever form you need to make sure that they are linking to the correct URLs.
• XML Sitemap. Today’s search engines are smart enough to follow a 301 redirect through to a new page and index it properly – with no loss in search engine presence.  How long will this take?  We’ve seen it take 3 months before an old site is entirely flushed.  Can you speed up the process?  Maybe. Some SEOs swear by XML Sitemaps for getting pages indexed quickly. We haven’t seen this in action, but it certainly wouldn’t hurt. Here is a helpful article on how to submit your sitemaps.

We know what you’re thinking: You worked so hard on your shiny new website, and just when you want to kick back and relax a little, we’ve created a substantial new pile of work for you. Is it worth it? Absolutely! You don’t want your redesign to turn away your most desirable visitors – people who are actively trying to come to your site! You could hand all this traffic over to your competitors, or you could identify your best sources of traffic, and take important steps to keep them coming!

With so many web analytics options out there, it’s easy to get lost. Here, we list some solutions you should know about, and point you in the right direction for more information.

Google Analytics

Google’s free offering offers a plethora of web analytics information for small businesses. We recommend it as a starting point for any do-it-yourselfer looking for a quick and inexpensiveway to understand the goings-on within their site.

• Go to Google Analytics
• Read more about it on Clickz.com: “Google Analytics: Rising Tide or Tsunami?”
• Read a May, 2006 review on e-Commerce Guide
• Read an Aug, 2006 review by Manoj Jasra comparing Google Analytics and VisiStats

WebTrends

WebTrends is one of the best-known names in web analytics, offering comprehensive tools that can be deployed as client-side (they call this “on-demand”) or server-side (they call this “software”) options. WebTrends is known for being high-end (ie, expensive) but has a variety of packages available.

• Go to the WebTrends website
• Read a 2004 review on PC Magazine
• Read WebTrends Small Business Reviewed on Atlas-Tuesday IT blog
• Read an infoworld review.
• Read Aug, 2005 info about various Web Stats Packages on webmasterworld forum.

Omniture

One of the biggest names in web analytics, Omniture offers a variety of comprehensive web analytics packages.

• Go to the Omniture website

Read a white paper comparing HBX and Omniture’s SIteCatalyst

• Read a product review on Infoworld
• Read a 2004 review on Phil Windley’s Technometria
• Look at all of their industry awards

Visual Sciences, Previously WebSideStory – Now Owned by Omniture

WebSideStory become Visual Sciences and was acquired by Omniture, however, many people are still using their legacy analytics systems.

• Read a white paper comparing HBX and Omniture’s SIteCatalyst
• Read an IT World Canada Review of HBX Web Analytics 2.5
• Read a June 2003 review of Hitbox Enterprise
• Look at all of their industry awards

ClickTracks

ClickTracks is known for offering a reasonably priced solution with many of the same features as the bigger names, in a more visually appealing format. They have a faithful following of satisfied customers. They offer both client-side and server-side options. ClickTracks offers a free version of their product.

• Go to the ClickTracks website
• Read a 2005 review on Mike’s Marketing Site
• Read a 2005 review on SearchEngineGuide.com
• Read about the free version of the product in a 2006 Searchday article
• Read Aug, 2005 info about various Web Stats Packages including ClickTracks on webmasterworld forum.

Coremetrics

Coremetrics offers only the hosted (client-side) web metrics solution, which can simplify the choices your organization must consider. They offer a special version of their system for each of these vertical markets: retail, financial, and travel services.

• Go to theCoremetricswebsite
• Read a product review on Infoworld
• Read a case study and interview on a biz that chose Coremetrics.

Unica, formerly NetTracker

Unica NetTracker offers client-side (they call it on-demand) and server-side (they call it software) solutions.

• Go to the Unica website
• Read “12 Cool Things You Can Do With Unica’s Netracker Insight”
• Read a customer review of a biz that chose NetTracker
• Read Aug, 2005 info about various Web Stats Packages including NetTracker on webmasterworld forum.

Performancing Metrics

Originally developed for bloggers, Performancing Metrics also offers a free version of their product.

• Go to the Performancing Metrics website

Measure Map

Another free web analytics tool designed specifically for blogs. (not signing up new customers as of 10/9/2006, but you can sign up to be informed when they begin offering accounts again)

• Go to the Measure Map website

AWStats

AWStats is a free logfile analyzer. If you pay a monthly hosting fee, it’s very possible that the statistics program that comes with your hosting is AWStats. This program requires a great deal of technical expertise to set up; it is not for the average website owner!

• Go to the AWStats website
• Read a review at Pay-Per-Click Universe
• Read a 2003 review which includes some technical details

Additional Resources

• For a comparison of several tracking solutions, see this 2005 article: Web Analytics Products
• A report that compares 73 web analytics services (prices $399/$549 as of 9/2006): www.idealobserver.com
• The Web Analytics Buyers Guide 2006 is $179 at http://www.e-consultancy.com
• Costly report by Forrester covers all the biggies. $995 at http://www.forrester.com/Research/Document/Excerpt/0,7211,36675,00.html

Q: I know from my Google Analytics that I get a large amount of traffic to my form page, but nobody is submitting the form. How can I create a form that people will submit?

A: First, congratulations are in order!   You have a good amount of traffic to your website; people are getting to your form page; and you have an excellent tracking program in place!  So before you get to work on improving your conversions, be sure to pat yourself on the back for getting this far.

Now you need to figure out why your audience isn’t taking the next step.  Ask yourself a few questions to see if you might be guilty of some common form errors we’ve observed:

• Does the form submittal work? Be sure to run a test submittal periodically to be sure.
• Are you asking too much? Your form should require only the minimum amount of information that you absolutely need. If an email address is sufficient, please don’t require a phone number, home address, and first-born child as well.
• Does your website establish trust? Filling out a form and submitting it is to some extent an act of faith. If your website lacks an identity or clear brand, is built from a generic template, or jumps between multiple domains or designs, it is unlikely to garner the necessary trust for a form submittal. Think of it this way: are you more likely to hand your phone number over to a stranger on the bus or a trusted acquaintance? We believe that blogs are a fabulous way of building trust and relationships – the more personal, the better.
• Have you explained the purpose of the form? Most importantly, let folks know exactly what will happen after they submit the form. Will they get a call? An email? Or access to a different part of the website? Make sure that there is a clear and (preferably) compelling reason for someone to move forward with a submittal.
• Does your website have a privacy policy? Make sure that site visitors know exactly what you will (and won’t) do with their submitted information.

Work on improving the above problems, and keep a close eye on your Google Analytics information, and we think you’ll see some excellent results.  And if none of the above is a problem for you, then it’s time for some A/B page design testing.  But that’s another article, another day.  Good luck!

Q: Our website averages about 5,000 hits per day. Then one day a few months ago it inexplicably jumped to around 15,000 hits, but it was just for that one day. They dropped right back down the next day. I was wondering, what could cause this? We didn’t have any special events or press releases that day. Nothing out of the ordinary happened in our company. Any ideas?

A: There are various reasons that traffic on a site can spike in the way you’ve described. As you mention, you would often expect to see that type of behavior when there is an unusual event, press release, or other online promotion that might cause more visitors to come to the site. These days, a sudden, short-lived increase in traffic might be a result of getting some notice on a social networking system such as digg, reddit or del.icio.us.

Another, less delightful possibility is that the site was repeatedly hit by some sort of robot. This could be a search engine robot or an undesirable visitor such as a scraper bot (bots that are coming to the site to steal its content) or other spam manifestation. And if the site is dynamic, sometimes it is possible for a bot to get into an infinite loop and bounce around the website for a while, which could throw off your statistics.

Your best bet at finding the answer to this mystery will be checking your log files or website analytics program. I would start by checking referrers on the day in question – this is a quick way to see if there’s a link in some prominent website that you were unaware of. If this doesn’t answer your question, then you might want to review the part of your stats where it tells you what type of browsers or spiders (usually called User-agents in analytics programs) were visiting. Look for unusually high numbers from certain user agents or any other striking activities here.

And the big take-away from all of this is that “hits” is often not the most meaningful metric for a website! You might want to use this experience to choose some more helpful key performance indicators, such as sales or leads.

Q: We’re huge fans of your blog, but then, we’re kinda nerdy gals.  Who is your audience, and what is the purpose of your blog as you see it?

Avinash: Are you calling me a nerd? Or maybe a nerd magnet?  I am quite ok with that (note to self: go to amazon to place an order for pocket protectors).

When I started the blog in my mind the audience was for the most part practitioners of “web analytics” and the senior management of websites / companies.

The web is an awesome living breathing organism and it sucks that analytics is still stuck in the stone ages. I am passionate about leveraging the web to get really close to our customers, to merge the worlds of qualitative and quantitative data, and to make it the most responsive lowest cost channel for anything that a company can imagine (ecommerce, lead generation, support, advocacy etc). My hope through the blog is to share that passion with more people and to broaden minds about what “web analytics” really is and how to leverage the web as a learning platform.

I am finding that in reality the audience is very diverse. Vendors, consultants, people interested in investing, product managers and a whole host of web marketing people. I am even more surprised at how much international audience there is for the blog, around 30%.

Q: We know, and you know, that analytics is fun and addictive, But so many website owners don’t actually look at their traffic analysis data – there’s something overwhelming about taking that first step.  What’s the best way to get your feet wet in analytics if you’re not an experienced marketer?

Avinash: My standard quote to a marketer is: “Without exception the web can help you earn a higher bonus.” There are two components of a bonus, usually, either you bring in more (acquisition) or you convert more (revenue). If you are a marketer start with either of those.

If you own the site start with the simplest possible report, Visitors by Referring URLs. If you are running campaigns then start by identifying which campaigns are bringing in the traffic. Really simple. Where is the traffic coming from?

Now marry that up with outcomes. Revenue by referring URLs or campaign codes etc.

In 10 minutes you now know where people are coming from and what and how much they are buying. Now look for surprises. I am positive that even with this simple data you’ll be surprised at what you find. Marry up the data with your spending. So you poured $500k into search campaigns and you got less traffic and revenue than you hoped for (or less than what you are getting from blogs). You are now in business.

Notice I did not say anything about page views or path analysis or average time on site or all those traditional metrics. IMHO those tend to not get marketers anywhere because like the rest of us those metrics really don’t “connect” and actions are hard to figure out.

Q: The pay-per-click ad services make it easy to measure the success of a paid campaign, and I think that’s partly why PPC has enjoyed such rapid growth.  But in trying to measure the success of organic SEO efforts, there’s still a lot of guesswork, expense, and a learning curve involved. Do you think Google Analytics will bring the same level of accountability to organic SEO that PPC efforts currently have?

Avinash: As an aside could I just confide that most people don’t measure PPC well. It is easier to show that you spend $10 and made $11. But most people don’t measure that $9 of that you would have made any ways just with SEO or that just the budget spent on PPC was $10 not including your salary and your agency fees and all that. True measurement of PPC effectiveness has yet to arrive at the scene, but it will soon as more accountability is demanded.

An important disclaimer first: I have absolutely no knowledge of any sort about Google Analytics’s plans or road map. {editors Note: This interview was in 2006. Things have changed since then! -gc} Any thing I say is pure speculation on my part.

Google has brought a lot of transparency and data APIs and standards, etc. to the world of PPC simply because of the amount of money spent there and the pressure on Google’s customers to show clean ROI. I think that kind of pressure will be on Google in the near future as folks increasingly spend more money on SEO and realize three months later nothing much came of it.

SEO measurement is much more complex. The success metrics are “outside” the site: page rank, page strength, results for company key phrases, your landing page quality score etc. Success metrics for PPC are in the site: conversion, purchases etc.

Google Analytics (GA) is a great way to measure website metrics (so PPC), as would be omniture or webtrends or clicktracks. Google has provided API’s now into adwords that any vendor can plug into.

For SEO the challenge that the data is outside your site is a tough one. Most of the data is with the search engines. I think Google, and others, will evolve to give us automated feeds of our key SEO metrics in order to bring more accountability to the SEO business. If they do that they might first roll it out via GA, but given the dynamics of the marketplace every web analytics vendor will have access to the data.

Q: We know you work for a large business (Intuit).  We’re curious if you think there are any analytics strategies that cross over into small businesses.  Do you have any web analytics advice that you can give to those mom-n-pop businesses out there that are hoping to improve the performance of their websites?

Avinash: The strategies or tips or advice that I mention on my blog or in my speeches might be valuable for any size of business. The strategies and approaches stay the same usually, it is just the scale of them changes. So for example a frequent advice is absolutely positively immerse yourself in segmentation. This would apply in either case. The 10/90 rule applies specially well for the small business, don’t put a lot of money into a tool to get very good world class data.

I did a very special post just for small businesses. It has a lot of tips and specific reports a small business can start with and then become more complex with time.

If I had to summarize it in a few words: Get a free tool like Google Analytics or Clicktracks, start measuring search engine traffic and referrers, look at top pages on the site that lead to conversion, measure site bounce rate (especially if you do PPC), do SEO and use the site overlay report. Viola! You are a million dollars richer!!

Q: It seems that everyone is looking for a “silver bullet:” one measurement they can look at that will indicate the overall success of their website. We’ve heard a lot of people talking about bounce rate as being this silver bullet.  Can you define bounce rate for our readers, and do you agree that this an important web metric?

Avinash: Ahh great timing, I just mentioned it above. Let us get one thing out of the way: There is no silver bullet. No matter if I say that or your grandma says that. It is a lie. The web and business on the web is simply too complex (and that is why I love it).

There are many definitions of site bounce rate but the one I have found to be most insightful is:

In % terms Site Bounce Rate = (Site visitors who stay on the site for less than 10 seconds) divided by (Total number of site visitors).

Site bounce rate is a great entry level metric. It is fantastic at identifying all sorts of, shall we say, “bummers”. If you compute it, as defined above, it is a great way to know how much site traffic is engaging with the site as a whole.

Where it becomes fun, remember I am a geek, is when you segment it out. So find out what you are spending money on and then compute site bounce rate for that. If you spending a lot on PPC compute site bounce rate for each campaign, very quickly you will find out which campaigns are sending you wrong traffic. Or you are a small business and you have paid xyz search engine $500 to list you in their directory, compute bounce rate for that traffic, see if that money was worth it.

So this is a great metric to start with. It tells you a lot and identifies some “bummers” very quickly. But then you have to get deeper into understanding the why it is happening and what you can do and how to fix it etc and you’ll graduate to other more complex metrics. By then Site Bounce Rate would have more than paid you back for the investment you made in computing it.

Q: Many business owners have goals that they want their website visitors to complete offline – things like recommending their business to other people, making a phone call, or walking into a store.  Have you worked on measuring these kinds of goals, and if so, what advice can you give to these folks?

Avinash: Multi-channel is a big challenge and everyone is making progress, not as much as one would like. Without speaking about my employer I’ll give out some general tips.

Phone is the easiest one. Simply use a unique 800 number on your website. Getting a ton of phone numbers is a very small amount of cost and it is a awesome way to know which calls are coming from the web. Now there are companies that will run your campaigns (PPC or radio or otherwise) that will put in a dynamic 800 number on your website and route the call from their “switch” to your company and in the process capturing the online to offline call. If you are small you can just do the former, use a unique number and then count.

Store is much harder.

If you own the retail store do online coupons that people can being to the store and you know they found you on the web. And it does not have to be a lot of money, I have seen a promotions for just one single pack of post-it notes if I brought this online printout thingy! Or I am sure you have seen many stores like Best Buy or Circuit City partner with BizRate. You’ll see a small survey code on the receipt in exchange for a $25 raffle (talk about cheap!) you go online and fill out a survey and tell them where you did research (online).

If you don’t own a store and you sell online and via stores this is exponentially harder, because of the “missing link” between you, website, and the selling channel, retail store. In this case user market research studies or online website surveys (that ask for purchase channel preference) have been used effectively to gauge the impact of the online channel on the offline channel.

Q: In one of Gradiva’s favorite posts on your blog, you talk about giving names to certain types of  website visitors (the abandoner, the flirt, and so on).  Why is it helpful to think in terms of “personas” instead of stats?  Do you have any other “nicknames” like these that you use in your analytics work?

Avinash: We often forget that human beings visit our websites and not “shopper_id’s” or “visitors” or “cookie values”.  The post was rooted in my hope to bring the human back into our minds. Humans interact with our website, just kooky ones like me and smart ones like you. And we are not the same.

I could write all day about personas but the main reason I love them is because they help us all step outside the “sanitized” world of numbers and think of our customers as people, atleast groups of people. Personas bring reality to our minds and then when you think of your website or analyze the numbers you’ll do it very differently. You’ll be solving for “Susan Simple” or “Tony Advanced” or “Avinash Nerd” etc.

I find that this specially works wonders with non-analysts, our management or marketers etc.

Q: How are you enjoying being a blogger?  Is it more work than you expected?  And do you blog “on the clock” as a part of your work efforts, or is this more of a hobby?

Avinash: Let me take that in the reverse order……

The blog is not a part of my work effort, it is a personal blog. I don’t want to call it a hobby (if it were I might have given it up after a week ). This is a bit corny: the blog is very much a labor of love.

I don’t have books or consulting services to sell. I am following the path of my “virtual mentor” Guy Kawasaki (he does not know I exist but I take a great deal of inspiration from his blog). His advice is “Eat like a bird, and poop like an elephant.” I am trying to live that Japanese quote. 

Blogging is much more work than I expected. I only post twice a week (my posts are usually quite long) and yet my wife Jennie’s calculation is that I am putting around 15 hours into it each week. That would include thinking and drafting and writing posts and commenting and replying to email and all that. 15 hours a week is not what I expected when I started (and it is a lot with a full time job and two beautiful little children and a wife and family and travel).

I am enjoying it very much I have to admit. I am humbled by the kind words people say on the blog and I am amazed at the reach of my little blog (I am surprised and thrilled to be ranked around 12k in Technorati) and I have met so many wonderful people (you for example) that I would never have met before. I think I enjoy it most of all because is a outlet for something I am deeply passionate about and I like to think I am adding some value in our little ecosystem.

I wanted to thank you both so much for the opportunity to do this interview, I had a lot of fun.

The pleasure was all ours!!