One of the more common emails we get is from people who think their website has disappeared from Google. If you’re in a similar situation, we have some ideas for you in this article.

If your previously high-ranking website seems to have fallen out of Google altogether, you are probably teetering somewhere between panic and desperation. As friendly consultants who care about your mental health, we want you to know this first:

Don’t panic! You probably didn’t get banned.
Give this a few hours and see if it works itself out.

We say that because it’s the most common scenario that we’ve seen.  But if you can’t bear to wait around to see if the problem will resolve itself, read on.

The first thing you need to do is clarify the problem. Specifically, has your entire site been removed from Google, or has it just been knocked down in the rankings?  To figure this out, you can do a site search, by typing this into Google:

site: www.myurl.com

(replacing “myurl” with your own URL, of course).  If Google doesn’t display any pages from your site, it’s likely that your entire site has been dropped.  Another way to check is to take a unique string of text from your website, one that is unlikely to be on anyone else’s site, put it in quotes, and search for it on Google. If Google displays the page from your site that contains this text, then your site hasn’t been dropped from Google.

If your site is still in Google, but your ranks have taken a sudden and dramatic nosedive, skip down to the “What Happened to my Ranks?” section, below. If, after several strategic searches, you still can’t find your site on Google, then yes indeed, your site has been removed from the index, and it’s time to figure out why.

Unintentional Spamming?
It’s possible that your site was found guilty of spamming in Google’s eyes, and got banned. Websites can be banned for SEO spam techniques such as “white on white” text, blatant keyword stuffing, or tricky redirects. J.C. Penney’s website was famously banned temporarily for using a link-buying SEO technique.  It’s not likely that your website would be banned if you’ve never done anything to intentionally trick Google into thinking that your site deserves a higher rank than is appropriate.

If you’re not sure about your status as spammer, SEO expert Jill Whalen has a forum that discusses “SEO No-nos.” Read it to get some indication of whether your site was spamming inadvertently.

You’ll also want to check all your messages in Google Webmaster Tools.  There, Google will sometimes (but not always) alert website owners that their site has been penalized for spam.

Victim of Foul Play?
Whether you’re into conspiracy theories or not, you should entertain the idea that someone else did the spamming for you, causing your site’s disappearance from Google. We’ve seen it more than once: a site’s domain expires for a short time, and during that period, the pages are taken over by a domain squatter that displays not-so-nice content. Or, your site may have been hacked and stuffed with nuggets of spammy badness (this is especially common with outdated WordPress versions, so keep your WordPress up-to-date, yo!).

The fallout from one of these scenarios is something we describe in our case study of Providence Restaurant. (It’s worth noting, by the way, that even though that site temporarily displayed spam content, it suffered with terrible ranks, but wasn’t entirely dropped from Google’s index.)

Again, Google Webmaster Tools can help you diagnose this problem.  Check your messages to see if there have been any alerts describing malware or other problems on your site.

Another form of devious doings on the web is content scraping, which is when other websites steal content from your site and then republish it, often multiple times.  If this has happened to you, it is possible that Google banned your site along with the “bad guys” who copied you. Try doing some searches on other search engines, looking for strings of text from your website.  You just might find an unexpected match.

Redirects or Canonical Tags Gone Wrong?
We once had a client who inadvertently added  a canonical tag on every page of his site pointing to a different domain.  Google followed this instruction diligently and removed every page of his site from its index.  Yeowch!  The good news: once the errant canonical tag was removed, ranks and indexing were back where they belonged within a couple of weeks.

We have also seen examples of sites that redirect their pages to another domain using a 301 redirect, and then are surprised when their domain is dropped from Google.   A 301 redirect is like a permanent change-of-address for search engines.  If you redirect your entire domain to somewhere else, Google is not going to continue to display your domain in its search results.

Robots.txt or Robots Meta Tag Deindexing?
Have you double-checked that you didn’t accidentally deindex your entire site with the robots.txt file or robots meta tags?

Here’s the line you really don’t want to have in your robots.txt file:

User-agent: *
Disallow: /

And here is a tag that really shouldn’t be on every page of your site:

<meta name=”robots” content=”noindex”>

Go on… check it now.  We won’t judge you.

Server Outage?
Your server might have hiccupped just when Googlebot came to visit your site recently, but that shouldn’t cause your site to be dropped from the index.  We believe that Google will check a site at least a couple of times before dropping it.   But, if the site went down for an extended period (days, weeks), this could be an explanation.

Most likely, the bot will come back and rediscover your site, and you’ll find your site back where it belongs after that.  But how long will you have to wait?  It depends on how many links there are to your site and how often Googlebot visits it. While there’s no sure-fire way to trigger a visit from Googlebot, getting some new inbound links, refreshing your content a bit, and submitting to the free URL submittal page probably won’t hurt.  You can submit an XML Sitemap via Google Webmaster Tools, which has the nice fringe benefit of telling you how many of your URLs are indexed.

In the future, if your site has anticipated downtime, set a 503 server status so that search engines will know it’s just a temporary glitch.

Reinclusion Request
If you are certain your site has been penalized for spam or malware, you can submit to Google’s request for reinclusion page. Matt Cutts of Google has posted some information on how to file a reinclusion request. But before you file a request, be very, very sure that there’s nothing spammy on your site.

What Happened To My Ranks?
If Google hasn’t actually removed your site, but you just noticed a sudden and substantial drop in rank for your favorite keywords, there are several possible explanations:

• Google could have made an intentional change to its results or its algorithm.  Panda and Mayday are a couple examples of famous algorithm shifts that blew some sites out of the water.  SearchEngineLand.com is a good place to look into industry chatter about algorithm updates.  See also our post about recovering from Panda.

• Google could be experiencing some temporary weirdness that makes the results different from what you’re used to. As we mentioned at the top of this article, we’ve seen this so many times with so many clients that this is probably the most likely scenario. In this case, just check your ranks again in a day or so . You might once again see that good rank you were used to.

• Your competitors could have gotten their acts together. OK, if your rank dropped by a few hundred spots, it’s probably not your competitors, as that would mean a lot of competitors getting their acts together at the same time!  But if you’re just being outranked by legitimate competitive activity, then it’s time for you to work on your site. And this work includes optimizing for, and tracking, additional keywords.

• Is there something funky going on with your homepage, but not other pages?  We’ve seen sites that have suffered big rank drops because their homepage had been deindexed due to redirect or canonical issues, and no other page had as much power as the homepage, so no other page on the domain was making it into those previous ranking spots.  If you’re seeing only a portion of your site in the index, run through the diagnostic steps above to see if you can figure out why.
• Is your website new?  Sometimes, websites show up in search results briefly after being launched, but then seem to drop dramatically out of the rankings after a couple of days, and are stuck in the depths for months.  It’s a phenomenon that some have called the “Google Sandbox.”  How to get out the sandbox? Get some inbound links to your site, and be patient.

• Is it possible that something has dramatically changed on your website? Say, you intended to update a single page but accidentally changed the page titles on every page of your site?  Don’t forget to check your site and make sure everything is in order.
• Could you be seeing different results because your personalization has changed?  We’ve known people who thought their ranks were phenomenal – only to be brought down to earth when they realized that the ranks they were seeing were personalized just for them by Google.   Read more about Google search personalization.

Hopefully we’ve given you some ideas about how to track down the source of your site’s disappearance from Google. Regardless of what you find, it’s always a good idea to make sure you’re offering lots of great content that makes people want to link to your site, and do some link building. In this way, you’ll have other sources of traffic that makes a Google problem like this less devastating.

Q: We have tons of great videos on our website. How can we get those little video thumbnails that sometimes show up in Google search results?

A: When Google understands that there is a video presented on a web page, it will often display a thumbnail as part of the search result for that page, as shown here:

(The exact layout of your results may vary, since Google is constantly tweaking its video result display.)

For the best chance at getting video thumbnails for your pages, you should embed videos on your site in a way that Google can easily recognize AND send Google as many signals as possible indicating that you have a video on the page. Here are some tips that can help:

• Include each video on a separate URL
• Embed your video in Quicktime or a standard Flash video player.  Google can’t crawl through a complicated Flash file, so if you use Flash, so we suggest using the built-in Flash video component for FLV playback using a hard coded URL path to the FLV streaming source (it’s OK if this is an off-domain location).
• Create a Google Video Sitemap and submit it to Google Webmaster Tools: click here for Google’s video sitemap instructions.
• To add even more signals that Google can read, tag the videos using RDFa: click here for Google’s video RDFa instructions.
• Use Video OG tags.  These are intended for social media, but Google sees them too, and we’ve seen clear evidence that Google will recognize video information in OG tags. See more about how to use Facebook OG tags.
• It is OK to host your videos on streaming service such as Brightcove.  However, you will probably be on your own when it comes to generating a video sitemap or well-formatted OG tags.

Video SEO is changing fast! About two years ago, video thumbnails were dominated by YouTube and Metacafe.  These days, YouTube still has a dominant role, but other domains are presented among the featured thumbnails more and more.  Watch this blog for updates in this quick-moving space.

Local listings are more important than ever. Here’s how to get your business listed in the major local search engines.

Google

Google’s local search results can be seen at http://maps.google.com. You can submit a free listing to Google Places (formerly the Local Business Center)  by logging in, but first you’ll need a Google Account.

To claim an existing Google Places listing, navigate to your business page and click on “I own this business.”

Yahoo!

Yahoo!’s local search results are available at http://local.yahoo.com.  Local listings are free, but listing enhancements can be purchased for $9.95/month. You can submit your local listing here: http://listings.local.yahoo.com/ (you’ll need a Yahoo acount).

Bing

• View Bing Live Search Maps: http://www.bing.com/maps/
• Add a new listing at the Bing Local Listing Center (click on “Add a Listing” at the bottom).
• UK Businesses –  Local search is fed by 118 Information; add a listing here: http://www.bingforbusiness.com/
• To claim your business if it already has a Bing Local listing, navigate to your business listing and click on “Change your business listing.”

Ask

Ask’s local listings (available at http://www.ask.com/local) are partnered with Citysearch.com. See below for information on how to add your business to Citysearch.

Citysearch

If your business already has a page on Citysearch, navigate to your business’s page and click on the “Own this Business?” link to claim the business.  You can manage profile details for free, or add premium features for a fee.If your business does not have a page on Citysearch, go here to add a listing:

http://national.citysearch.com/profile/add_business.

(you will need to create an account on Citysearch, or log in using Facebook).

Yelp

Yelp is an online city guide made up of user-generated reviews. A listing here is most appropriate for consumer-oriented brick-and-mortar businesses, but could be used for any business.

• Read the Yelp business owners guide
• To add a listing, first perform a search for your business name & location. If no results come up, you will see a large, red “Add Business” button on the page. You will need to sign up for a free Yelp account.
• To claim your business listing, navigate to your business page and click on “Edit Business Info.”

YellowPages.com

Yellowpages.com does offer a free listing. Follow the instructions to add your listing here: http://listings.yellowpages.com/.

Infogroup/InfoUSA

Infogroup/InfoUSA has a database with millions of local businesses, and is used by many other search engines to fill out their listings.  You can submit your business at www.expressupdateusa.com.  You will need to be verified with a phone call at your business number to confirm.

Localeze

Similar to Infogroup/InfoUSA, Localeze lists local businesses in an extensive database.  Add or claim your business listing at www.localeze.com.

Many websites built with Ajax contain a wealth of content that is hidden from search engines. Google’s specification for Ajax indexing, promises to fix all that.  Here, we interview a site that recently implemented the specification.  How did it work out for them?

In a recent post, we researched several examples of sites using the Google Ajax indexing specification.  One that jumped out at us was Bookwhack.com (since rebranded to YourNextRead). YourNextRead.com is a website for socially sharing book reviews in a pleasant browsable environment. Says co-founder, Robert Boland: “YourNextRead combines simple one page navigation with multiple book reviews and user recommendations.”  We spoke with Robert about his team’s choice to make their Ajax-generated content indexable on Google.

Gravity: Why did you decide to implement the Google specification for Ajax indexing on YourNextRead.com?

YourNextRead: Since we were starting a new project, we had the freedom to use the latest technology. We decided to develop in GWT Ajax [Gravity's Note: "GWT" stands for Google Web Toolkit]. We looked into how to make YourNextRead crawlable and thought using Google’s new technique might give us an edge.

Gravity: Did you have an Ajax-based site previously that did not have the spec in place?  If so, have you seen a difference between the old & new site performance in search?

We noticed a great improvement after implementing this technique on YourNextRead. Beforehand, none of the Ajax content was visible to the searchbots – only the index page was being crawled.

Gravity: I notice that your site doesn’t include degraded content for Yahoo! or Bing.   Was this a conscious decision or do you plan to implement some other approach for Yahoo and Bing?

We have not included degraded content for Yahoo or Bing, as other development tasks have been prioritised at the moment. Google is the largest search engine and tends to lead the way on search engine standards. We believe/hope Yahoo and Bing will soon follow Google’s lead.

Gravity: Did you have any technical difficulties or challenges when implementing?  Anything in Google’s instructions that could be clearer?

We did find it difficult to implement the specification at first. In particular, it was hard to find examples or advice on the web as very few other sites are using this technique.

Generally Google’s documentation was clear, although it was tricky to understand how to practically implement the section on ‘Creating HTML Snapshots’. Many of the websites we found describing the new technique focussed on the HTMLUnit implementation. However, we are using a PHP backend to generate much of our content so we could not use this. There are some pseudo-code snippets in the documentation to help with server-side content generation, and it took a while to get the results we wanted.

Gravity: How do you feel about the outcome and the indexing you’ve gotten for your site on Google?

We are still in the process of refining the method and our keywords. However, the indexing of YourNextRead has improved enormously. Webmaster tools indicated there are now many queries linking to our site, whereas previously, there was only one.

You can see for yourself with the attached graph of crawl stats, taken from Webmaster tools:

Gravity: Is there anything you’d like to share with any other website owners who are considering the specification?

The GWT forum is a great place to ask questions as some of Google’s staff are looking out for questions on the new spec and are happy to help. [Gravity's note: See this thread on "Crawlable AJAX - jQuery Load" as an example.]

The ‘Fetch as Googlebot’ option in webmaster tools was invaluable for testing purposes. The spec would have been much more difficult to implement without this being available.

Gravity: Thanks for the interview!

We’re grateful that Robert was willing to share a few of his experiences with us and our readers!  In future posts, we hope to connect with more sites that are implementing this new specification.  If you have experience with Google’s Ajax indexing, we hope you’ll leave a comment sharing your thoughts!

Rich internet applications (RIA), using Ajax, Flash, or other methods, have always been a problem for search engines.  That’s because search bots don’t have the ability to access content interactively the way humans do. In 2009, Google introduced a proposal for indexing Ajax.  We’ve got clients using lots of Ajax, so naturally, we’ve been watching closely to see what sites are adopting Google’s specification, and how it’s working out for them.

If you need the background on Google’s Ajax crawling specification, here it is in a few bullets:

• In October of 2009, Google proposed a method for developing Ajax-based sites so that the content in those sites would be crawlable.
• Vanessa Fox provided some helpful interpretation on Search Engine Land: “Google May Be Crawling AJAX Now – How To Best Take Advantage Of It.“
• For those who are seriously considering implementation, you can read Google’s full specification.

Public Service Announcement: Because we have a few clients who are considering this specification, we reached out to Google and were pleasantly surprised that they were responsive to our questions regarding implementation. Google’s eagerness to encourage web-wide adoption of the specification means that they may be more open than you might expect in terms of communicating with early adopters. If you are seriously considering implementing Google’s Ajax crawling specification, there is no better time to reach out to Google directly with your technical questions.

Here are some examples of sites using Google’s Ajax indexing specification, and a look at their current indexing status in the Google search engine:

Holiday Inn

The site Holidayinn.com appears to have implemented Google’s Ajax crawling specification (we’ll call it Google-Ajax for short) to a limited extent.

An example of the Google-Ajax protocol can be seen on this page:

http://www.holidayinn.com/hotels/us/en/san-francisco/sfocc/hoteldetail

The Google-Ajax links can be seen by hovering over the links labeled: Rooms, Amenities, Dining and so on.  “Amenities” links to /hoteldetail#!Amenities , “Dining” links to /hoteldetail#!Dining, and so on.

In each of these URLs, the fully rendered javascript can be seen by replacing “#!” with “_escaped_fragment_=”, as seen here:
http://www.holidayinn.com/hotels/us/en/san-francisco/sfocc/hoteldetail?_escaped_fragment_=Amenities — this is the fully generated page that Google uses for indexing purposes.

You can see the “Dining” page in Google’s search results, here:

At this writing, Holiday Inn’s Google-Ajax pages have been live for weeks, but these pages are frustratingly hard to find in Google’s index.We do not know whether the incomplete indexing we’ve observed is because of a glitch in implementation, or if Google’s robot simply hasn’t gotten around to visiting all of these sub-pages.

Facebook

Facebook appears to have fully implemented Google-Ajax.  After some digging, we were able to find examples of Google-Ajax URLs indexed in Google, as seen here:

Facebook is an interesting case study, because in addition to implementing Google’s crawling specification, the site also has alternate content that is browsable without javascript (we call this “degraded” content). After several weeks of watching Google’s index, we are finding that the degraded pages are far more prevalent in Google’s index than the Google-Ajax pages. It’s redundant to have fully degraded content and Google-Ajax in place, so we wonder:  is Facebook planning to keep this redundancy for the sake of Yahoo! and Bing? (There is no official word yet on whether Yahoo! and Bing will support this specification.) Or is Facebook holding off on getting rid of its degraded content until it feels more confident about Google-Ajax? Or are other factors at play?

Google Itself

An excellent example of a successful implementation of the protocol can be found at Google’s GWT showcase site:

http://gwt.google.com/samples/Showcase/Showcase.html

Here are some pages within the showcase:

http://gwt.google.com/samples/Showcase/Showcase.html#!CwCheckBox

http://gwt.google.com/samples/Showcase/Showcase.html#!CwFileUpload

http://gwt.google.com/samples/Showcase/Showcase.html#!CwBasicPopup

A Google site search within the Showcase section shows 154 pages have been indexed from within the Showcase – including pages that are clearly within the Ajax experience:

It’s readily apparent from looking at Google’s cached versions of these pages that Google is indexing the content on each individual page within the Ajax interactive experience on the site. We can conclude that Google web search is indexing Google-Ajax pages just like any other webpage.  We would call this one a total success, which is hardly a surprise, since it’s Google’s own implementation.

It does not appear that Yahoo or Bing have indexed any of these pages.

Bookwhack.com

Bookwhack.com appears to have fully implemented Google’s Ajax Indexing Specification.  Unlike the Facebook implementation, this site relies fully on the Ajax crawling specification to generate crawlable text. We’ll be watching this site closely.

As of December 4, 2009, Google’s search results will never be the same. Seriously, they’ll never be the same, even from one computer to another.

In our previous post, we discussed how Google personalized search will affect your business. Now, we look at how a business can influence personalized results, and whether we think that’s a good idea.

Should Businesses Try to Influence Google Personalized Search Results?

Some blackhat SEOs will say “of course, businesses should game the system any way they can.” Others might say that trying to influence personalized search results in any way is unethical (but those people probably work for Google – which, last time we checked, is neither a non-profit organization nor a branch of government). Clearly, the techniques described above could be abused; for example, an unscrupulous SEO company could trick its clients into thinking their ranks are getting better and better, or into thinking the SEO company’s ranks are higher than they really are.

But working with the system is not necessarily abuse. We always advocate 100% ethical SEO, and we advise the following: go ahead and be smart about personalized results, but never use any tactics that you wouldn’t want your potential clients or customers to find out about. Your potential customers are the last people you want to alienate – and they don’t want to be tricked or manipulated.

Influencing Google Personalized Search Results

Here are a few ways that a company could influence personalized results:

• Create a promotion with an unique or unusual name, and build a page for it on your site, for example, “Bring on the PicklePal Pickles!”. Wait for your page to indexed in Google and check to see that you’re ranking #1 for this unusual phrase. (Don’t forget to opt out of personalized search before you check!)  (As we’ve said in our book, it’s easy to get a #1 Google ranking for a unique phrase. Now there’s a good reason to do so.) Then promote the Google search results page via Twitter, for example, “Free Pickles this Tuesday! Click on our link here:http://[point to the Google page, not your website.]]”
• If you’re absolutely certain you have very high ranks for a particular phrase, you could include links to your Google results for this phrase – rather than directly to your site – in email campaigns.
• Run print ads with a callout to a Google search rather than your website. (“Hey kids, Google ‘PicklePal Pickles Forever!’ to get your pickle fix!”)  Just be sure your site stays at the #1 spot for the phrase! You may have seen something like this on billboards for the movie 2012, which suggested searching for the term “2012.”
• Run an AdWords campaign that includes a call to action telling users to perform a Google search that returns your business’s website. Keep in mind that Google has strict editorial guidelines and this tactic may require some trial and error on your part.
• Seed forums and blog comments with a search call to action where relevant and appropriate. For example, in a forum thread about finding discount pickles, a business can post a reply saying: “We are currently running a discount pickle promotion at www.mysite.com – you can Google ‘discount PicklePal coupon’ [link to the Google search result for this term] and click on the link to see the coupon.”

Caution: We haven’t tried these tactics yet – and we may not recommend them to our clients. We’ll keep you posted as we embark on this new SEO journey.

As of December 4, 2009, Google’s search results will never be the same. Seriously, they’ll never be the same, even from one computer to another.

Google’s expanded personalized search is a game-changer in search engine design. It’s a big deal but it has gone largely under the radar. All Google searchers will now get customized results that are influenced by past searching and clicking behavior, bookmarks, and other factors. Any time a user clicks through from Google to a website, it gives that site a boost in that user’s future search results. Organic search results – previously considered an “objective” third-party viewpoint – now differ based on who is doing the searching.

Personalized search is activated for all Google searchers whether they are logged into a Google account or not, unless they opt out. We don’t expect many people to opt out.

Why isn’t this making more headlines? For one, we’ve all grown comfortable with seeing localized results (“bakeries near San Francisco, California”). Plus we’re all starting to get used to more and more personalization in advertising, particularly on Facebook screens (“37 year old woman in San Francisco California? Click here”). Enhanced personalization of search results is a natural evolution.

Personalized search will affect your business

The full ramifications of this change are still unknown, but here are our preliminary thoughts on what personalized search means to website owners:

• When you check Google, don’t assume that the ranks you see for your site are the same as what your customers are seeing. To turn off personalized search, opt out, or add the tag &pws=0 to the end of the Google page URL.

For example, if you search for “early reader comic books” in Google you may get a URL like this:

http://www.google.com/search?q=early+reader+comic+books

But you’ll get unpersonalized results with this URL:

http://www.google.com/search?q=early+reader+comic+books&pws=0

• Personalized search will increase the importance of search queries that occur early in the buying cycle. If people click on your site during the browsing, comparing, and information-gathering phase, you’re likely to get increased Google exposure later, when they’re ready to buy. Does your site offer a good destination for keywords containing “compare” and “review?” Do you know what people search for in the early stages of buying your product?
• The advent of more personalized search places a higher value on clickthrough rate (how many times a person clicks from Google search results to your site). Your clickthrough rate can be improved with compelling, well-written titles, URLs, and meta descriptions.
• Personalized search may increase the percentage of site visitors who are repeat visits. Does your website offer something useful for the second and third visit and beyond?
• Do you have a website with an easy-to-remember or easy-to-guess domain? Lots of people are probably typing your URL directly into the browser instead of finding you in Google.  This is very good for your site traffic in general, but now there is a down side to having an easy-to-remember domain name: if users are less likely to use Google as a navigation tool for your site, you  might be at a disadvantage in personalized search.
• It isn’t clear what percentage of Google search results are personalized. If personalization is very heavy, it may be more difficult for newly established websites to use SEO to gain market share from already-established business.

Can personalized search results be influenced? Certainly. Your website’s ranks in personalized search results will be improved by any method that encourages people to navigate to your site using a Google search.

In our next post, we’ll discuss ways to influence personalized results.

Dynamic content used to be a red flag for search engine friendly design, but times have changed. Search engines now include dynamically-generated pages in their indexes, but some particulars of dynamic pages can still be obstacles to getting indexed.

Whether it’s keeping in synch with inventory or updating a blog, more than likely if you’re a website owner you have some level of dynamic or CMS-managed content on your site (and if not, you should really be looking into it for your next redesign). Follow the guidelines here to avoid major pitfalls and ensure that your dynamic body of work is search engine friendly from head to toe.

Rule #1: Be sure that search engines can follow regular HTML links to all pages on your site.

Any website needs individually linkable URLs for all unique pages on the site.   This way every page can be bookmarked and deep linked by users, and indexed by search engines.  But dynamic websites have an additional concern: making sure the search engine robots can reach all of these pages.

For example, suppose you have a form on your website: you ask people to select their location from a pull-down, and then when people submit the form your website generates a page with content that is specifically written for that geographical area.  Search engine robots don’t fill out forms or select from pull-down menus, so there will be no way for them to get to that page.

This problem can be easily remedied by providing standard <a href> type HTML links that point to all of your dynamic pages. The easiest way to do this is to add these links to your site map.

Rule #2: Set up an XML site map if you can’t create regular HTML links to all of your pages, or if it appears that search engines are having trouble indexing your pages.

If you have a large (10K pages or more) dynamic site, or you don’t think that providing static HTML links is an option, you can use an XML site map to tell search engines the locations of all your pages.

Most website owners tell Google and Yahoo! about their site maps through the search engines’ respective webmaster tools (Links: Google Yahoo!). But if you’re an early adopter, you should look into the new system whereby a site map can be easily designated in the robots.txt file using sitemap autodiscovery. Ask.com, Google and Yahoo! currently support this feature. Cool!

Rule #3: If you must use dynamic URLs, keep them short and tidy

Another potential problem – and this is one that is subject to some debate – is with dynamic pages that have too many parameters in the URL.  Google itself in its webmaster guidelines states the following: “If you decide to use dynamic pages (i.e., the URL contains a “?” character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them few.”

Here are a few guidelines you should follow for your website parameters:

• Limit the number of parameters in the URL to a maximum of 2
• Use the parameter “?id=” only when in reference to a session id [this is no longer a problem.]
• Be sure that the URL functions if all dynamic items are removed
• Be sure your internal links are consistent – always link with parameters in the same order and format

Rule #4: Avoid dynamic-looking URLs if possible

Besides being second-class citizens of search, dynamic-looking URLs are also less attractive to your human visitors.  Most people prefer to see URLs that clearly communicate the content on the page.  Since reading the URL is one of the ways that people decide whether to click on a listing in search engines, you are much better off having a URL that looks like this:

http://www.yoursite.com/church-bells/discount/

rather than this:

http://www.yourseite.com/prod.php?id=23485&blt=234

We also think that static-looking, “human-readable” URLs are more likely to receive inbound links, because some people will be less inclined to link to pages with very long or complicated URLs.

Furthermore, keywords in a URL are a factor, admittedly not a huge one, in search engine ranking algorithms. Notice how, in the above example, the static URL contains the keywords “discount” and “church bells” while the dynamic URL does not.

There are many tools available that will re-create a dynamic site in static form.  There are also tools that will re-write your URLs, if you have too many parameters, to “look” like regular non-dynamic URLS.  We think these are both good options for dynamic Intrapromote has a helpful post on dynamic URL rewriting.

Rule #5: De-index stubs and search results

Have you heard of “website stubs?”  These are pages that are generated by dynamic sites but really have no independent content on them.  For example, if your website is a shopping cart for toys, there may be a page generated for the category “Age 7-12 Toys” but you may not actually have any products in this category.  Stub pages are very annoying to searchers, and search engines, by extension, would like to prevent them from displaying in their results.  So do us all a favor and either figure out a way to get rid of these pages, or exclude them from indexing using the robots.txt file or robots meta tag.

Search results from within your website is another type of page for which Google has stated a dislike: “Typically, web search results don’t add value to users, and since our core goal is to provide the best search results possible, we generally exclude search results from our web search index.” Here’s our advice: either make sure your search results pages add value for the searcher (perhaps by containing some unique content related to the searched term), or exclude them from indexing using the robots.txt file or robots meta tag.

Bonus Points: Handling duplicate content

While it’s not a problem that’s specific to dynamic sites, this rule is one that dynamic sites are more likely to break than static ones. If multiple pages on your site display materials that are identical or nearly identical, duplicates should be excluded from indexing using the robots.txt file or a robots meta tag.  Think of it this way: you don’t want all your duplicate pages competing with each other on the search engines.  Choose a favorite, and exclude the rest. [Editor's note: we no longer (2009) recommend de-indexing duplicate content. A better approach is to either redirect your duplicate pages to the primary page using a server-side, 301 redirect, or to set up a <link rel="canonical"> tag for any page that has been duplicated. A good explanation of best practices for handling duplicate content in 2009 can be found at Matt Cutts' Blog]

Dynamic content is usually timely and useful, which is why users love it, and the search engines want to list it. And now you know how to help your dynamic website reach its full search engine potential.

Q: In organic search, Google is coming up with my legal statement and privacy statement pages for certain unrelated terms. These two pages do not house any of the search terms in their copy. How can I get Google to see other more pertinent pages for these keywords instead of these two pages which have nothing to do with the terms?

A: This is a common problem, because so many websites point links from their footer or global navigation to their privacy & legal pages. Google and other search engines interpret this as meaning that these are very important pages on your site. There are a few ways to remedy this – review the list, and choose the method that’s best for you:

• Using the robots.txt file, you can exclude these pages from being listed. This option is the most extreme, and will also cause any previously accumulated search engine “power” (aka PageRank) to be lost for these pages.
• Add the following code to the links to these pages: “rel=”nofollow”" This code will cause the search engines not to follow the links, thus focusing power onto other pages of your site This is a good solution if you have links to undesirable landing pages in your global nav, and you don’t want to pass a lot of search engine power to these pages. Here is an example of what your link would look like: Visit my <a href=”http://www.example.com/legal” rel=”nofollow”>privacy</a> page.
• Change the pages to make them into decent landing pages. Perhaps adding navigation, links, or other information at the top of the pages would be an option?
• Create a Google XML Sitemap, which allows you to give the various pages on your site your own relative level of prioritization. (We have yet to see this work in practice.)

In addition to the above, be sure that your preferred landing pages are linked from the home page as well as the global navigation if possible. That will help inform the search engines of what pages are most important to you.

Googling ourselves. We’ve all done it – and privacy experts agree that it should be done on a regular basis. But if the results that come up on Google when you search for your own name are less-than-flattering, you may be scrambling for a solution.

These days, potential employers (and mothers-in-law) are checking Google results for your name rather than just reading your resume. Whether it’s outdated contact info, unprofessional forum postings from 10 years ago, or a photo of you in a wet g-string contest (and you’re aguy!) you may be in need of a personal Google make-over.  Here are a few tips to improve your personal presence on this important search engine:

• Set up a Google Profile. Google will sometimes display your profile picture and link in the search results for your name. Although the listing may not be prominent, it’s accompanied by a thumbnail photo, which goes a long way toward getting people to notice it! Even better, you have 100% control over your profile contents – how much does that rock? It’s very easy to set up a Google Profile. Just go to http://www.google.com/profiles and click on the big “create a profile” link.
• Regularly review Google’s ORM (online reputation management) tool, called “Me on the Web.”  Read Google’s June, 2011 Me on the Web announcement for more info.
• See something bad? Ask the owner of the page to update or remove it. Almost anything can be removed from a website – if the site owner is accessible and agreeable.  After all, most website owners want updated information just as badly as you do. If you can find contact information for the website that bears outdated or improper information about you, obtaining an update or removal may be as simple as asking for it.  Be sure that you are very specific about the page or pages that contain the problem, and clear about why you want it changed.   If removal is not an option, you might request that the site owner add text such as “This information was last updated March 1996.  ZappyCo cannot vouch for the current status of this information” or “This page is no longer maintained.”
• Crowd out the results. It’s likely that you won’t find anyone who’s willing or able to change the offending content for you. Items such as archived forums, news and media content, and postings by people that simply hate your guts are destined to stay out there indefinitely.  Your best bet will be to try to outrank them on the search engines. Strategies for outranking your competitors can – and do – fill many websites and books.  Some quick and easy ways to start are:
  • Start posting comments in blogs or forums using your full name.  Choose big, popular forums that are likely to have a good presence, and make sure that the pages are accessible without a login.  (And, before you get any big ideas, make sure they’re relevant comments, or else when you Google yourself you’ll be embarrassed by your own spam messages.)
  • If you have a business or personal website, you can make your name one of your top target keywords and proceed with an SEO plan like the one described in our book, Search Engine Optimization: An Hour a Day.
  • If you have the opportunity to build your own web page within the website of your school, employer, or any organization of which you are a member (many schools offer this to their students and staff), be sure to do so. A nicely written bio page would be a welcome result on a Google search for your name!
  • If you don’t have a website or bio page, build one on any of the zillions of free website services out there (you can experiment with Google’s own here: [Google Link]) and make a page that’s all about you! (Also see the “pay your way to the top” bullet below – Naymz offers a free profile page.)
  • Get active on Linkedin or another business networking site.
• Use your middle name. Often, problems with Googling yourself arise because there are other people sharing the same first and last name.  For example, there are two authors on Amazon.com named Jennifer Grappone! If you are plagued by this type of problem, you might wish to incorporate your middle name into business correspondence, add it to your personal signature on emails, use it in forum and blog postings, and be sure that it is included on any web content about you.  In the long run, this will help to distinguish you from all of those online doppelgangers.
• …or no name at all. Some content stays on the internet for a surprisingly long time. Use your full name as described in the previous bullet when you’re making professional postings, but consider using a “pen name” for dating profiles or other material that you may not want to show up in 10 years when a potential employer is Googling you. And go anonymous for some of those those rants that you can’t resist!
• If all else fails, consider paying your way to the top. There’s even a company designed specifically to help you with this: Naymz, at https://www.naymz.com. (special thanks to Andy Beal for this suggestion from his blog). The basic service allows you to create a free personal profile page on their domain. The premium service will sponsor ads listing your profile page in Google, MSN, and Yahoo! (Be warned, as reader Darin Newberry points out: “Naymz web site is for the personal use of individual members only and may not be used in connection with any commercial endeavors.”)
• If pictures are poisoning your reputation, read more about image search at our post: How To Improve Image Ranks

Follow these steps to improve your Google self-esteem.  It’s a lot easier than changing your name!