ajax Archives - Gravity Search Marketing

Google’s Ajax Indexing Specification: Developer Talk

June 16, 2010
by Gradiva Couzin

Many websites built with Ajax contain a wealth of content that is hidden from search engines. Google’s specification for Ajax indexing, promises to fix all that. Here, we interview a site that recently implemented the specification. How did it work out for them?

In a recent post, we researched several examples of sites using the Google Ajax indexing specification. One that jumped out at us was Bookwhack.com (since rebranded to YourNextRead). YourNextRead.com is a website for socially sharing book reviews in a pleasant browsable environment. Says co-founder, Robert Boland: “YourNextRead combines simple one page navigation with multiple book reviews and user recommendations.” We spoke with Robert about his team’s choice to make their Ajax-generated content indexable on Google.

Gravity: Why did you decide to implement the Google specification for Ajax indexing on YourNextRead.com?

YourNextRead: Since we were starting a new project, we had the freedom to use the latest technology. We decided to develop in GWT Ajax [Gravity’s Note: “GWT” stands for Google Web Toolkit]. We looked into how to make YourNextRead crawlable and thought using Google’s new technique might give us an edge.

Gravity: Did you have an Ajax-based site previously that did not have the spec in place? If so, have you seen a difference between the old & new site performance in search?

We noticed a great improvement after implementing this technique on YourNextRead. Beforehand, none of the Ajax content was visible to the searchbots – only the index page was being crawled.

Gravity: I notice that your site doesn’t include degraded content for Yahoo! or Bing. Was this a conscious decision or do you plan to implement some other approach for Yahoo and Bing?

We have not included degraded content for Yahoo or Bing, as other development tasks have been prioritised at the moment. Google is the largest search engine and tends to lead the way on search engine standards. We believe/hope 😀 Yahoo and Bing will soon follow Google’s lead.

Gravity: Did you have any technical difficulties or challenges when implementing? Anything in Google’s instructions that could be clearer?

We did find it difficult to implement the specification at first. In particular, it was hard to find examples or advice on the web as very few other sites are using this technique.

Generally Google’s documentation was clear, although it was tricky to understand how to practically implement the section on ‘Creating HTML Snapshots’. Many of the websites we found describing the new technique focussed on the HTMLUnit implementation. However, we are using a PHP backend to generate much of our content so we could not use this. There are some pseudo-code snippets in the documentation to help with server-side content generation, and it took a while to get the results we wanted.

Gravity: How do you feel about the outcome and the indexing you’ve gotten for your site on Google?

We are still in the process of refining the method and our keywords. However, the indexing of YourNextRead has improved enormously. Webmaster tools indicated there are now many queries linking to our site, whereas previously, there was only one.

You can see for yourself with the attached graph of crawl stats, taken from Webmaster tools:

Gravity: Is there anything you’d like to share with any other website owners who are considering the specification?

The GWT forum is a great place to ask questions as some of Google’s staff are looking out for questions on the new spec and are happy to help. [Gravity’s note: See this thread on “Crawlable AJAX – jQuery Load” as an example.]

The ‘Fetch as Googlebot’ option in webmaster tools was invaluable for testing purposes. The spec would have been much more difficult to implement without this being available.

Gravity: Thanks for the interview!

We’re grateful that Robert was willing to share a few of his experiences with us and our readers! In future posts, we hope to connect with more sites that are implementing this new specification. If you have experience with Google’s Ajax indexing, we hope you’ll leave a comment sharing your thoughts!

Uncategorized

Examples of Sites Using Google’s Ajax Indexing Specification

May 11, 2010
by Gradiva Couzin

Rich internet applications (RIA), using Ajax, Flash, or other methods, have always been a problem for search engines. That’s because search bots don’t have the ability to access content interactively the way humans do. In 2009, Google introduced a proposal for indexing Ajax. We’ve got clients using lots of Ajax, so naturally, we’ve been watching closely to see what sites are adopting Google’s specification, and how it’s working out for them.

If you need the background on Google’s Ajax crawling specification, here it is in a few bullets:

In October of 2009, Google proposed a method for developing Ajax-based sites so that the content in those sites would be crawlable.
Vanessa Fox provided some helpful interpretation on Search Engine Land: “Google May Be Crawling AJAX Now – How To Best Take Advantage Of It.“
For those who are seriously considering implementation, you can read Google’s full specification.

Public Service Announcement: Because we have a few clients who are considering this specification, we reached out to Google and were pleasantly surprised that they were responsive to our questions regarding implementation. Google’s eagerness to encourage web-wide adoption of the specification means that they may be more open than you might expect in terms of communicating with early adopters. If you are seriously considering implementing Google’s Ajax crawling specification, there is no better time to reach out to Google directly with your technical questions.

Here are some examples of sites using Google’s Ajax indexing specification, and a look at their current indexing status in the Google search engine:

Holiday Inn

The site Holidayinn.com appears to have implemented Google’s Ajax crawling specification (we’ll call it Google-Ajax for short) to a limited extent.

An example of the Google-Ajax protocol can be seen on this page:

http://www.holidayinn.com/hotels/us/en/san-francisco/sfocc/hoteldetail

The Google-Ajax links can be seen by hovering over the links labeled: Rooms, Amenities, Dining and so on. “Amenities” links to /hoteldetail#!Amenities , “Dining” links to /hoteldetail#!Dining, and so on.

In each of these URLs, the fully rendered javascript can be seen by replacing “#!” with “_escaped_fragment_=”, as seen here:
http://www.holidayinn.com/hotels/us/en/san-francisco/sfocc/hoteldetail?_escaped_fragment_=Amenities — this is the fully generated page that Google uses for indexing purposes.

You can see the “Dining” page in Google’s search results, here:

At this writing, Holiday Inn’s Google-Ajax pages have been live for weeks, but these pages are frustratingly hard to find in Google’s index.We do not know whether the incomplete indexing we’ve observed is because of a glitch in implementation, or if Google’s robot simply hasn’t gotten around to visiting all of these sub-pages.

Facebook

Facebook appears to have fully implemented Google-Ajax. After some digging, we were able to find examples of Google-Ajax URLs indexed in Google, as seen here:

Facebook is an interesting case study, because in addition to implementing Google’s crawling specification, the site also has alternate content that is browsable without javascript (we call this “degraded” content). After several weeks of watching Google’s index, we are finding that the degraded pages are far more prevalent in Google’s index than the Google-Ajax pages. It’s redundant to have fully degraded content and Google-Ajax in place, so we wonder: is Facebook planning to keep this redundancy for the sake of Yahoo! and Bing? (There is no official word yet on whether Yahoo! and Bing will support this specification.) Or is Facebook holding off on getting rid of its degraded content until it feels more confident about Google-Ajax? Or are other factors at play?

Google Itself

An excellent example of a successful implementation of the protocol can be found at Google’s GWT showcase site:

http://gwt.google.com/samples/Showcase/Showcase.html

Here are some pages within the showcase:

http://gwt.google.com/samples/Showcase/Showcase.html#!CwCheckBox
http://gwt.google.com/samples/Showcase/Showcase.html#!CwFileUpload
http://gwt.google.com/samples/Showcase/Showcase.html#!CwBasicPopup

A Google site search within the Showcase section shows 154 pages have been indexed from within the Showcase – including pages that are clearly within the Ajax experience:

It’s readily apparent from looking at Google’s cached versions of these pages that Google is indexing the content on each individual page within the Ajax interactive experience on the site. We can conclude that Google web search is indexing Google-Ajax pages just like any other webpage. We would call this one a total success, which is hardly a surprise, since it’s Google’s own implementation.

It does not appear that Yahoo or Bing have indexed any of these pages.

Bookwhack.com

Bookwhack.com appears to have fully implemented Google’s Ajax Indexing Specification. Unlike the Facebook implementation, this site relies fully on the Ajax crawling specification to generate crawlable text. We’ll be watching this site closely.