Ask the Experts: How can I keep private materials out of Google?

by Gradiva Couzin on April 18, 2012

Q: We have some private materials on our site, but we’re not able to use robots.txt or a robots meta tag to disallow the pages from indexing.  As long as we don’t link to the pages, and only send them out as links in emails, will Google have any way of indexing these pages?

A: As we’ve said many times, the only way to keep materials truly private online is to password-protect them.  Even if you don’t link to your pages, here are a few ways that your private URLs might find their way into Google’s index:

  • People have reported seeing links from within Gmail messages spidered by Google, although this isn’t something that we have experimented with first-hand.  We do know that Google spiders Gmail content emails and uses “content extraction” in order to match advertisements, but there is no documentation of other uses of the spidered content.
  • Private URLs can be seen by Google in other odd ways.  For example, if somebody clicks from your private page to another website, then your private URL would show up as a referrer in the server logs.  Some server logs are public, or find their way into the public realm, and that could expose the private URL.
  • If someone visits the private URL while having the Google toolbar activated, then the URL could get collected and find its way into Google’s index.
  • If the link is included in a listserv email (seemingly private), then that could be scraped and republished on the web.
  • One of your authorized visitors could post a link within a forum post, on Facebook or Twitter, or somewhere else that he/she believes is private or semi-private, and that link could eventually be followed and indexed.

As you can see there are a lot of possible sources for leaks!

The best safeguard would be to password protect the individual pages, and your second-best approach is to deindex using the robots meta tag.  Without password protection, robots.txt or meta robots to prevent indexing, your next best line of defense is to watch for indexing and then do one of two things:

  • remove any files that have been indexed and put them in a different URL; or
  • place all of the private content in the same folder on your server, and then use Google Webmaster Tools and Bing Webmaster Tools to remove the files or folder from the indexes if/when it gets indexed.

These should give minimize your exposure in Google, but if your materials are truly confidential, you need password protection.  And without a doubt, ixnay on the social security umbers-nay!

 

{ 0 comments }

We’ve been helping customers improve their search engine ranks and conversion rates for over a decade. We consult on SEO and social media for major brands, one-person shops, and everything in between. Get in touch to find out how we can help you!

Google Semantic Sidebar ["Knowledge Graph"] Spied in Australian Results

by Jennifer Grappone on April 12, 2012

We recently noticed that Google Australia is showing much more information-rich search results for certain searches than we’ve ever seen. We’re not seeing this type of result in US searches. Could this be a peek at future, more widespread Google results?

Update: 5/16/12 Google has given the sidebar a name – Knowledge Graph — and has officially announced it in this YouTube video.  Universal roll-out is likely imminent at this time.

We don’t know how long this feature has been in testing mode, and not everyone on our staff was able to trigger these results. Here are some examples:

A search for “the walking dead” results in a detail-rich sidebar, with cast, episodes, and more.

Google Australia Results for "the walking dead"

 

A search for “apple fruit” results in a sidebar containing scientific details.

Google Australia search for "apple fruit"

(Admittedly, we’re giving the search engine a lot of context here. A search for just “apple” results in both Apple stores as well as some details about the fruit.)

A search for “Iron and Wine,” the name of a singer/songwriter, results in a list of songs.

Google Australia search for "iron and wine"

Why Is This Important?

We think it’s worth watching as a possible indicator of future semantic/information-rich results on Google.com and worldwide. It’s been reported in The Wall Street Journal that, over the next several months, Google will be adding more facts to prominent positions in search results in addition to standard links. Mashable discussed this evolution, too, describing a the development of a resource called Google’s “knowledge graph.”

This Isn’t New, Just Bigger

Google behaving as an “answer engine” is not new. Currently, for example, if you search Google.com for “what is mitosis” you will see a definition, provided by Google, above the regular search results.

Google US result for "what is mitosis"

Similarly, if you search for “men in black 3 release date” you will see Google’s best guess for the movie release date, based on several sources.

Google US search result for "men in black 3 release date"

But there’s a big difference between these definitions and the big semantic sidebar we’re seeing in Australian results. This is the first we’ve seen something so extensive.

What’s Driving These Semantic Sidebars?

The results are coming from a mix of sites, and the links within the sidebars go to various websites as well as Google search results. For example, if you click on a movie poster, you may go to a site such as freemovieposters.net, but if you click on the cast image, you may be taken to a Google search result for the actor name.

What Google does to gather this information is crawl websites and look for data that it can understand. This doesn’t always involve semantic tagging (such as schema.org microformatting) but semantic tagging would certainly make it easier for Google.

We tried to find a clear connection among the sites chosen for inclusion in the sidebar, but so far we haven’t identified one. The image results in the sidebar are not the top ranking Google image results, and we see no evidence that the featured sites are collectively using semantic markup such as Schema microformatting.

What Does This Mean for Site Owners?

If what we’re seeing on Google.com.au is a harbinger of a more widespread Google search result, site owners may feel a pinch in their click-through-rates. We believe Google’s goal in adding more semantic answers in its results is primarily to keep searchers on Google; you might say that Google is eliminating the need for searchers to click away from Google.

Some site owners have expressed concern about this as a growing trend for Google to steer visitors to its own properties, and we share this concern. There’s not much we can do but roll with these changes. But now we may have a new featured spot to vie for: how to get sites featured in these semantic sidebars is likely to be a topic of much research and speculation.

UPDATE: Matt McGee of Small Business Search Marketing pointed out that Google was previously testing this change in 11/2011. You can read the story on Search Engine Land.  Thanks, Matt!

{ 2 comments }

We’ve been helping customers improve their search engine ranks and conversion rates for over a decade. We consult on SEO and social media for major brands, one-person shops, and everything in between. Get in touch to find out how we can help you!

Ask the Experts: How do Pinterest Price Banners Work?

Q: I have an e-commerce site and my products have recently started getting “Pinned” by people on Pinterest.  Sometimes a price shows up in the corner banner.  Why do these price banners sometimes happen?  Can I control them? A: Price banners in Pinterest can be a nice feature for product pages, but site owners have [...]

Continue reading…→

Google Dropped My Website! What Should I do? [UPDATED]

One of the most common questions we get is from people who think their website has disappeared from Google. If you’re in a similar situation, we have some ideas for you in this article.

Continue reading…→

Google’s Magic G-bounce (“Dwell”) Time: 7.5 minutes

Upon bouncing back to Google from a search result, you may see a notice allowing you to block that domain from future Google results.  Or you may see a notice encouraging you to +1 that domain.  It all depends on the amount of time you spent on the page. Google is well aware of what [...]

Continue reading…→

11 Places to Submit Your App For a Free Review

When you’re launching a new app, naturally you want everyone in the world to see it!  But many of the app review sites you’ll find in Google are spammy or require a payment for submittal.  Try this list for free app review submittals. Before you submit, do a little prep work.  You’ll need a promo [...]

Continue reading…→

Ask the Experts: What SEO To Require From My Developer?

Q: I just hired a developer for my small-budget site development project. Is there a Top 10 list of things I should insist on SEO-wise to be built into the site? A: As we all know by now, SEO consists of several different endeavors: Technical SEO: ensuring a search-friendly and social-friendly site from a structural/technical [...]

Continue reading…→

Ask the Experts: Recovering from Google’s Panda Update

Q: I have an ecommerce site containing a useful library of product information that the manufacturer allows me to use on my site.  My site was hammered in Google’s Panda update, and I think it was because of this “duplicate” content.  What should I do now?  I don’t want to remove this library because it [...]

Continue reading…→

Canonicalization for Pagination – Roundup of SEO Wisdom [UPDATED]

Tongue twister, anyone?  Pagination canonicalization is one of the trickier challenges for the technical SEO set today.  We’ve been closely monitoring the hints and tips that occasionally drop from our favorite Googlers’ lips (sure, we all adore Matt Cutts, but let’s save a little love for JohnMu and Maile Ohye, shall we?).  Here’s a roundup [...]

Continue reading…→

Ask the Experts: Semantic Tags for Rich Snippets

Q: I’m confused. What are Schema.org tags and are these the same as semantic tags for rich snippets? Which ones should I use? A:  Have you ever wondered how some websites’ listings in Google and Bing get those great-looking review stars, price ranges, and other enhancements, like the ones shown here? This is all achieved [...]

Continue reading…→