Google Analytics has had a bots issue for sometime, and there are various bots that end up causing some site owners to second guess their data.

Some of these bots are easy to clean-up, almost two years ago now Google introduced a filter that can be applied to the views in an account to remove known bot traffic. If you haven’t checked that box, you should do it now — you can find it in your view settings, called “Bot Filtering”.

For most sites, this filter will likely fix a majority of bot traffic.

If you are still having issues, a great and very comprehensive resource can be found here:

http://help.analyticsedge.com/spam-filter/definitive-guide-to-removing-google-analytics-spam/

If after you spent time following AnalyticsEdge’s guide, and you still have issues, then the the following guide may help. 

The Humanoid Bots

The Problem

A little over a year ago, a bot network of sorts started affecting Google Analytics , and continues to affect sites (likely centering on eCommerce sites).

For a client we have, this bot problem is capable of sending 50,000-100,000+ sessions a month. It is all direct traffic.

The issue that makes this bot problem an issue is the scale of the traffic, and that none of the solutions above fix it. On top of the existing solutions not solving the issue: The bot traffic looks human-like, and continually sends different parameters. So, once you remove one, another pops up. To see this in action, view the following chart:

bots-ot

At the start a January, one bot was removed, then there was a new bot in town at the end of January. It was partially removed at the start of February, then a new one showed up. 

Identifying

This traffic has some similarities, at least in the data we’ve seen — They are generally on outdated versions of Chrome and Flash.

Spotting them can be easy, but they can blend in and get passed you for some time. For example:

chrome-versions

This traffic is just for one month. You can see see the issues with the top browser version:

  • It is outdated – Chrome 43.0.2357.130 – Almost a year since released.
  • The traffic has a high bounce rate.
  • The traffic has a high % of New Sessions.
  • Only a (very very small) fraction of the total site Transactions

After looking at this issue for some time, the only surefire secondary dimension we can use to remove this data  is a Flash Version (while retaining as much real-user data as possible).

Let’s take a look at the Flash Versions:

chrome-flash

We can see that there are only three non-bot user-groups here, and they have sent transactions. The rest (or a very high majority) of the traffic is from bots.  With the most coming from Chrome 43.0.2357.130 and flash version (not set).

Removing the Bot Traffic

Now, this traffic can be excluded using Advanced Segments. But, we don’t want that, we really just don’t want to see the data at all because of how much it is influencing the overall values. And, not everyone is going to look at an Advanced Segment, and sometimes, you may forgot and you’d be looking at incorrect data.

What we want is a way to exclude this traffic with a Filter.

Sometimes you can get away with just a simple exclude Filter based a single factor, either the Browser Version or Flash Version.

For sites with Revenue (which this is most likely affecting), you’ll need to make the call: Will this single exclusion filter remove too much Revenue?

Sometimes the answer is no, and you can add a filter like this  for flash version 11.5:

 

flash11.5

The likelihood of any real users being on flash version 11.5 is very (very) small.

If you are a bit weary of a blanket exclusion, then you’ll want to use multiple fields for your exclusion.

For this exclusion to work, it’ll require a two-step (multi-condition) filter:

Step 1 –  An Exclusion Filter, excluding the traffic you specify in filter #2

Step 2 – An Advanced Filter, specifying two Dimensions/Fields to filter.

Setting up the Exclusion Filter

Below is an image of how you would set-up the exclusion filter, it is pretty simple:

Set the Filter Field to “Custom Field 1” and type in a filter pattern (it can be anything as long as it matches with the second step)

custom-filter-exclude

Setting up the Advanced Filter

To setup your advanced filter, you will want to make a note of the Chrome version(s) and the Flash version(s) you want to block which will be added to the Filter.

The following image shows a setup specifying one Chrome version, and multiple flash versions.

advanced-filter

The key component of this filter is to output these two conditions to a Custom Field. In this case, we are naming this value “Exclude” in order to be used by the filter we created in the first step.

With these two filters setup, you’ll be able to remove the bot traffic, while retaining real-users (and more Transaction data for better quality data)

removebot-keep

You can now continually create new Advanced Filters to exclude this bot traffic as long as you continue to specify the same Custom Field defined in Step 1. And, if your site is like our clients’ you’ll be visiting this, at least, once a month.

Of Note: Keep an eye on your filter order. If your Advanced Filter is placed after your Exclusion Filter, it will not work. This is likely to happen as you’ll need to continually push up the order of your Exclusion filter as you create new Advanced Filters. 

Last Note: Always keep a View that is not filtered, so you can verify you’re not removing too much traffic (or in case you mess up and have to reference that raw data). 

I haven’t seen much information on using filters this way, so I hope you find this helpful, and if you have any questions or see any issues, please feel free to comment.

{ 0 comments }

We’ve been helping customers improve their search engine ranks and conversion rates for over a decade. We consult on SEO and social media for major brands, one-person shops, and everything in between. Get in touch to find out how we can help you!

Q: My company’s Knowledge Graph card includes our Google+ profile. How can we get our other social profiles to display as well?

A: The Knowledge Graph is a great way for users to learn more about your brand, such as your website address, logo, and company description. Below is a working example of Apple’s Knowledge Graph card:

Apple Knowledge Graph card

Since the Knowledge Graph takes up so much real estate in Google’s search results, you’ll want to maximize that space as much as possible. Getting your social profiles to display is a good way to start. Currently, Google only supports the following social profiles in the Knowledge Graph:

  • Facebook
  • Twitter
  • Google+
  • Instagram
  • YouTube
  • LinkedIn
  • Myspace
  • Pinterest
  • SoundCloud
  • Tumblr

So how do you get your company’s social profiles to appear? Here are a few options:

1. Add structured markup to your website specifying your social profiles.

According to Google’s documentation, you can use structured data markup to add your company’s social media profiles to the Knowledge Graph. It’s worth noting that if one of these social platforms has a verification process (i.e. Facebook and Twitter), Google will only display the verified profiles.

You can add markup to your official website by using schema.org vocabulary and JSON-LD, microdata, or RDFa markup. Here’s an example of what the JSON markup would look like:

sample json schema

Before adding markup to your site, test it with Google’s Structured Data Testing Tool. And make sure to place your markup on a page that isn’t blocked within your robots.txt file. Once Google crawls your site with the new markup, your social profiles will be eligible to appear in your company’s Knowledge Graph.

2. Request a Change to the Knowledge Graph as an “Official Representative.”

Recently, Google has begun letting official representatives of a website directly influence Google’s Knowledge Graph by suggesting edits and adding social profiles.

According to Google’s Help Page, to be considered an official representative, you need to meet the following requirements:

  • The Knowledge Graph card includes the topic’s official website, YouTube channel, or Google+ page.
  • You’re an owner of the online presence (the topic’s official website, YouTube channel, or Google+ page) that appears in the card.
  • You’re signed in to Google with the account you use as the owner of the topic’s online presence.
  • Your Web & App Activity is turned on.

To confirm that you’re an official representative, log into the appropriate Google account and search for something that brings up your company’s Knowledge Graph. If you see, “Is this info up-to-date?” above the Knowledge Graph card, that means you have the appropriate access.

is this up to date?

To begin adding social profiles, click “Suggest a Change” and then choose to give “general feedback.” Let Google know that you’d like to add an “official social profile.” Include the URL of your social profile and state that it can be found on your official website.

Once you submit feedback, Google will email you to let you know whether or not this addition has been approved.

{ 0 comments }

We’ve been helping customers improve their search engine ranks and conversion rates for over a decade. We consult on SEO and social media for major brands, one-person shops, and everything in between. Get in touch to find out how we can help you!

How to Get Your Local Business Listed on Google

Anyone who runs a business knows how important it is for customers to be able to find you online. That’s why getting your company listed on Google My Business should be your first step in creating your online presence. It enables your business information to be displayed on various Google properties, such as Search, Google+ […]

Continue reading…→

SEO Interview with Gradiva Couzin

Gravity partner Gradiva Couzin was delighted to be interviewed on SuperKaizen.com this week, answering questions including: How would you explain specifically what you do as an SEO? What is your primary marketing goal when it comes to delivering results? What do you find most rewarding about SEO? How do you stay updated with the latest […]

Continue reading…→

Benefits of Creating YouTube Playlists

There are numerous ways to optimize your company’s YouTube channel. One of the simplest methods is to create a playlist, which is a collection of videos that play in order, one right after the other. Playlists can help define your company’s brand and make it easier for viewers to navigate your channel. Additional benefits for […]

Continue reading…→

How to Link Your YouTube Channel to Your Website

You may have heard that there is no SEO silver bullet that will improve your ranks, and that’s true. Good SEO requires lots of little details that can add up to help your search engine ranks and visibility, and encourage engagement with your brand. One little detail you can get started on today is linking […]

Continue reading…→

How to Link Your Google+ Brand Page to Your Website

Even if you don’t put a lot of effort into updating your Google+ Brand page, it’s a good idea to link your Google+ page to your website. It won’t magically improve your Google ranks. But it helps strengthen Google’s understanding of your brand, which may improve your branded search results. It also helps you connect with your […]

Continue reading…→

What’s The Fastest Way to Remove Our Site From Google?

Q: One of our sites with private information got accidentally indexed in Google. We’ve deleted the site from the server entirely, but the listing is still showing up in Google. Help! How can I get our site out of Google quickly? A: There are two ways to remove your site from Google quickly: 1. Verify the site in Google […]

Continue reading…→

Should I Pay for a Press Release Distribution Service?

Q: We’re thinking about putting out some press releases for our B2B using a wire service. What is your view from an SEO perspective on this approach? A: Press release distribution has become much less effective as an SEO tactic in recent years, but in some cases press releases can still be helpful for SEO. Google […]

Continue reading…→

Our Favorite Spider Emulators for SEO

Q:  In your book you recommended using a spider emulator to see what a webpage looks like to search engine robots. What tool do you use when you’re doing this? A: These days, we use three different methods for reviewing a site with the “spiders-eye view.” Here they are in order of simplest to most complicated: […]

Continue reading…→