Our Favorite Spider Emulators for SEO

by Gradiva Couzin on March 6, 2015

Q:  In your book you recommended using a spider emulator to see what a webpage looks like to search engine robots. What tool do you use when you’re doing this?

A: These days, we use three different methods for reviewing a site with the “spiders-eye view.” Here they are in order of simplest to most complicated:

* Rex Swains HTTP viewer at http://www.rexswain.com/httpview.html.  Type in any URL and see what a typical client (browser or spider will see).  This is a great tool for a quick and easy look at a page.

* Fetch as Googlebot. This only works for sites that you have a Google Webmaster Tools account for (so it can’t be used for competitive review, only for your own site).  Sign up for Google Webmaster Tools by following the instructions here: https://www.google.com/webmasters/tools.  Once you have your website verified, click Crawl > Fetch as Google and enter the URL of the page that you want to see.  This method has the advantage that it is an accurate rendition of what Google sees, according to Google itself.

* Microsoft’s IIS SEO Toolkit. This is useful when you want to crawl an entire site, and this is what we use for our site technical audits. It is a free crawler, requiring a PC platform to run: http://www.iis.net/downloads/microsoft/search-engine-optimization-toolkit


We’ve been helping customers improve their search engine ranks and conversion rates for over a decade. We consult on SEO and social media for major brands, one-person shops, and everything in between. Get in touch to find out how we can help you!

YouTube generates more than a billion views on a daily basis. With such a wide audience, it’s no wonder companies want to make this video-sharing website part of their marketing strategy. Especially since it’s a great way to drive brand awareness and consumer engagement. But with the sheer volume of videos competing on the platform, how can you make sure yours shows up in YouTube search results? Here are a few overlooked techniques that could help get your video discovered.

Add Subtitles and Captions to Your Video
Since YouTube is a global platform, adding subtitles and captions allows you to extend your reach to audiences in other countries. By utilizing closed captions (CC), you can appeal to both foreign-language speakers and the hearing impaired. Plus your video will be more likely to appear in a YouTube search when a user filters by Subtitles/CC.

Upload Your Own Transcript
Along with titles, descriptions and tags, transcripts can also help with your video’s SEO. Sometimes YouTube will automatically transcribe your video’s content using speech recognition software. But this auto-generated text is often riddled with mistakes and inaccuracies. So we recommend creating and uploading your own transcripts whenever possible. Uploaded text is indexable by YouTube and used as another form of metadata for your video. This can be beneficial for your visibility in YouTube search results.

How Do You Create and Upload a Transcript?

If YouTube created an auto-generated transcript for your video, use that as a working draft. It may not be completely accurate, but it’s a good starting point. You can view YouTube’s transcript by visiting your video’s page and clicking “More.”

To create your own transcript:

  1. Copy the automatically generated transcript from your YouTube video page.
  2. Open Notepad (for PCs) or TextEdit (for Macs).
  3. Paste the copied contents from your YouTube video transcript into the document.
  4. Remove the time code and edit the transcript to match the actual audio. Use a blank line to indicate a long pause, a new sentence or the start of a new caption.
  5. Save the document in the proper format.
    • For PCs, click File then Save As. Select Text Document and choose UTF-8 encoding.
    • For Macs, click Format and select “Make Plain Text.” Click File, then Save and select Unicode (UTF-8).

For additional information, visit YouTube’s Transcript Help Page.

To upload your transcript to YouTube:

  1. Log into YouTube and visit Video Manager.
  2. Click the Edit dropdown button next to the video you’d like to add the transcript to.
  3. Click Subtitles and CC.
  4. Click the Add Subtitles or CC button and choose the language of the subtitles you’d like to create.
  5. Click Upload a File, navigate to the text file you created, and select it.
  6. Click Upload and the uploaded text will appear on the screen.
  7. Make any adjustments to the text as you see fit.
  8. Press Publish.

For additional questions, visit YouTube’s Add Subtitles and Closed Captions Help Page.

Adding captions and transcripts may seem simple, but these basic techniques could be just the boost your YouTube video needs.


We’ve been helping customers improve their search engine ranks and conversion rates for over a decade. We consult on SEO and social media for major brands, one-person shops, and everything in between. Get in touch to find out how we can help you!

Tumblr SEO Tips

If your marketing strategy includes Tumblr, you’re probably putting a lot of effort into content creation. But your hard work deserves to be seen by the biggest audience you can get, so don’t forget to think about discoverability and visibility! Here are some SEO and discoverability tips that we’ve compiled in the course of our […]

Continue reading…→

I Googled myself – and I don’t like what I saw! What should I do? [Updated]

If the results that come up on Google when you search for your own name are less-than-flattering, this article may provide the solution you’re looking for.

Continue reading…→

Pogo-Sticking, Bounce Rate – Ranking Factors?

Can Pogo-Sticking affect your rankings? Pogo-Sticking: A search engine user clicking through to a site, and going back to the search results to click through to other pages. If you Google “pogo-sticking seo” you get a page full of resulting saying it’s an SEO factor, and some go as far to say that if Google notices “they will […]

Continue reading…→

Wikipedia’s Disclosure Rule: The Impact on Marketers

In June 2014, Wikipedia made a big change that affects the crowdsourcing community by adding a new amendment to its terms of service. It requires anyone who receives compensation for editing Wikipedia articles to divulge this information. Gone are the days for business stakeholders and marketers making anonymous edits for the companies they represent. Instead, […]

Continue reading…→

Site Redesign SEO: Redirect Best Practices

Redirects are crucial in any redesign in which URLs change or old pages are retired. These redirects ensure a seamless user experience, and also help search engines to update their indexes from old URLs to new URLs and transfer SEO power from old to new pages. We’ve worked with many site redesigns; many that were seamless and […]

Continue reading…→

Five Signs Your SEO is Scamming You

Because we’re published SEO authors with some degree of credibility, we often hear from friends or acquaintances who are feeling uncertain about their SEO vendors.  “What are we getting for our money?” is a common question.  Sadly, the SEO industry has far too many charlatans in it. We’ve seen some painful examples of good companies getting burned […]

Continue reading…→

Ask the Experts: Should I use something other than Google Analytics?

Q: We are a medium-sized business and we’ve been using Google Analytics for a long time and never really questioned it. Should we switch to an analytics program other than Google Analytics?  A: The short answer is: probably not. Most businesses of your size that we’ve worked with get everything they need from Google Analytics. Even when an alternative […]

Continue reading…→

Site Speed and SEO

Q. How much does page load time affect my Google ranks? A. Ever since page load time was announced as a ranking factor back in 2010, the conventional wisdom has been that it’s not a major factor and isn’t something to worry too much about in terms of SEO rankings, unless a website is very, very slow, […]

Continue reading…→