Announcing Keyword Explorer: Moz's New Keyword Research Tool

Posted by randfish

A year ago, in April of 2015, I pitched a project internally at Moz to design and launch a keyword research tool, one of the few areas of SEO we've never comprehensively tried to serve. The pitch took effort and cajoling (the actual, internal pitch deck is available here), but eventually received approval, with one big challenge... We had to do it with a team already dedicated to the maintenance and development of our rankings collections and research tools. This project wouldn't get additional staffing — we had to find a way to build it with only the spare bandwidth of this crew.

Sure, we didn't have the biggest team, or the ability to work on the project free from our other obligations, but we had grit. We had passion. We wanted to prove ourselves to our fellow Mozzers and to our customers. We had pride. And we desperately wanted to build something that wasn't just "good enough," but was truly great. Today, I think we've done that.

Meet our new keyword research tool, Keyword Explorer:

If you want to skip hearing about it and just try it out, head on over. You can run 2 free searches/day without even logging in, another 5 with a free community account, and if you're a Pro subscriber, you've already got access. For those who want to learn more, read on!

The 5 big, unique features of Keyword Explorer

Keyword Explorer (which we've taken to calling "KWE" for short) has lots of unique features, metrics, and functionality, but the biggest ones are pretty obvious and, we believe, highly useful:

1. KWE takes you all the way through the keyword research process — from discovering keyword ideas to getting metrics to building a list, filtering the keywords on it, and prioritizing which ones to target based on the numbers that matter.
2. KWE features metrics essential to the SEO process — two you're familiar with — Volume and Difficulty — and three that are less familiar: Opportunity, Importance, and Potential. Opportunity estimates the relative CTR of the organic web results on a SERP. Importance is a metric you can modify to indicate a keyword that's more or less critical to your campaign/project. And Potential is a combination of all the metrics built to help you prioritize a keyword list.
3. Our volume score is the first volume estimation metric we know of that goes beyond what AdWords reports. We do that using Russ Jones' volume bucket methodology and adding in anonymized clickstream data from ~1 million real searchers in the US. From there, Russ has built a model that predicts the search volume range a keyword is likely to have with ~95% accuracy.
4. Keyword suggestions inside KWE come from almost all the sources we saw SEOs accessing manually in their research processes — Keyword Planner data, Google Suggest, Related Searches, other keywords that the ranking pages also ranked for, topic-modeling ideas, and keywords found from our clickstream data. All of these are available in KWE's suggestions.
5. Import and export functionality are strongly supported. If you've already got a list of keywords and just want KWE's metrics, you can easily upload that to us and we'll fetch them for you. If you like the KWE process and metrics, but have more you want to do in Excel, we support easy, powerful, fast exports. KWE is built with power users in mind, so go ahead and take advantage of the tool's functionality however works best with your processes.

These five are only some of the time-saving, value-adding features in the tool, but they are, I think, enough to make it worthwhile to give Keyword Explorer a serious look.

A visual walkthrough

As an experiment, I've created a visual, slide-by-slide walkthrough of the tool. If you'd rather *see* vs. read the details, this format might be for you:

The Power of Moz's Keyword Explorer from Rand Fishkin

And, for those of you who prefer video, we made a short, 2 minute demo of the tool in that format, too:

Of course, there's a ton of nuance and complexity in a product like this, and given Moz's dedication to transparency, you can find all of that detail in the more thorough explanation below.

Keyword Explorer's metrics

KWE's metrics are among the biggest data-driven advances we've made here at Moz, and a ton of credit for that goes to Dr. Pete Meyers and Mr. Russ Jones. Together, these two have crafted something extraordinary — unique metrics that we've always needed for SEO-based keyword research, but never had before. Those include:

Keyword volume ranges

Nearly every keyword research tool available uses a single source for volume data: Google AdWords' Keyword Planner. We all know from studying it that the number AdWords provides is considerably off from reality, and last year, Moz's Russ Jones was able to quantify those discrepancies in his blog post: Keyword Planner's Dirty Secrets.

Since we know that Google's numbers don't actually have precision, but do indicate a bucket, we realized we could create ranges for volume and be significantly more accurate, more of the time. But, that's not all... We also have access to anonymized clickstream data here at Moz, purchased through a third-party (we do NOT collect or use any of our own user data via, for example, the MozBar), that we were able to employ in our new volume ranges.

Using sampling, trend data, and the number of searchers and searches for a given keyword from the clickstream, combined with AdWords' volume data, we produced a volume range that, in our research, showed ~95% accuracy with the true impression counts Google AdWords would report for a keyword whose ad showed during a full month.

We're pretty excited about this model and the data it produces, but we know it's not perfect yet. As our clickstream data grows, and our algorithm for volume improves, you should see more and more accurate ranges in the tool for a growing number of keywords. Today, we have volume data on ~500mm (half a billion) English-language search queries. But, you'll still see plenty of "no data" volume scores in the tool as we can access considerably more terms and phrases for keyword suggestions (more on suggestion sources below).

NOTE: KWE uses volume data modeled on the quantity of searches in the US for a given term/phrase (global English is usually 1.5-3X those numbers). Thus, while the tool can search any Google domain in any country, the volume numbers will always be for US-volume. In the future, we hope to add volume data for other geos as well.

An upgraded, more accurate Keyword Difficulty score

The old Keyword Difficulty tool was one of Moz's most popular (it's still around for another month or so, but will be retired soon in favor of Keyword Explorer). But, we knew it had a lot of flaws in its scoring system. For Keyword Explorer, we invested a lot of energy in upgrading the model. Dr. Pete, Dr. Matt Peters, myself, and Russ had 50+ reply email threads back and forth analyzing graphs, suggesting tweaks, and tuning the new score. Eventually, we came up with a Keyword Difficulty metric that:

• Has far more variation than the old model — you'll see way more scores in the 20s and 30s as well as the 80s and 90s than the prior model, which put almost every keyword between 50–80.
• Accounts for pages that haven't yet been assigned a PA score by using the DA of the domain.
• Employs a smarter, CTR-curve model to show when weaker pages are ranking higher and a page/site may not need as much link equity to rank.
• Adjusts for a few domains (like Blogspot and Wordpress) where DA is extremely high, but PA is often low and the inherited domain authority shouldn't pass on as much weight to difficulty.
• Concentrates on however many results appear on page 1, rather than the top 20 results.

This new scoring model matches better with my own intuition, and I think you'll find it vastly more useful than the old model.

As you can see from one of my lists above (for Haiku Deck, whose board I joined this year), the difficulty ranges are considerably higher than in the past, and more representative of how relatively hard it would be to rank in the organic results for each of the queries.

A true Click-Through Rate Opportunity score

When you look at Google's results, it's pretty clear that some keywords are worthy of pursuit in the organic web results, and some are not. To date, no keyword research tool we know of has attempted to accurately quantify that, but it's a huge part of determining the right terms and phrases to target.

Once we had access to clickstream data, we realized we could accurately estimate the percent of clicks on a given search result based on the SERP features that appeared. For example, a classic, "ten-blue-links" style search result had 100% of click traffic going to organic results. Put a block of 4 AdWords ads above it, though, and that dropped by ~15%. Add a knowledge graph to the right-hand side and another ~10% of clicks are drawn away.

It would be crazy to treat the prioritization of keywords with loads of SERP features and little CTR on the organic results the same as a keyword with few SERP features and tons of organic CTR, so we created a metric that accurately estimates Click-Through-Rate (CTR), called "Opportunity."

The search above for "Keanu" has an instant answer, knowledge graph, news results, and images (further down). Hence, its Opportunity Score is a measly 37/100, which means our model estimates ~37% of clicks go to the organic results.

But, this search, for "best free powerpoint software" is one of those rare times Google is showing nothing but the classic 10 blue links. Hence, its Opportunity Score is 100/100.

If you're prioritizing keywords to target, you need this data. Choosing keywords without it is like throwing darts with a blindfold on — someone's gonna get hurt.

Importance scores you can modify

We asked a lot of SEOs about their keyword research process early in the design phases of Keyword Explorer and discovered pretty fast that almost everyone does the same thing. We put keyword suggestions from various sources into Excel, get metrics for all of them, and then assign some type of numeric representation to each keyword based on our intuition about how important it is to this particular campaign, or how well it will convert, or how much we know our client/boss/team desperately wants to rank for it.

That self-created score was then used to help weight the final decision for prioritizing which terms and phrases to target first. It makes sense. You have knowledge about keywords both subjective and objective that should influence the process. But it needs to do so in a consistent, numeric fashion that flows with the weighting of prioritization.

Hence, we've created a toggle-able "Importance" score in Keyword Explorer:

After you add keywords to a list, you'll see the Importance score is, by default, set to 3/10. We chose this number to make it easy to increase a keyword's importance by 3X and easy to bring it down to 1/3rd. As you modify the importance value, overall Keyword Potential (below) will change, and you can re-sort your list based on the inputs you've given.

For example, in my list above, I set "free slideshow software" to 2/10, because I know it won't convert particularly well (the word "free" often does not). But, I also know that churches and religious organizations love Haiku Deck and find it hugely valuable, so I've bumped up the importance of "worship presentation software" to 9/10.

Keyword Potential

In order to prioritize keywords, you need a metric that combines all the others — volume, difficulty, opportunity, and importance — with a consistent, sensible algorithm that lets the best keywords rise to the top. In Keyword Explorer, that metric is "Potential."

Sorting by Potential shows me keywords that have lots of search volume, relatively low difficulty, relatively high CTR opportunity, and uses my custom importance score to push the best keywords to the top. When you build a list in Keyword Explorer, this metric is invaluable for sorting the wheat from the chaff and identifying the terms and phrases with the most promise.

Keyword research & the list building process

Keyword Explorer is built around the idea that, starting from a single keyword search, you can identify suggestions that match your campaign's goals and include them in your list until you've got a robust, comprehensive set of queries to target.

List building is easy — just select the keywords you like from the suggestions page and use the list selector in the top right corner (it scrolls down as you do) to add your chosen keywords to a list, or create a new list:

Once you've added keywords to a list, you can go to the lists page to see and compare your sets of keywords:

Each individual list will show you the distribution of metrics and data about the keywords in it via these helpful graphs:

The graphs show distributions of each metric, as well as a chart of SERP features to help illustrate which types of results are most common in the SERPs for the keywords on your list:

For example, you can see in my Rock & Grunge band keywords, there's a lot of news results, videos, tweets, and a few star reviews, but no maps/local results, shopping ads, or sitelinks, which makes sense. Keyword Explorer is using country-level, non-personalized, non-geo-biased results, and so some SERPs won't match perfectly to what you see in your local/logged-in results. In the future, we hope to enable even more granular location-based searches in the tool.

The lists themselves have a huge amount of flexibility. You can sort by any column, add, move, or delete in bulk, filter based on any metric, and export to CSV.

If your list gets stale, and you need to update the metrics and SERP features, it's just a single click to re-gather all the data for every keyword on your list. I was particularly impressed with that feature; to me it's one of the biggest time-savers in the application.

Keyword Explorer's unique database of search terms & phrases

No keyword research tool would be complete without a massive database of search terms and phrases, and Keyword Explorer has just that. We started with a raw index of over 2 billion English keywords, then whittled that down to the ~500 million highest-quality ones (we collapsed lots of odd suggestions we found via iterative crawls of AdWords, autosuggest, related searches, Wikipedia titles, topic modeling extractions, SERPscape — via our acquisition last year — and more) into those we felt relatively confident had real volume).

Keyword Explorer's suggestions corpus features six unique filters to get back ideas. We wanted to include all the types of keyword sources that SEOs normally have to visit many different tools to get, all in one place, to save time and frustration. You can see those filters at the top of the suggestions page:

The six filters are:

1. Include a Mix of Sources
   • This is the default filter and will mix together results from all the others, as well as ideas crawled from Google Suggest (autocomplete) and Google’s Related Searches.
2. Only Include Keywords With All of the Keyword Terms
   • This filter will show only suggestions that include all of the terms you’ve entered in the query. For example, if you entered “mustache wax” this filter would only show suggestions that contain both the word “mustache” and the word “wax.”
3. Exclude Your Query Terms to Get Broader Ideas
   • This filter will show only suggestions that do not include your query terms. For example, if you entered “mustache wax,” suggestions might include “facial grooming products” or “beard oil” but nothing with either “mustache” or “wax.”
4. Based on Closely Related Topics
   • This filter uses Moz’s topic modeling algorithm to extract terms and phrases we found on many web pages that also contained the query terms. For example, keywords like “hair gel” and “pomade” were found on many of the pages that had the words “mustache wax” and thus will appear in these suggestions.
5. Based on Broadly Related Topics and Synonyms
   • This filter expands upon the topic modeling system above to include synonyms and more broadly related keywords for a more iterative extraction process and a wider set of keyword suggestions. If “Closely Related Topics” suggestions are too far afield for what you’re seeking, this filter often provides better results.
6. Related to Keywords with Similar Results Pages
   • This filter looks at the pages that ranked highly for the query entered and then finds other search terms/phrases that also contained those pages. For example, many pages that ranked well for “mustache wax” also ranked well for searches like “beard care products” and “beard conditioner” and thus, those keywords would appear in this filter. We're big fans of SEMRush here at Moz, and this filter type shows suggestions very similar to what you'd find using their competitive dataset.

Some of my favorite, unique suggestions come from the "closely related topics" filter, which uses that topic modeling algorithm and process. Until now, extracting topically related keywords required using something like Alchemy API or Stanford's topic modeling software combined with a large content corpus, aka a royal pain in the butt. The KWE team, mostly thanks to Erin, built a suitably powerful English-language corpus, and you can see how well it works:

NOTE: Different filters will work better and worse on different types of keywords. For newly trending searches, topic modeling results are unlikely to be very good, and on longer tail searches, they're not great either. But for head-of-demand-curve and single word concepts, topic modeling often shows really creative lexical relationships you wouldn't find elsewhere.

SERPs Analysis

The final feature of Keyword Explorer I'll cover here (there are lots of cool nooks and crannies I've left for you to find on your own) is the SERPs Analysis. We've broadened the ability of our SERP data to include all the features that often show up in Google's results, so you'll see a page much more representative of what's actually in the keyword SERP:

Holy smack! There's only 3 — yes, THREE — organic results on page one for the query "Disneyland." The rest is sitelinks, tweets, a knowledge graph, news listings, images — it's madness. But, it's also well-represented in our SERPs Analysis. And, as you can see, the Opportunity score of "7" effectively represents just how little room there is for organic CTR.

Over time, we'll be adding and supporting even more features on this page, and trying to grab more of the metrics that matter, too (for example, after Twitter pulled their tweet counts, we had to remove those from the product and are working on a way to get them back).

Yes, you can buy KWE separately (or get it as part of Moz Pro)

Keyword Explorer is the first product in Moz Pro to be available sold separately. It's part of the efforts we've been making with tools like Moz Local, Followerwonk, and Moz Content to offer our software independently rather than forcing you to bundle if you're only using one piece.

If you're already a Moz Pro subscriber, you have access to Keyword Explorer right now! If you're not a subscriber and want to try it out, you can run a few free queries per day (without list building functionality though). And, if you want to use Keyword Explorer on its own, you can buy it for $600/year or $1,800/year depending on your use.

The best part of Keyword Explorer — we're going to build what you want

There's lots to like in the new Keyword Explorer, but we also know it's not complete. This is the first version, and it will certainly need upgrades and additions to reach its full potential. That's why, in my opinion, the best part of Keyword Explorer is that, for the next 3–6 months, the team that built this product is keeping a big part of their bandwidth open to do nothing but make feature additions and upgrades that YOU need.

It was pretty amazing to have the team's schedule for Q2 and Q3 of 2016 make the top priority "Keyword Explorer Upgrades & Iterations." And, in order to take advantage of that bandwidth, we'd love to hear from you. We have dozens (maybe hundreds) of ideas internally of what we want to add next, but your feedback will be a huge part of that. Let us know through the comments below, by tweeting at me, or by sending an email to Rand at Moz.com.

A final note: I want to say a massive thanks to the Keyword Explorer team, who volunteered to take on much more than they bargained for when they agreed to work with me :-) Our fearless, overtime-investing, never-complaining engineers — Evan, Kenny, David, Erin, Tony, Jason, and Jim. One of the best designers I've ever worked with — Christine. Our amazingly on-top-of-everything product manager — Kiki. Our superhero-of-an-engineering-manager — Shawn. Our bug-catching SDETs — Uma and Gary. Our product marketing liaison — Brittani. And Russ & Dr. Pete, who helped with so many aspects of the product, metrics, and flow. You folks all took time away from your other projects and responsibilities to make this product a reality. Thank you.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!

from Moz Blog http://ift.tt/1ri6V9r