Exactly how hard is it to rank for a particular keyword?
Answering this question is a key component of good keyword research, and software companies like Moz have developed "keyword difficulty" metrics to answer it.
A good keyword difficulty metric will help you pick out terms you can win handily, maximizing your return on investment, while also avoiding terms which would cause you and your business to waste resources on unattainable rankings.
Unfortunately, determining the difficulty of a particular keyword is not a precise science.
We know generally what factors are responsible for ranking in Google, like how many links you have, so most keyword difficulty metrics attempt to measure how highly the current top 10 URLs for a particular keyword score on those known ranking factors. The higher the score on these ranking factors, the higher the keyword difficulty.
This standard methodology for determining keyword difficulty, also called keyword competitiveness, has been around for the better part of a decade, but it now faces a far more difficult foe: RankBrain.
RankBrain is Google’s machine-learning artificial intelligence system that was integrated into Google’s ranking algorithm to better determine search engine users’ intent.
Much has been attributed to RankBrain without warrant, but at least one thing we know is that RankBrain endeavors to answer queries by interpreting their meaning beyond the keywords used.
A simple example of this in practice (which was likely true before RankBrain, and certainly true afterwards) would be Google's understanding of the phrase "cab NYC." Google now has the capacity to understand this phrase in the context of all the synonyms and related terms like taxi, sedan, NY, New York and New York City. Google can "expand" a query to include these synonyms and thereby have a much larger set of pages to consider as relevant - more relevant terms means more relevant pages.
So, how does this exactly throw a wrench into keyword difficulty scores?
The Impact of RankBrain
Well, let's take a closer look by starting with an obviously long-tail phrase which hasn’t been targeted yet: "this keyword should have low difficulty despite RankBrain." According to our understanding of ranking factors and competition, this keyword should be easy to rank for.
This eight-word phrase had never been targeted, and now, the only page which targets it is on my personal blog. My site, at the time of writing, ranks #1 for this term — and rightfully so, as it is the only existing page on the web truly relevant to the query. It ranked for the term within hours of publication without any external links.
However, at the same time, popular tools like Ahrefs and Moz give this keyword a mid-range difficulty rating of 30 and 39 respectively. How can this be?
If you continue down the search results for this long phrase, you will see that Google has decided that the most relevant articles on the web to this query are on sites like Ahrefs.com, Moz.com and SearchEngineLand.com. RankBrain rightfully identified the articles on these sites as relevant to the topics of keyword difficulty, RankBrain, and keyword research.
In doing so, it lifted up more generic content into the top 10 from powerful sites, which makes the phrase appear more competitive than it really is. Authoritative sites ranking for a term due to RankBrain’s interpretation of a query, and the high content quality on those sites, can artificially inflate the keyword difficulty scores shown by tools like Moz or Ahrefs.
I assure you that had I instead written an article titled "cab NYC," it would have not jumped to #1, despite the fact that it has a lower keyword difficulty according to both Ahrefs and Moz.
Choosing Good Keywords
While the industry at large works to solve these confusions, there is one step you can take to increase the likelihood you will select good keywords.
Instead of relying on a single term, group them together with related terms and find an average. If, for example, you were to put "cab NYC," "taxi NYC," "new york taxi," and "new york cab" into a list in Moz's Keyword Explorer, you would find the average score is much higher than the long tail phrase I presented before.
Moreover, be wary of keyword difficulty scores for terms that have no exact matches in the search results (ie: the phrase does not appear in exact order in the titles or descriptions of the search results). This can often indicate that the term is related to popular phrases, but is otherwise unoptimized - giving you an opportunity which your competitors might miss.
In the meantime, your best bet is to dig into the search results yourself. Use tools to cut out terms and phrases which you know are too competitive, but take the time to hand check the remaining opportunities — you might just find that RankBrain has created some hidden gems.
