SEO Strategy
Why keyword research keeps producing the wrong content
The problem is rarely the research. The failure is in the handover, where everything specific about a query disappears and a person becomes a phrase.
Most keyword research is fine. The queries are real, the volumes are roughly right, the difficulty scores are as reliable as such things ever are.
Then the spreadsheet gets handed to someone who writes a brief, and the brief says: “Topic: content marketing strategy. Primary keyword: content marketing strategy. Volume: 8,100. Word count: 2,000.”
Everything useful has been thrown away. A person with a specific problem, at a specific point in a decision, has been compressed into a phrase — and the writer, who never saw the original data, produces exactly what the brief specified: a generic article about a broad subject, indistinguishable from the nine already ranking.
The research was not the problem. The translation was.
A keyword is not a topic
This is the error underneath most of the others.
“Project management software” is a phrase that people type. It is not a subject to write about. Treating it as one produces a page that tries to be generally about project management software, which means it is specifically about nothing, and there is no reason for it to outrank anything.
The queries around it are not variations on a theme. They are different people at different moments:
- project management software — someone building a mental map of a category
- project management software for construction — someone who has ruled out the generalists
- asana vs monday — someone down to two options
- how to migrate from trello to asana — someone who has already decided and is de-risking the switch
- asana pricing for 5 users — someone with a team size and a budget, about to buy
- 01 project management software Building a mental map of a category.
- 02 project management software for construction Has ruled out the generalists.
- 03 asana vs monday Down to two.
- 04 how to migrate from trello to asana Already decided. De-risking the switch.
- 05 asana pricing for 5 users Has a team size and a budget.
A tool will cluster all five as one topic, because it clusters on semantic similarity. Five different pages need to exist, aimed at five different moments, and the last two are worth far more than the first — while having a fraction of the volume.
Semantic similarity is not the same as the same question. Clustering tools optimise for the first. Your customers experience the second.
Volume is the least useful column
Not useless — least useful. It is the one everybody sorts by, and it deserves less authority than it gets, for three reasons.
The numbers are estimates, and the error is not evenly distributed. Tools are reasonably good at high-volume head terms and poor at exactly the specific, long-tail and newly emerging queries that tend to be most commercially valuable. Independent analyses have repeatedly found keyword planners over-reporting volume for the large majority of terms they cover.
“Zero volume” usually means “below the reporting threshold”. It rarely means nobody searches it. If your sales team answers a question on every call, that question has demand, whatever the tool says. The tool has never listened to one of your calls.
Volume is decoupled from value. A term with 20,000 monthly searches attracts people who are curious. A term with 200 attracts people who are choosing. If your product costs £40,000 a year, you do not need 20,000 visitors. You need eleven of the right ones — a calculation the cost per lead calculator will do properly for a specific deal size.
And the number itself has become a weaker proxy for traffic than it used to be, because a growing share of searches are answered on the results page — in AI overviews, featured snippets, and the various panels that surround the ten blue links. A high volume figure no longer implies a proportional number of clicks available to win.
The columns that actually predict outcomes, in the order I would weight them:
| Column | What it tells you | How much to trust it |
|---|---|---|
| Distance to decision | Whether this person is close to buying | High — and you have to judge it yourself |
| Commercial value | Whether you want the customer at the end of it | High — comes from the business, not the tool |
| What already ranks | Whether there is room for you at all | High — it is direct observation, not an estimate |
| Difficulty | Roughly how much authority the query demands | Moderate — a modelled proxy, useful for relative comparison |
| Search volume | Whether demand exists at all | Low as a decision signal, and worse for long-tail terms |
- The head term Broad category query. Everybody's first instinct.
- The qualified term Category plus the constraint that matters to your buyer.
- The objection The question your sales team answers on every call. Often reported as zero volume.
Use volume as a sanity check that demand exists at all. Then decide with the columns that actually predict outcomes.
The volume and difficulty numbers are modelled, not measured
Worth stating plainly, because it changes how much weight those two columns deserve. No keyword tool measures search volume by counting searches — no third party has access to a search engine’s query logs.
What every tool publishes is a model built from whatever proxy data it can get: a base figure derived from Google’s own advertising data, refined against clickstream panels and each vendor’s own corrections. That is why two tools routinely report two different numbers for the identical query. It is not a bug in one of them. It is two different models of the same unmeasured thing.
Two specific mechanics are worth knowing before trusting a figure. Google Keyword Planner, the common ancestor of most of those models, groups phrases with a similar meaning into one “bucket” and reports the volume for the whole bucket rather than a number per exact phrase. That is a mechanical reason a “zero volume” long-tail query is often not zero demand at all: it has been folded into a broader group and never surfaced on its own.
Averaging then destroys anything seasonal. Ahrefs’ own worked example in its guide to keyword research is “black friday,” where the reported average of 282,000 searches is assembled from around 50,000 in most months and 2.5 million in November 2024. The annual figure describes no month that actually exists.
Difficulty has a parallel weakness, and the mechanism explains exactly where it misleads. Ahrefs describes its own Keyword Difficulty score in one sentence: it pulls the top ten ranking pages for a query and counts how many websites link to each of them. It is an absolute score, built entirely from the backlink profile of whoever currently ranks — not a measurement of how hard the term actually is to win.
Ahrefs’ own documentation lists what that leaves out. It ignores a site’s own authority, giving a baseline for an average site rather than for yours. It does not weight a strong link differently from a weak one, which the company concedes makes the score “merely an estimation”. And it captures none of the SERP nuances — content freshness, an intent mismatch, a low-authority incumbent — that can make a high-difficulty term winnable and a low-difficulty one unwinnable.
Three situations are where this modelling is weakest, and they are not marginal cases — they are exactly the terms good keyword research is meant to find:
- New or emerging terms. No historical query pattern exists for a volume model to lean on, and a difficulty score built from whoever happens to rank first says more about who arrived early than about genuine competitiveness.
- Terms with volatile seasonality. The Black Friday case above, generalised to anything tied to a date, a season or an event.
- Terms where the SERP itself has changed meaning. A category redefined, a word that has picked up a new dominant sense, a query Google now answers in a different format than it did a year ago. Volume and difficulty scores update on a lag; meaning can move faster than the model tracking it.
None of this argues for ignoring the numbers. It argues for using them the way their own publishers describe them: a modelled proxy, useful for rough comparison, weakest exactly where a content plan’s more interesting decisions tend to sit.
The volume bet is getting weaker, not stronger
Even where the numbers hold up, there is a second reason to be careful about “publish more, against more of the list” as a strategy: the return on it appears to be falling, and there is now a data point for that beyond argument.
Whitespark’s Local Search Ranking Factors survey has run long enough, with the same 47-person expert panel, to show trend rather than a single snapshot.
Between the 2023 and 2026 editions, the factor labelled “volume of quality content on service pages” fell 40 places in the panel’s scoring of what predicts local pack placement, from 32nd to 72nd. Internal links from the Business Profile landing page fell 65 places, to 110th; linking out to authority sites fell 64 places, to 177th. Over the same period, Business Profile completeness and review and behavioural signals rose.
Two things are worth labelling honestly here, because the source is careful about them even where the industry quoting it isn’t. This is a panel of practitioners’ aggregated professional judgement, not a controlled measurement of what actually moves a ranking: 47 experienced people scoring a factor is real evidence, of a different kind than a causal test. And it describes local pack placement specifically, not organic rankings in general.
Read narrowly, it is defensible as exactly what it is — a considered, repeated expert opinion that the return on stacking service-page content has fallen at the same time the return on a complete, current Business Profile has risen.
Set alongside everything above it in this piece, the direction is consistent rather than contradictory. A keyword list invites a “more pages against more terms” instinct by its shape, because it is a list and lists invite completion. The full breakdown of what moved and why sits in this site’s local SEO statistics.
The marginal page written to catch one more term on the list is now working against odds that have fallen, on the judgement of the people watching closely. The hour it takes is very often better spent finishing the profile field the same panel says rose fastest, or building the one page that resolves an actual decision, rather than the seventh page that resolves none.
Cluster around decisions, not subjects
The reframe that fixes most of this: group queries by the decision the searcher is trying to make, not by what they are about.
“Asana vs monday”, “monday.com alternatives for agencies”, and “is monday worth it for a small team” are three different phrasings of one decision: should we choose this tool or that one? One page, built properly, resolves all three.
Meanwhile “what is project management software” and “project management software pricing” are superficially the same topic and are separated by the entire buying process. Two pages, and they should look nothing alike.
One decision, one page. That constraint does more for content quality than any word count.
It also produces a natural sequence. A decision cluster close to a purchase gets built first — less competition, faster ranking, and revenue you can point at when someone asks whether the programme is working. The broad exploratory clusters get built later, when there is something for them to lead to.
The cluster a tool builds and the cluster a decision needs are not the same
The decision-based clustering above is the principle. Here is what it looks like against a realistic keyword list, because “cluster by intent” is easy to agree with and easy to get wrong in practice, including with a capable clustering tool.
Most clustering tools use one of two methods, and vendors are open about what each one actually measures. Ahrefs’ own explanation of its “Parent Topic” clustering is that it takes the page currently ranking top for a query and groups the query under whichever keyword sends that page the most traffic. Dedicated clustering tools more often compare the top ten or hundred results for each query directly and group the ones with heavy overlap.
Both are legitimate starting points. Neither is a judgement about what the searcher has decided. Ahrefs’ own guide concedes as much in a worked example of its own, on chocolate cake recipes, where an inconsistent results page caused both methods to cluster two phrasings together even though the pages behind them averaged a Domain Rating of 74 on one side and 33 on the other.
Shared vocabulary and shared results pages correlate with shared intent. They are not the same thing. Here is a realistic list for a commercial cleaning company, and what happens to it twice.
| Query | What a clustering tool groups it as | What the searcher has actually decided | The page it needs |
|---|---|---|---|
| commercial cleaning services | “Commercial cleaning” | Exploring the category, no vendor chosen | An explainer: what it is, what it costs, how it works |
| office cleaning cost per square foot | “Commercial cleaning” | Building a budget before approaching anyone | Part of the explainer above |
| how often should offices be professionally cleaned | “Commercial cleaning” | Planning, still pre-vendor | Part of the explainer above, not a separate page |
| commercial cleaning companies near me | “Commercial cleaning” | Choosing between local vendors right now | A Business Profile and a location page, not an article |
| commercial cleaning company reviews | “Commercial cleaning” | Vetting a shortlist before committing | Genuine review presence, not owned content at all |
| switching commercial cleaning providers | “Commercial cleaning,” occasionally its own cluster | Already a customer of a competitor, dissatisfied | A displacement page addressed to that specific frustration |
| commercial cleaning contract template | “Commercial cleaning” | A procurement task, possibly not a prospect at all | Low priority unless it reliably attracts real buyers |
A string- or results-similarity tool sees one topic here, because every query shares vocabulary and several of them share a results page dominated by the same handful of generalist listicles. Built to that cluster, the output is a single, competent, 2,500-word “everything about commercial cleaning” page that is matched to none of the seven decisions actually sitting in the list.
Split by decision instead, and the list resolves into at least four things to build, only two or three of which are even articles. The near-me pair is not a content gap at all; it is a visibility problem a blog post cannot fix, however well it is written — it belongs to the work described in the complete local SEO guide, not to a content calendar.
Reading the cluster this way is the only version of “keyword research” that tells you a page is the wrong output before a brief gets written for it. That is worth more than most difficulty scores.
Read the results before writing the brief
The single step most processes skip, and the cheapest source of advantage available.
Before a brief is written, someone should search the target query and read what is there. Not the metrics for those pages — the pages themselves.
What you are looking for:
Format. Guide, comparison, product page, tool, forum thread, video? If nine of ten results are third-party listicles, a vendor page will not rank there. That is not a quality problem you can write your way out of.
Who publishes. Vendors, independent reviewers, communities? A SERP dominated by Reddit and independent blogs has a trust profile that commercial pages struggle to meet.
Depth and structure. What do all of them cover? That is the table stakes — the questions your page must answer to be considered a credible response at all.
The shared blind spot. What does every ranking page fail to do? There is nearly always something: an assumption they all make, a step they all skip, a reader they all ignore, an honest trade-off none of them will state. That gap is the only reason for your page to exist.
If you genuinely cannot find a gap, that is a useful finding. It means the query is well served and your effort belongs elsewhere. Deciding not to write a page is a legitimate output of keyword research, and almost no process is set up to produce it.
The step everyone skips
Between the research and the brief there is a conversation nobody schedules.
It takes an hour and it involves someone who knows the commercials — margins, deal sizes, which customers are worth having, which enquiries the sales team dreads. Cross that against the query clusters and the priority order changes almost every time. It is the same question the FOUND, TRUSTED, CHOSEN framework asks of every marketing decision: which layer of the problem is this actually solving?
Two things fall out of it that no tool will give you.
The high-volume cluster you should skip. Every category has one: a large, tempting, obviously relevant set of queries searched almost entirely by people who will never buy — students, competitors, job seekers, or a segment you deliberately do not serve. Ranking for it produces a beautiful traffic chart and no revenue.
The specific query with no measurable volume that closes deals. The objection, the compatibility question, the “does it work with X” that comes up on every call. It looks like nothing in a spreadsheet. It is often the most valuable page you will publish this year.
Ranking research and citation research are not the same research
Everything above assumes the destination is a ranking on a results page that a human then clicks. That assumption is now only half true, and the half that is changing does not reward the same things a keyword-and-difficulty process optimises for.
Start with what does not work, because it is the more surprising finding and it was tested directly rather than surveyed. Ahrefs ran a controlled before-and-after study on 1,885 pages that added JSON-LD schema markup between August 2025 and March 2026, all already receiving heavy AI Overview citation, against 4,000 matched control pages that made no change.
The result was a small but statistically significant decline of 4.6% in AI Overview citation rate, and changes of +2.4% for AI Mode and +2.2% for ChatGPT, both indistinguishable from noise.
The study’s own honest limitation is worth repeating rather than smoothing over: it tested only pages already cited heavily, so it says nothing about whether schema helps a page earn its first citation. Within that scope, though, adding structured data bought nothing.
Google’s own current documentation says the same thing more bluntly: “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary,” and “There’s also no special schema.org structured data that you need to add.” Schema still earns its place for the things it demonstrably does do — it is the AI-citation claim specifically that the evidence does not support.
Now the finding that matters more for keyword research specifically. Across roughly 863,000 keyword results pages and four million AI Overview citations, the share of citations going to a page that also ranked in the organic top ten for the same query fell from around 76% to 38% in eight months, July 2025 to March 2026.
Ahrefs links the shift to the Gemini 3-powered upgrade to AI Overviews in January 2026, and theorises — its own word — that query fan-out is playing a bigger role in which sources get picked, pulling citations from related results pages rather than the original query’s.
No local-query-specific version of this finding was identified in the sources reviewed on 9 September 2026; it is a general, web-wide measurement. It earns a place here anyway, because the size of the move in eight months is itself the relevant fact: whatever currently governs AI citation is not stable enough for last year’s advice to describe this year’s mechanism.
Set those two findings against a third. Whitespark’s 2026 panel scored a new dimension for the first time this year: AI search visibility. Its top five reads: presence on expert-curated “best of” lists, dedicated service pages, prominence on industry-relevant domains, quality unstructured citations, and authority review site presence.
Four of those five are about being mentioned favourably somewhere other than the business’s own website. The one exception is instructive rather than reassuring — it is dedicated service pages, a page that resolves one thing, which is precisely the output this piece has been arguing for. What does not appear anywhere near the top is the volume play: more pages, against more terms, because the list had more rows.
A query worth building a page for, in the traditional sense, is a different question from a query worth having a correct, citable answer to exist somewhere.
For a commercially competitive, genuinely ranking-contestable term, the process in this piece still applies unchanged. For a question increasingly resolved inside an AI Overview or an assistant’s own answer, the better use of the same hour may not be a fourth page competing for a click that is happening less often. It may be making sure an honest answer to that exact question exists somewhere a model already draws from: a review, a directory profile, an industry list, a named mention.
That is a genuinely different deliverable from a keyword cluster and a brief, and most keyword-research processes are not currently set up to tell the two apart, because they were built for a results page that gave every answer a click. It is a sharper version of an argument this site has made before — search intent is a business question, not a keyword attribute — and increasingly, so is the choice between researching for a ranking and researching for a citation.
A spreadsheet that looked complete, and what it still got wrong
A realistic version of the failure this piece is about, walked through step by step, because it is more persuasive shown than described.
The brief: a marketing manager at a growing B2B services firm needs a page to catch demand around employee onboarding software. She runs it through a keyword tool and gets back a spreadsheet that looks, by any conventional standard, complete — a head term at a respectable volume, a dozen related phrases, a difficulty score within reach, and a search-intent label reading “commercial” or “informational” against every row.
She sorts by volume, picks the top three related terms, and writes the brief: Topic: employee onboarding software. Primary keyword: employee onboarding software. Related terms: best onboarding software, onboarding software for small business, HR onboarding checklist. Word count: 2,200.
The page that gets written is a competent, thorough guide to employee onboarding software in general. It does not rank in the top twenty after three months, and the reason was visible in the spreadsheet the whole time, if the columns had been asked different questions.
The volume column hid two audiences behind one number. “HR onboarding checklist” carried real volume and got folded into the brief as a related term. It is searched almost entirely by HR practitioners at companies with no budget and no intention of buying software, because building a checklist manually is the entire point of that search. Included as related, it pulled the brief toward a broader, more generic piece than either of the other two terms needed.
The intent labels were too coarse to separate two different buyers. “Best onboarding software” and “onboarding software for small business” both carried a “commercial” label. The first is searched by people surveying the market broadly, some of them analysts and competitors rather than buyers. The second is searched by someone who has already ruled out enterprise tools on cost and wants a shortlist sized to them, a smaller and far more qualified group, worth a dedicated page rather than a paragraph inside a general guide.
Nobody read the results page. Had someone searched the head term before writing the brief, they would have found a page dominated by comparison and listicle formats from review sites, not vendor explainer pages. A generic guide was never going to displace that format, whatever its word count. That is not a fact a keyword tool reports. It is a fact a person notices by looking.
The fix was not more research. Every fact needed to avoid this outcome was already sitting in the spreadsheet; the tool had done its job.
What was missing was the same translation this piece has argued for throughout: someone converting a spreadsheet into a specification a writer could use, rather than a title and a word count. A brief built the way this site recommends would have carried the results-page finding, the audience split hiding behind the volume number, and the two different buyers sitting under one intent label — none of which a keyword tool was ever going to supply on its own.
What good research produces
Not a ranked list of keywords. A document a writer can use without a translation step. For each planned page:
- The exact query, unedited, in the words people actually type
- Who is asking it and what they have already decided
- What is already ranking, and specifically what those pages get wrong
- The angle — the thing this page will say that the others do not
- The questions it must answer to be a complete response
- What it is worth, and therefore how much effort it justifies
- What happens next for someone who reads it and is convinced
That is a specification for a piece of content that has a reason to exist. A keyword and a word count is a specification for something average, and average has never ranked well.
Most keyword research fails not in the research but in the handover. Everything specific about the searcher is known at the point the spreadsheet is built, and almost none of it survives the trip to the writer.
What survives is whatever the brief carries — so the next step from here is a content brief a writer can actually use, and the judgement it depends on is the subject of search intent as a business question.
Questions
- Is search volume still useful in keyword research?
- It is useful as a rough sense of scale and misleading as a decision criterion. Tool volumes are estimates, they are frequently wrong for long-tail and emerging terms, and a term with a tenth of the volume can be worth far more if the people searching it are close to buying. Use volume to sanity-check that demand exists, then decide with intent, difficulty and commercial value.
- What is the biggest keyword research mistake?
- Sorting by volume and difficulty and treating the top of that list as a plan. It produces content aimed at the largest audiences rather than the most valuable ones, and it strips out everything specific about who is searching and why — which is exactly the information a writer needs to produce something better than what already ranks.
- Should I target zero-volume keywords?
- Often, yes. Tools estimate poorly for specific, long-tail and newly emerging queries, and 'zero volume' frequently means 'below the tool's reporting threshold' rather than 'nobody searches this'. Questions your sales team answers on every call are real demand whatever the tool says, and they usually sit far closer to a purchase than the high-volume head term.
- How do I turn keyword research into a content plan?
- Group queries by the decision the searcher is trying to make, not by topical similarity. Each cluster becomes one page that resolves one decision. Then sequence by commercial value rather than by volume, and write a brief for each that carries the original query, who is asking it, what already ranks, and what those pages get wrong.
- How many keywords should one page target?
- One decision, which usually means one primary query plus the cluster of variations and follow-up questions that belong to the same decision. Counting keywords is the wrong frame — a page that comprehensively resolves one question will naturally rank for dozens of phrasings of it, while a page built to cover four unrelated terms tends to rank for none of them.
- How are search volume and keyword difficulty scores actually calculated?
- Search volume is modelled, not measured — no third-party tool has access to a search engine's query logs. Most tools pull a base figure from Google Keyword Planner, which buckets similar phrases into one number rather than reporting each exactly, and then refine it with their own methods, which is why two tools routinely report two different figures for the same query. Difficulty scores such as Ahrefs' KD are typically built from the backlink profile of whoever currently ranks in the top ten — a proxy for authority, not a measurement of how hard a term actually is to win, and one that says nothing about SERP nuances like an intent mismatch among the current results. Both metrics are weakest for new, seasonal and recently-shifted terms, which is exactly where a content plan's more interesting decisions tend to sit.
- How should keywords be clustered into pages?
- By the decision the searcher is trying to make, not by string similarity or even by which page currently ranks for several of them at once. Clustering tools group on shared vocabulary or shared results pages, and the vendors are open that both are proxies for intent rather than a measurement of it — queries can share a results page and still represent buyers at completely different stages. A useful test: if two queries would need different content to satisfy the person asking, they belong in different clusters, whatever a tool says. Local, near-me-style queries often belong in no content cluster at all — they need a Business Profile or a location page, not an article.
- Does keyword research for search rankings still work for AI search results?
- Partly, and it is worth treating as a separate question rather than assuming one keyword list serves both. A controlled study found adding schema markup produced no meaningful increase in AI citation for pages already receiving it, and Google's own documentation states there are no special technical requirements to appear in AI Overviews or AI Mode. Separately, the share of AI Overview citations going to pages that also rank organically in the top ten fell from 76% to 38% over an eight-month period through early 2026 — a general, web-wide shift, not a local-specific one, but a large one. For a genuinely ranking-competitive term, traditional keyword research still applies; for a question increasingly answered inside an AI response, the more useful output may be a correct, citable answer somewhere other than your own page, rather than another article built to rank.
- Why has publishing more content against the keyword list stopped working as well as it used to?
- Because the return on it appears to have fallen, according to the people who track it closely. Whitespark's 47-person expert panel, surveyed consistently since 2023, scored 'volume of quality content on service pages' 40 places lower in its 2026 local pack ranking factors than in 2023, alongside falls for internal linking and outbound authority links, while Business Profile completeness and review signals rose over the same period. That is a panel's aggregated professional judgement, not a controlled experiment, so treat it as a strong signal about where practitioners now see the return rather than proof of causation. Either way, it argues for finishing the Business Profile and building the one page that resolves an actual decision before writing a seventh page aimed at one more term on the list.
- What should I do with a 'near me' keyword that turns up in keyword research?
- Recognise it as a different kind of problem before briefing an article for it. 'Near me' and similar local-intent phrasings are usually a request to be found and chosen locally, right now — a job for a complete, well-categorised Google Business Profile and, where relevant, a genuinely differentiated location page, not for another blog post. Writing content aimed at a near-me query while the underlying local presence is thin is effort spent on the wrong layer of the problem. It is one of the clearest cases where good keyword research produces a decision other than 'write a page.'
