Why small sample sizes are not the problem you think they are

It's a natural reaction. Most people are used to thinking about dataglossary×DataData is raw, uninterpreted information collected and stored so it can be analysed, processed, or used to inform decisions. On its own it carries no meaning; context and interpretation are what make it useful.Open glossary term→ in terms of scale. Bigger numbers feel safer. More representative. Easier to defend in a room full of stakeholdersglossary×StakeholderA stakeholder is any individual or group with an interest in a product, project, or outcome, including internal teams and external parties.Open glossary term→. But that way of thinking comes from a different type of work. User researchservice×User ResearchUnderstand user behaviour, validate ideas, and make clearer product decisions with evidence you can act on.Open service→ at the discovery stage isn't trying to measure how many people experience something. It's trying to understand why it's happening at all.

The two kinds of question get conflated because both are called researchservice×User ResearchUnderstand user behaviour, validate ideas, and make clearer product decisions with evidence you can act on.Open service→. How many of our customers hit this, and why does anyone hit it at all, need completely different methods. Asking the second question with a sample sized for the first is expensive; asking the first with a sample sized for the second is misleading.

What tends to surprise people, especially if they haven't sat through many sessionsglossary×SessionA session is a single period of user interaction with a product, from entry to exit within a defined timeframe.Open glossary term→ themselves, is how quickly patternsglossary×PatternA pattern is a reusable solution to a design problem that recurs across products and contexts. It captures an approach already shown to work, so a team solves the problem once rather than every time it appears.Open glossary term→ start to appear. Not in a perfectly consistent way, not with users saying exactly the same thing, but in behaviourglossary×BehaviourBehaviour refers to how users interact with a system, including actions, patterns, and responses.Open glossary term→ that feels familiar almost immediately. A hesitation in the same place. A moment of uncertainty before committing. Going back to check something that should already be clear. Individually, those things are easy to dismiss. Together, they start to form a picture. And once you've seen that picture a few times, it's very hard to ignore.

Watching the sessionsglossary×SessionA session is a single period of user interaction with a product, from entry to exit within a defined timeframe.Open glossary term→ yourself is what makes this credible, and it is the part that gets delegated most often. A finding read in a summary is somebody else's claim. The same finding watched three times becomes something you know, and it survives challenge in a way a slide never does.

There's a reason the five-users idea has stuck around. Jakob Nielsen's thinking (that a small number of users is often enough to uncover the majority of usabilityglossary×UsabilityUsability is how easy and efficient it is for users to complete tasks within a product. It focuses on clarity, simplicity, and reducing effort so users can achieve their goals without confusion or friction.Open glossary term→ issues) gets quoted a lot, sometimes too rigidly, but the reason it resonates is because it reflects what actually happens in practice. By the time you've seen the same frictionglossary×FrictionFriction refers to anything that slows users down or makes it harder for them to complete a task. It can be caused by poor design, unnecessary steps, unclear messaging, or technical issues.Open glossary term→ play out across a handful of people, you're no longer dealing with coincidence. You're seeing something structural, something about the way the journey is put together, the way information is presented, or the way decisions are being framed. And structural problems don't tend to affect just a few users. They show up wherever the same conditions exist.

The number is regularly quoted without its conditions, which is where it goes wrong. It applies to one coherent user group attempting the same task. Several distinct audiences need several small samples rather than one, and that is usually the real reason a study feels underpowered.

Where teams often get stuck is waiting for the signal to feel bigger. More sessionsglossary×SessionA session is a single period of user interaction with a product, from entry to exit within a defined timeframe.Open glossary term→, more users, more confirmation. As if the problem will somehow become more valid the more times it's observed. But in doing that, they end up circling the same insightglossary×InsightAn insight is a meaningful understanding that explains why something is happening and what it means.Open glossary term→ without actually moving anything forward. The learning doesn't deepen. It just repeats. That's usually where progress slows down, not because the team doesn't understand the issue, but because they're still looking for permission to act on it.

It is worth naming what would actually change with more sessionsglossary×SessionA session is a single period of user interaction with a product, from entry to exit within a defined timeframe.Open glossary term→. If the answer is nothing, the hesitation is about confidenceglossary×ConfidenceConfidence is the level of certainty in a decision or outcome based on available evidence.Open glossary term→ rather than evidence, and another round will not supply it. That is a decision-making problem, and it needs a different intervention.

There is a cost to the delay that rarely gets counted. Every additional round is weeks in which the problem continues to affect real users, and the researchservice×User ResearchUnderstand user behaviour, validate ideas, and make clearer product decisions with evidence you can act on.Open service→ budget spent confirming it is budget not spent fixing it.

Small samples don't give you distribution. They don't tell you how widespread something is or how it varies across different segments. That kind of understanding comes later, once changes are live and you're looking at real-world behaviourglossary×BehaviourBehaviour refers to how users interact with a system, including actions, patterns, and responses.Open glossary term→ at scale. But at the point of discoveryglossary×DiscoveryDiscovery is the phase of understanding problems, users, and opportunities before building solutions.Open glossary term→, that's not what you're trying to answer. What you're trying to understand is whether something is off, and why. And once you've seen enough to explain that clearly, more researchservice×User ResearchUnderstand user behaviour, validate ideas, and make clearer product decisions with evidence you can act on.Open service→ doesn't make the decision easier. It just delays the moment where something actually changes. The more useful question isn't is this enough users? It's have we seen enough to understand the problem?

Pairing the two is what makes both stronger. A small qualitative study tells you what is going wrong and why; the analytics you already have will tell you how many people reach that point. Neither needs to be large for the combination to support a decision.