Why small sample sizes are not the problem you think they are

Five, maybe six, and suddenly the conversation shifts. The insightglossary×InsightAn insight is a meaningful understanding that explains why something is happening and what it means.Open glossary term→ hasn’t changed, but the confidenceglossary×ConfidenceConfidence is the level of certainty in a decision or outcome based on available evidence.Open glossary term→ in it starts to wobble slightly.

It’s a natural reaction. Most people are used to thinking about dataglossary×DataData is raw information collected and stored for analysis, processing, or decision-making.Open glossary term→ in terms of scale. Bigger numbers feel safer. More representative. Easier to defend in a room full of stakeholdersglossary×StakeholderA stakeholder is any individual or group with an interest in a product, project, or outcome, including internal teams and external parties.Open glossary term→.

But that way of thinking comes from a different type of work.

User researchservice×User ResearchUnderstand user behaviour, validate ideas, and make clearer product decisions with evidence you can act on.Open service→ at this stage isn’t trying to measure how many people experience something.

It’s trying to understand why it’s happening at all.

What tends to surprise people, especially if they haven’t sat through many sessionsglossary×SessionA session is a single period of user interaction with a product, from entry to exit within a defined timeframe.Open glossary term→ themselves, is how quickly patternsglossary×PatternA reusable solution to a common design problem.Open glossary term→ start to appear. Not in a perfectly consistent way, not with users saying exactly the same thing, but in behaviourglossary×BehaviourBehaviour refers to how users interact with a system, including actions, patterns, and responses.Open glossary term→ that feels familiar almost immediately.

A hesitation in the same place.

A moment of uncertainty before committing.

Going back to check something that should already be clear.

Individually, those things are easy to dismiss. Together, they start to form a picture.

And once you’ve seen that picture a few times, it’s very hard to ignore.

There’s a reason the five users idea has stuck around for so long. Jakob Nielsen popularised the thinking that a small number of users is often enough to uncover the majority of usabilityglossary×UsabilityUsability is how easy and efficient it is for users to complete tasks within a product. It focuses on clarity, simplicity, and reducing effort so users can achieve their goals without confusion or friction.Open glossary term→ issues in a journeyglossary×Pain PointA specific problem or frustration users experience when trying to complete a task.Open glossary term→.

It gets quoted a lot, sometimes a bit too rigidly, but the reason it resonates is because it reflects what actually happens in practice.

You don’t need large numbers to spot problems.

You need repetition.

In reality, by the time you’ve seen the same frictionglossary×FrictionFriction refers to anything that slows users down or makes it harder for them to complete a task. It can be caused by poor design, unnecessary steps, unclear messaging, or technical issues.Open glossary term→ play out across a handful of people, you’re no longer dealing with coincidence. You’re seeing something structural. Something about the way the journeyglossary×Pain PointA specific problem or frustration users experience when trying to complete a task.Open glossary term→ is put together, the way information is presented, or the way decisions are being framed.

And structural problems don’t tend to affect just a few users.

They show up wherever the same conditions exist.

Where teams often get stuck is waiting for the signal to feel bigger.

More sessionsglossary×SessionA session is a single period of user interaction with a product, from entry to exit within a defined timeframe.Open glossary term→, more users, more confirmation, as if the problem will somehow become more valid the more times it’s observed. But in doing that, they end up circling the same insightglossary×InsightAn insight is a meaningful understanding that explains why something is happening and what it means.Open glossary term→ without actually moving anything forward.

The learning doesn’t deepen. It just repeats.

That’s usually where progress slows down.

Not because the team doesn’t understand the issue, but because they’re still looking for permission to act on it.

Small sample sizesglossary×Sample SizeSample size refers to the number of participants included in a study or test.Open glossary term→ don’t give you distribution. They don’t tell you how widespread something is, or how it varies across different segments. That kind of understanding comes later, once changes are live and you’re looking at real-world behaviourglossary×BehaviourBehaviour refers to how users interact with a system, including actions, patterns, and responses.Open glossary term→ at scale.

But at the point of discoveryglossary×DiscoveryDiscovery is the phase of understanding problems, users, and opportunities before building solutions.Open glossary term→, that’s not what you’re trying to answer.

What you’re trying to understand is whether something is off, and why.

And once you’ve seen enough to explain that clearly, more researchservice×User ResearchUnderstand user behaviour, validate ideas, and make clearer product decisions with evidence you can act on.Open service→ doesn’t necessarily make the decision easier. It just delays the moment where something actually changes.

In practice, the more useful question isn’t is this enough users?

It’s have we seen enough to understand the problem?

If the answer is yes, then you’re already in a position to move.

And in most cases, the thing holding teams back isn’t a lack of dataglossary×DataData is raw information collected and stored for analysis, processing, or decision-making.Open glossary term→.

It’s a lack of confidenceglossary×ConfidenceConfidence is the level of certainty in a decision or outcome based on available evidence.Open glossary term→ in acting on what they already know.