The Evaluation Questions Nobody Asks — But Should

Most nonprofits approach evaluation with a familiar set of questions. How many people did we serve? Did participants improve? Did we meet our targets? These are reasonable questions. They're also incomplete ones.

The problem isn't that organizations ask the wrong questions. It's that the questions they default to were often chosen years ago, shaped by a funder's template or a logic model that made sense at the time. And once questions get embedded in a reporting cycle, they rarely get revisited.

But what you measure — and how you frame the question — quietly determines what you're able to learn. And in most organizations, the most important questions are the ones nobody thought to ask.

What happened is only half the story

Outcome questions tend to focus on change over time: did skills increase, did behaviors shift, did well-being improve? That's valuable. But outcomes don't exist in a vacuum. They happen to specific people, under specific conditions, inside specific contexts.

When evaluation stops at 'what happened,' it can't answer the questions that actually shape strategy:

• What happened for whom?

• Under what conditions did the program work — and when did it struggle?

• What was true about the people who made the most progress, compared to those who didn't?

These aren't harder questions to ask. They're just less common ones. And they generate insight that average outcomes never can.

Compared to what?

Another question that rarely makes it into evaluation plans: compared to what?

When an organization reports that 72% of participants improved on a given measure, that number sits in isolation. Is that good? Is it what you'd expect without the program? Is it better or worse than it was three years ago? Better or worse than a similar organization working with similar people?

Without some point of comparison — prior data, a benchmark, a control group, a peer organization — a single number doesn't tell you much. Context is what gives data meaning. Organizations that build comparison points into their evaluation design end up with findings that are genuinely informative, not just reportable.

What would have to be true?

Perhaps the most underused evaluation question of all is this one: What would have to be true for our theory of change to hold?

Every program is built on assumptions. That participants will engage consistently. That the dosage is sufficient. That the problem being addressed is actually the root cause. That external conditions won't undermine the intervention.

Naming those assumptions — and then asking whether the data supports them — is a different kind of evaluation work. It's less about measuring outcomes and more about testing logic. And it's often where the most important learning happens.

The shift that matters

Better evaluation doesn't always mean more evaluation. It often means slowing down and interrogating the questions before collecting a single data point.

What are we actually trying to understand? Who needs to benefit, and what does benefit mean for them specifically? What would change our thinking? What do we already believe — and might we be wrong?

These questions don't fit neatly on a funder's reporting form. But they're the ones that turn evaluation from a compliance exercise into something that genuinely improves the work.