This article covers what program evaluation is and how to make it useful. We’ll work through these topics in order so feel free to skip ahead to whatever is most relevant to you:
What program evaluation is for
Why collecting data is only part of the work
Outputs and outcomes: what you delivered and what changed
Formative and summative evaluation, and the questions each can address
Whether you need an impact evaluation
Program evaluation versus project evaluation
How evaluation relates to research, monitoring, and performance measurement
How to start a program evaluation
What makes an evaluation good
Building evaluative thinking into everyday decisions
Program Evaluation for Nonprofits
Is this working?
How do we know?
And what are we going to do differently because of what we've learned?
Those are the questions I want program evaluation to help us answer.
Program evaluation is a systematic way to gather and interpret evidence so you can assess and better understand a program and make decisions about it. In plain language: what works, what doesn't, for whom, and how can we improve?
To simplify it even more, evaluation asks: what? so what? now what? I often add “to what end?” which I suppose is just a fancy way of saying “why?” in the end, but I find it helps the people I’m working with get closer to the objective.
Figuring out the answer to those questions can involve surveys, interviews, existing records, observations, statistical analysis, or a combination of methods (mixed methods). But before choosing any of them, you need a reason to collect the information.
A survey is very easy to send. Deciding what you'd do with the answers takes more thought.
What is program evaluation for?
You might need to decide whether to continue a program, change how it's delivered, expand it to a new setting, or stop investing in something that isn't producing enough value.
You might need to understand why one site is having a very different experience from another. Or whether the people you intended to reach can actually participate.
The question determines the work.
The CDC's 2024 Program Evaluation Framework provides a formal structure for evaluation. It places collaboration, equity, and learning and use throughout the process. Those are useful principles to keep in view even when the program you're evaluating has nothing to do with public health. [*]
For me, a useful starting question is: Who needs this information, and what decision will it help them make?
That emphasis on intended use is central to utilization-focused evaluation, developed by Michael Quinn Patton and described with Charmagne E. Campbell-Patton in the fifth edition of their book. [*]
Collecting data doesn't finish the evaluation
Imagine a fictional nonprofit that delivers 20 workshops to 400 people.
You know it delivered workshops. You know something about its reach.
You don't yet know whether those workshops were useful, whether people could apply what they learned, whether some participants were excluded, or whether the format was worth the time and cost.
A spreadsheet can be full and still leave the important questions unanswered.
Evaluation brings the information back to a question and a basis for judgment. What were you trying to achieve? What would count as good enough? What does the evidence support? What might explain what you're seeing?
A dashboard, interview, or survey can contribute to that work. None makes the judgment for you.
Outputs and outcomes: what happened, and what changed?
Outputs describe the work delivered. Outcomes describe the changes you hoped that work would help produce.
For the fictional workshop program, outputs might include the number of sessions, the number of participants, and the materials distributed.
Outcomes might include participants learning a skill or applying it afterward.
You need both. Attendance matters, especially if a program isn't reaching the people it was designed to serve. But a high attendance count doesn't establish that anyone benefited.
It also matters how you know an outcome occurred. A participant saying they intend to use a skill is different evidence from observing them use it later. Either could be useful, depending on the question. Label the difference.
The main types of program evaluation
Evaluation labels can get confusing because they describe different aspects of the work. Some refer to purpose, others to questions or timing. They aren't a menu of mutually exclusive options.
A useful way to organize evaluation is by its purpose: formative or summative. [*]
Formative evaluation helps you improve a program as it develops or operates. You might use it to understand why recruitment is difficult, whether participants can use a new resource, or what needs to change before expanding.
Summative evaluation supports a judgment about a program's merit, worth, or effectiveness at a defined point. It can help inform whether to continue, expand, redesign, or stop a program. The program doesn't have to be over for you to make that judgment.
Within either purpose, you can ask different kinds of questions. Process or implementation evaluation examines delivery: what happened, whom the program reached, whether it operated as intended, and what helped or got in the way.
Outcome evaluation examines whether the changes you're interested in occurred. Observing a change doesn't, by itself, establish that your program caused it.
Impact evaluation often addresses that causal question, including what might have happened without the program. Terminology varies across fields, so describe the question and design clearly. [*]
These labels don't form a rigid sorting system. Implementation findings can guide improvements or help explain an overall judgment. Outcome findings can inform a decision about continuing a program or reveal something to change now. An evaluation can serve both formative and summative purposes.
For example, a workshop evaluation might examine delivery and early learning to improve the next round. It might later combine implementation, outcome, and cost evidence to judge whether the program is worth continuing.
Start with what you need to decide, then choose the questions and evidence that will help you decide it.
Do you need an impact evaluation?
Maybe. But probably not. Either way, don't choose it because “impact” sounds more impressive.
If a program isn't reaching its intended participants, that's worth understanding. If sites are delivering quite different versions of the program, that's worth understanding too.
Those questions can be consequential without requiring a claim about causality.
Attribution asks whether a program caused a change. Contribution asks about its role alongside other influences. Being careful about what your evidence can establish strengthens the work; it doesn't make the work less valuable.
I've written more about implementation evaluation, including why understanding delivery matters when you're trying to interpret outcomes. The practical point is to know what was actually implemented before making a confident claim about whether it worked.
Program evaluation vs. project evaluation
A project often has a more bounded scope, deliverable, or timeframe. A program may include several activities or projects working toward shared outcomes.
That can change the size and focus of the evaluation. It doesn't create an entirely different set of principles.
If the terminology is becoming a distraction, describe the work plainly: what are we evaluating, over what period, for whom, and for what decision?
Evaluation, research, and performance measurement
Research and evaluation can use the same methods. A useful distinction is purpose: research often seeks knowledge that extends beyond a particular decision, while evaluation is typically organized around judging or improving something specific. There is overlap, so avoid treating that distinction as an absolute rule. I like to think of it as similar to “a square is a rectangle but a rectangle is not a square.” Research can have evaluation, but evaluation is not research.
Monitoring and performance measurement help you track what is happening over time. How many people took part? How much did delivery cost? Did an indicator change? If you've met every activity target but still don't know whether the work helped, counting more activities may not answer the question. Evaluation can use that information and ask further questions: Why did it change? How should we judge the result? What else could explain it? What should we do?
How to start a program evaluation
Start with the decision.
“What do we need to understand, and what will we do differently depending on the answer?”
Then describe how the program is supposed to work. Include its resources, activities, intended outcomes, and important assumptions. Talk with the people who deliver and experience it. Their understanding may differ from what's in the funding proposal.
Choose a manageable set of questions. Review the information you already have before asking anyone to complete another form.
From there, decide what additional evidence you need, how you'll collect it, and how you'll analyze and interpret it. Be clear about whose perspectives are included and what the evidence won't tell you.
Plan the discussion after the analysis, too. A report doesn't organize its own use.
This is an outline of the work, not a requirement that every evaluation follow a perfectly straight line. You may need to revisit the question when you understand the context better.
What makes an evaluation good?
I always come back to usefulness, equity, consistency where it matters, and feasibility. These words summarize my approach to evaluation generally, but there are other guiding principles you can follow, like these from the American Evaluation Association (AEA).
Useful: Someone can make a better-informed decision with what you learn.
Equitable: People affected by the work have a meaningful role in shaping the questions, interpreting the findings, and defining what matters. My piece on equitable evaluation explores that further.
Consistent where it matters: You collect enough comparable information to examine patterns across sites or over time, while noticing when the context calls for a different approach.
Feasible: The organization can do the work and use the findings without asking staff or participants to carry an unreasonable burden.
These considerations belong alongside credible methods and honest reporting of limitations. A small study still needs to be well designed. A sophisticated analysis still needs to answer a useful question.
Evaluation can become a habit
You won't commission a formal study every time you need to learn something. But you can build the habit of evaluative thinking: stating what you hope to achieve, examining your assumptions, asking what evidence would help, and changing your approach when the evidence gives you a reason.
That can happen in a planning conversation as well as in a formal report.
Before collecting more data, try writing down one decision the information would change. If you can't identify one, spend some time there first.
And if you want help working out the questions, methods, or next decisions for your organization, read about my evaluation services.