The Kellogg Foundation Evaluation Handbook says “Effective evaluation is not an ‘event’ that occurs at the end of a project, but is an ongoing process which helps decision makers better understand the project; how it is impacting participants, partner agencies and the community; and how it is being influenced/impacted by both internal and external factors.” For more information about evaluation, download the Evaluation Handbook here.
To begin, it is important to distinguish among some of the kinds of evaluation. Summative evaluation takes place after a project is concluded and is generally used to determine if a project met its goals and what its impact was. Formative evaluation takes place during project design and during a project’s implementation and is generally used to modify and improve a project. Today’s article is looking more deeply into formative evaluation by examining process, also called implementation evaluation.
Many times, however, we unknowingly use neither formative nor summative evaluation and instead use another system of evaluation… the Black-Box Evaluation System.
“Black box evaluation is measuring the effects of a program or service without examining the nature of that program or service.” Black-box evaluation looks at what happens after an intervention and attributes the change or lack of change to what happened in the black box. But too often, we don’t open the black box to see what is inside. So we don’t know what elements of the program or service had an effect and why they had the effect. We don’t know what we need to change or what we can replicate in the future.
Let’s reexamine how we design, carry out and use evaluation and let’s open that Black Box. One effective way to get inside the Black Box is through the use of Process or implementation Evaluation. This type of evaluation is used worldwide by many organizations including the Bureau of Justice, the Center for Disease Control, and the World Health Organization. (Note to reader: while both terms are used, for this article we will be using the term process evaluation unless when implementation evaluation is used in a quote.)
What is process evaluation?
Process evaluation collects detailed information about the “active ingredients” in a project as it is being implemented and provides valuable information about what worked and what did not and helps to identify where improvements might be made. Process evaluation aims to enhance your current projects by understanding them more fully. It analyzes if the strategies were implemented as planned and whether the expected outcome was achieved. Its focus is internal and precedes any external accountability.
Why is process evaluation important?
Process evaluation shifts our focus to understanding our programs more fully, to knowing what works and to looking for solutions. We use evaluation to prove a project worked but also to learn how to improve it. We can examine the relationship among our program resources, the quality of the program and the anticipated outcomes. Consider the difference in what we as an organization can learn if we examine the entire program instead of only the outcome for funders.
Again from the Kellogg Handbook of Evaluation, “An implementation evaluation allows you to put this outcome data in the context of what was actually done when carrying out the project. In fact, without knowing exactly what was implemented and why, it is virtually impossible to select valid effectiveness measures or show causal linkages.”
The Stanford Social Innovation Review broadens this perspective and suggests process evaluation can help us look at how we can improve our impact. They suggest four questions to consider: 1. Are there particular conditions or program components that have a greater impact on results? 2. Does the program have better results with certain subgroups? 3. How does impact change over time? 4. What are the positive and negative effects on the communities not directly accessing the programs? To answer these questions, one must first understand the program’s implementation.
How do we do a process evaluation?
A process evaluation does not need to be overly complex but it does need to include some basic components and requires careful input when first planning your program. Many of us use a logic model to develop our programs. Evaluating the input, the activities and the output – the first three columns in a logic model – is just as important as evaluating the last two columns – outcomes and impact.
KEY ELEMENTS OF A PROCESS EVALUATION:
1. Set yourself up for success
Your time is limited, so setting up a system at the beginning of a project is important.
These steps will help to ensure that the implementation process is an ongoing focus during the entirety of the project and also sends a clear message to the organization that data and evaluation are important to the entire organization.
2. Understand the design of the project
Before designing a process implementation, be sure all stakeholders know about the project. If a logic model has been developed, be sure to use that. Everyone on the evaluation team should be able to answer these questions:
3. Collect the data and analyze the project’s implementation
Your key here is to collect data from multiple sources and perspectives, to use a variety of methods, and to make it ongoing. The information collected can range from individual interviews to observation to focus groups to quantitative data. Just be sure to think carefully about what information you need and develop your system for collecting and analyzing the information.
Some key questions to consider:
4. Use the evaluation findings!
Use the evaluation findings to improve the project.
Consider what your organization has learned through the evaluation.
A Final Thought… Changing the mindset of how we view evaluation puts evaluation into a different perspective, one that helps our clients and our organization! Evaluation can be much more than reporting the end-of-project outcomes when we use it as a learning tool to understand our projects and to improve what we do. By opening the black box, we began to understand how our programs are working and how our impacts are achieved.