The Kellogg Foundation Evaluation Handbook says “Effective evaluation is not an ‘event’ that occurs at the end of a project, but is an ongoing process which helps decision makers better understand the project; how it is impacting participants, partner agencies and the community; and how it is being influenced/impacted by both internal and external factors.” For more information about evaluation, download the Evaluation Handbook here.
To begin, it is important to distinguish among some of the kinds of evaluation.
Summative evaluation takes place after a project is concluded and is generally used to determine if a project met its goals and what its impact was. Formative evaluation takes place during project design and during a project’s implementation and is generally used to modify and improve a project. Today’s article is looking more deeply into formative evaluation by examining process, also called implementation evaluation.
Many times, however, we unknowingly use neither formative nor summative evaluation and instead use another system of evaluation… the Black-Box Evaluation System.
“Black box evaluation is measuring the effects of a program or service without examining the nature of that program or service.” Black-box evaluation looks at what happens after an intervention and attributes the change or lack of change to what happened in the black box. But too often, we don’t open the black box to see what is inside. So we don’t know what elements of the program or service had an effect and why they had the effect. We don’t know what we need to change or what we can replicate in the future.
Let’s reexamine how we design, carry out and use evaluation and let’s open that Black Box. One effective way to get inside the Black Box is through the use of Process or implementation Evaluation. This type of evaluation is used worldwide by many organizations including the Bureau of Justice, the Center for Disease Control, and the World Health Organization. (Note to reader: while both terms are used, for this article we will be using the term process evaluation unless when implementation evaluation is used in a quote.)
What is process evaluation?
Process evaluation collects detailed information about the “active ingredients” in a project as it is being implemented and provides valuable information about what worked and what did not and helps to identify where improvements might be made. Process evaluation aims to enhance your current projects by understanding them more fully. It analyzes if the strategies were implemented as planned and whether the expected outcome was achieved. Its focus is internal and precedes any external accountability.
Why is process evaluation important?
Process evaluation shifts our focus to understanding our programs more fully, to knowing what works and to looking for solutions. We use evaluation to prove a project worked but also to learn how to improve it. We can examine the relationship among our program resources, the quality of the program and the anticipated outcomes. Consider the difference in what we as an organization can learn if we examine the entire program instead of only the outcome for funders.
Again from the Kellogg Handbook of Evaluation, “An implementation evaluation allows you to put this outcome data in the context of what was actually done when carrying out the project. In fact, without knowing exactly what was implemented and why, it is virtually impossible to select valid effectiveness measures or show causal linkages.”
The Stanford Social Innovation Review broadens this perspective and suggests process evaluation can help us look at how we can improve our impact. They suggest four questions to consider: 1. Are there particular conditions or program components that have a greater impact on results? 2. Does the program have better results with certain subgroups? 3. How does impact change over time? 4. What are the positive and negative effects on the communities not directly accessing the programs? To answer these questions, one must first understand the program’s implementation.
How do we do a process evaluation?
A process evaluation does not need to be overly complex but it does need to include some basic components and requires careful input when first planning your program. Many of us use a logic model to develop our programs. Evaluating the input, the activities and the output – the first three columns in a logic model – is just as important as evaluating the last two columns – outcomes and impact.
1. Set yourself up for success
Your time is limited, so setting up a system at the beginning of a project is important.
- Develop a reporting calendar – Make clear when implementation data are to be reported and by whom.
- Identify a “Data Chief/Top Dog” – one person who is in charge of collecting, organizing and presenting the data.
- Schedule a regular time to review the reports on the program’s implementation.
- Set up an evaluation team to help review the data and recommend any modifications.
- Establish how follow up will take place. Do the follow up!!
These steps will help to ensure that the implementation process is an ongoing focus during the entirety of the project and also sends a clear message to the organization that data and evaluation are important to the entire organization.
2. Understand the design of the project
Before designing a process implementation, be sure all stakeholders know about the project. If a logic model has been developed, be sure to use that. Everyone on the evaluation team should be able to answer these questions:
- What are the goals and intended outcomes of the project?
- What has been determined to constitute success?
- What major strategies are planned?
- Who are the clients?
- What are the critical components/activities of the project?
- What resources will be used?
- What is the data collection plan?
3. Collect the data and analyze the project’s implementation
Your key here is to collect data from multiple sources and perspectives, to use a variety of methods, and to make it ongoing. The information collected can range from individual interviews to observation to focus groups to quantitative data. Just be sure to think carefully about what information you need and develop your system for collecting and analyzing the information.
Some key questions to consider:
- Whose perspective will we be seeking (staff, clients, administration, stakeholders)?
- Was the project successful? Why or why not?
- How has the project been received?
- Was the communication effective?
- What could be changed to improve the project?
- How many who needed the service actually used or received it?
- Were all the planned activities/interventions implemented?
- Did the implementation process take place as planned?
- What problems were met during implementation? How were they handled?
- Were objectives, plans or schedules changed? How? Why?
- Were any personnel changed?
- Did any external or internal variables take place that affected the program and its implementation?
4. Use the evaluation findings!
Use the evaluation findings to improve the project.
- Did the project accomplish what it set out to do?
- What were the strengths of the project?
- What activities or practices will be maintained? Which ones need to be revised or removed?
- What other changes, if any, need to be made?
- How will we monitor these changes?
Consider what your organization has learned through the evaluation.
- Do you have a better idea of how or why the project worked?
- Do you know for which client the projects worked most effectively?
- Did you learn something that can be shared across the organization or between organizations?
- What lessons have been learned that may be helpful in the future?
- Will we replicate this design in other settings?
- Have you documented the changes?
A Final Thought… Changing the mindset of how we view evaluation puts evaluation into a different perspective, one that helps our clients and our organization! Evaluation can be much more than reporting the end-of-project outcomes when we use it as a learning tool to understand our projects and to improve what we do. By opening the black box, we began to understand how our programs are working and how our impacts are achieved.
