by Floyd Rumohr
Imagine a world without valid judgment: doctors unable to evaluate x-rays. Drinking water deemed safe just by looking at it. Judges drawing conclusions based on how they feel about a case rather than from standards and laws. Sounds like chaos.
Disorder at nonprofits isn’t always imaginary. It can be embedded in one organization’s culture and be a momentary lapse in judgment at another. Regardless of how often it appears, chaos is almost always certain to ensue if decision-making lacks valid reasoning and credibility, which is the degree to which judgments and conclusions are trusted and believed by others. Credibility is undermined when nonprofits make unsubstantiated claims about their work or if outputs alone are considered “impact.” The number of tickets sold at a theater performance is an output, for example, while the degree to which the audience is affected by that same performance is the impact.
Sean Stannard-Stockton defines outcomes as “observed effects of the outputs on the beneficiaries of the nonprofit.” Impact is “the degree to which the outcomes observed are attributable to its activities.” Knowing how inputs and activities are causally connected to organizational impact is essential if nonprofits want to understand what’s working and what’s not, improve performance, and back-up claims. That’s where evaluation comes in.
Two basic kinds of evaluation are described by Dr. Jens J. Hansen of the Woodhill Park Research Retreat:
- Formative evaluation is a kind of “stock-taking.” It can and should be done at regular intervals and provides vital information from which midcourse corrections will be made. Assessment and research play a big role here. Teachers often use formative evaluation and assessment methods to monitor how well students are learning and to make changes to the instructional methods and content depending on how those assessments come out. Focus is on the process.
- Summative evaluation is the “final test” and demonstrates what has or has not been achieved in the strategic plan. “Here evaluation rigorously interrogates organizational performance with respect to whatever goals and objectives are nominated in the strategic plan.” A standardized reading test for fourth grade students is an example of summative evaluation. Focus is on outcomes.
Both approaches play a role in strategic planning because one provides information during which course corrections at the program level are made (formative) and the other generates credible information for broader organizational decisions (summative). Both reflect evidence-mining processes that characterize learning cultures and help to connect an organization’s inputs (causes) to its outcomes/impact (effect) — otherwise known as the logic model.
The logic model is not about simple math. You can’t “prove” it no matter how sensible it is. “The term ‘scientific proof’ is a misnomer: proof is a concept germane to mathematics, not science…Scientists, including social scientists, develop hypotheses about how the world works and then gather evidence to support or undermine those hypotheses. Whereas proof is black-and-white, evidence has shades of gray: it can be strong or weak, circumstantial or conclusive” (Moss).
Put more simply, there are degrees of evidence for claims about impact but not proof of those claims. For a small nonprofit, this process of evidence gathering is a significant investment especially if outside researchers are involved. Research grants from funders can be so highly competitive as to put them out of reach. So how does a small nonprofit with limited resources credibly gather such evidence?
Start with the organization’s logic model. What are the inputs and activities? What are the outcomes and impact of those activities? How do you know the impact is a result of organizational activities? If the organization has several outcomes described in its impact statement, it’s probable only one of them can be tackled at a time. Don’t try to do everything at once. Credible evidence in support of one outcome or strategic goal can be a good road to take when resources are limited. Independent researchers and evaluators are a helpful “outside eye” and more financially accessible to small nonprofits when evaluative activities are focused.
Like it or not, persuasive impact statements are here to stay. Emerging philanthropists with voracious data appetites will be looking more and more for credibility behind claims in public presentations on Guidestar, in grant proposals, and web sites.
If your organization relies more on fiction than on fact to describe its impact, it might be time for some research, evaluation, or assessment.
Next up: alliances and partners.
References and recommended reading:
- “A Strategic Planning Template for Dummies” by Dr. Jens J. Hansen, Woodhill Park Research Retreat: a helpful guide for nonprofit organizations with a comprehensible summation of evaluation in the context of strategic planning.
- “In Defense of Logic Models” by Ian David Moss, Huffpost Arts & Culture Blog: unpacks logic models and explains theory of change/program theory from arts organizations’ perspectives.
- “Getting Results: Outputs, Outcomes and Impact,” Stanford Social Innovation Review by Sean Stannard-Stockton.
- Charting Impact: BBB Wise Giving Alliance, GuideStar USA, and Independent Sector developed Charting Impact as a common presentation that allows staff, boards, stakeholders, donors, volunteers, and others to work with and learn from each other.
- “Stages of Learning Research Study” by Jane Remer and Floyd Rumohr: this four-year study at PS 145, Brooklyn, postulated research questions that provided an evidentiary basis for subsequent organization-wide summative evaluation.