By Dr. Peter Schubert
Discussions around impact and impact measurement have become ubiquitous in the nonprofit sector. While originating from settings of social service provision, the impact debate has now also reached the cultural sector. Reasons for focusing on evidence that organizations are in fact making a difference in the world are manifold: Stakeholders nowadays expect to see results as part of an organization’s accountability and transparency – even more so in times when select cases of financial misconduct gain high public visibility through the media and can easily erode trust in the sector at large. Likewise, the availability of new digital information technology increases expectations of information availability and facilitates the ease of capturing data through organizational networks. Seen more proactively, organizations themselves can profit from measuring impact both to differentiate themselves from competitors on the market for donations and grants and to genuinely improve organizational practice and innovation.
In fact, demands for greater accountability of nonprofit organizations are not a new phenomenon. They have been around ever since the 1990s when nonprofit organizations themselves were seen as a means to increase transparency and trust in public institutions (Anheier, 2009). What has however accelerated the contemporary impact debate is the so-called social investment paradigm. While philanthropic behavior and institutional grantmaking have traditionally been conceptualized as a one-way transaction from the donor to the recipient, a social investment lens suggests a fundamentally different logic: a two-sided transaction in which donors expect returns for their contributions. Unlike a traditional investment, these returns are usually not captured by donors themselves, but instead by society at large, comprising multiple dimensions, including social, cultural, political or economic returns (Then et al., 2018). The proliferation of such an investment logic in the nonprofit sector can be attributed in a substantial part to a new generation of major philanthropists who have introduced business parlance and practice into civil society, claiming to tackle grants societal challenges through innovative approaches.
Even when embracing the social investment logic, it is important to acknowledge that it is all but trivial to precisely define and measure impact. We all share a common understanding of what impact is in general: the “significant or lasting changes in people’s lives following an activity.“ (Roche, 1999); or “the lasting results achieved at a community or societal level.” (Ebrahim & Rangan, 2014). However, developing a sound logic model, that clearly maps through what mechanisms an organization believes its activities change something in the world, is not an easy endeavor. Just like in many empirical research projects, developing a theoretical framework and reflecting on its assumptions is key before operationalizing key indicators, engaging in the costly effort of gathering data, and accurately interpreting results. As organizations develop their logic models, they would ideally embrace a participatory process that engages various organizational stakeholders. This is important as different groups can have fundamentally different conceptions of what an organization is trying to achieve and how it affects people’s lives.
This holds true for cultural organizations in particular, where impact is notoriously difficult to capture holistically. Cultural organizations do not only provide “tangible” or “instrumental” value as a direct means for another sphere of life (e.g., gaining knowledge and skills, fostering community identity, contributing to social inclusion). These organizations also provide numerous “intrinsic” values where impact is an end in itself (e.g., changes in attitudes, fostering creativity, cultural empowerment or joy) (Bollo, 2013). A common observation is that organizations choose the “easy way out”, capturing program participation data (such as the number of visitors) as an impact measure. However, participation only captures an output, i.e. what the organization did or how many people showed up. It does not tell is much about outcomes or impact, i.e. what it ultimately achieved.
The complexity of impact measurement does not lie with operationalizing accurate indicators, but more fundamentally with developing a sound research design that acknowledges key measurement challenges.
“Impact implies causality; it tells us how a program or organization has changed the world around it. Implicitly this means that one must estimate what would have occurred in the absence of the program—what evaluators call “the counterfactual.” (Gugerty & Karlan, 2018)
Although solving the counterfactual problem can be achieved through carefully controlled and randomized studies, this is usually beyond the scope of what most organizations can do. In fact, drawing on a mix of non-experimental quantitative methods (such as surveys) and complementary qualitative methods (such as interviews) represents for many organizations a much more feasible approach to capture impact dimensions. Nevertheless, taking into account the key impact measurement challenges should always be part of any serious impact study: What could have happened anyway, what happened because of others? How lasting was the effect? Do we have non-measurable effects? (Then et al., 2018)
The key point here is that any effort of measuring impact requires a substantial investment from the organization: It takes financial, human and technological resources to develop a logic model, design the study, carry out data collection etc. What complicates this investment from a nonprofit management perspective, is that these required resources oftentimes fall within the realm of overhead costs. While donors and institutional funders alike want to see organizations demonstrate their impact, they also prefer that organizations use granted funds for programs, and not for general administration or operations. Our research finds that donors prefer giving to organizations with lower overhead (Charles, Sloan, & Schubert, 2020). We also know that many governmental agencies tie their grantmaking to very low indirect cost rates for organizational infrastructure (Gregory & Howard, 2009), as they themselves are subject to democratic accountability and pressures for financial leanness.
This creates a systemic problem known as the nonprofit starvation cycle (Gregory & Howard, 2009; Schubert & Boenigk, 2019; Charles, Sloan, & Schubert, 2020). The cycle suggests that unrealistic expectations for low overheads exert pressure on organizations, as they compete over scarce external resources. This pressure in turn keeps organizations from investing in their organizational infrastructure (including financial, human and technological resources needed for measuring impact), in turn reaffirming funders in their expectation that organization can realistically operate on low overheads.
Against this background, increasing the prevalence of impact measurement will hinge on the implementation of a different approach to resourcing organizations. Much of this push needs to come from external funders and donors. In the foundation world, we lately observe concerted efforts to increase capacity-building and supporting general infrastructure of grantees (Eckhart-Queenan et al., 2019). However, a move towards more impact measurement equally hinges on how organizations themselves try to signal accountability and trustworthiness. They need to resist the temptation to equate efficiency (or even effectiveness) with individual financial metrics, such as low overhead ratios in favor of more holistic reporting frameworks of multiple performance metrics.
Author bio
Peter Schubert is a Postdoctoral Researcher in Nonprofit Mangement at the University of Hamburg. His research interests include nonprofit accountability, stakeholder management, and financial management. He completed his doctoral dissertation on the topics of overhead costs and the nonprofit starvation cycle.
References
Anheier, H. K. (2009). What kind of nonprofit sector, what kind of society? Comparative policy reflections. American Behavioral Scientist, 52(7), 1082-1094.
Bollo, A. (2013). The Learning Museum. Report 3 – Measuring Museum Impacts, Available at https://online.ibc.regione.emilia-romagna.it/I/libri/pdf/LEM3rd-report-measuring-museum-impacts.pdf
Charles, C., Sloan, M. F., & Schubert, P. (2020). If Someone Else Pays for Overhead, Do Donors Still Care? The American Review of Public Administration, 50(4-5), 415-427.
Ebrahim, A., & Rangan, V. K. (2014). What impact? A framework for measuring the scale and scope of social performance. California Management Review, 56(3), 118-141.
Eckhart-Queenan, J., Etzel, M. Lanney, J, & Silverman, J. (2019). Momentum for change: Ending the nonprofit starvation cycle. Boston, MA. Retrieved from: https://www.bridgespan.org/ending-the-nonprofit-starvation-cycle
Gregory, A. G., & Howard, D. (2009). The nonprofit starvation cycle. Stanford Social Innovation Review, 7(4), 49-53.
Gugerty, M. K., & Karlan, D. (2018). Ten reasons not to measure impact—And what to do instead. Stanford Social Innovation Review, Summer 2018, 41-47.
Roche, C. (1999). Impact Assessment for Development Agencies: Learning to Value Change.
Schubert, P., & Boenigk, S. (2019). The nonprofit starvation cycle: Empirical evidence from a German context. Nonprofit and Voluntary Sector Quarterly, 48(3), 467-491.
Then, V., Schobert, C., Rauscher, O., Kehl, K. (2018). Social Return on Investment Analysis. Measuring the Impact of Social Investment. Palgrave.
