Opinion: Impact indices can supercharge impact measurement
When organisations select their preferred impact metrics in isolation from others, it wastes time and makes it difficult to compare results with others. But there's an alternative – and surprisingly underused – solution, say Tom Adams and Madeline Copp of 60 Decibels.
Measuring social impact, while short of being quixotic, is undoubtedly complex. Our lives, our values, our hopes and dreams even, are multi-faceted and dynamic. How we experience the world around us and what we hold as material to our wellbeing is determined both by innate personal beliefs, as well as changing societal norms. For those of us in the business of creating or measuring social impact this is enough to make your head spin.
Faced with this challenge we have typically opted for one of two extreme approaches. Ultra-simplicity: searching for a holy grail metric; or ultra-complexity: bamboozling ourselves with frameworks containing metric overload. An alternative and, to date, surprisingly underutilised approach to encourage more effective impact performance measurement and management is to adopt impact indices.
Precedent at a macro level
Indeed, when it comes to measuring the social wellbeing of countries there is a long precedent of using indices. An index is an aggregation of multiple individual indicators into a single overarching metric. As far back as 1980, Pakistani economist Mahbub ul Haq, frustrated that GDP was a poor proxy for the social wellbeing of a country, designed the Human Development Index (HDI). Based on the thinking of the Nobel prize winning economist Amartya Sen, the HDI uses multiple measures across three themes – health, education and living standards – to compare poverty rates across countries.
In recent years, interest in indices in international development has increased further. The equivalent of the Marvel’s Avengers of economics—Amartya Sen, Joe Stiglitz and Jean-Paul Fitoussi—recently published the Report by the Commission on the Measurement of Economic Performance and Social Progress, which highlights the essential role that indices can play to give us a better sense of our health and social wealth. The report argues that the growing number of internationally robust and comparable datasets represent an untapped opportunity. With them, we might be able to better compare living standards between nations. This could be a boon for setting both international policies and flows of international aid.
Given the popularity of indices for considering country-level social wellbeing, it may be surprising that the use of micro-indices for understanding social performance at the level of individual organisations is almost unheard of. And while worthy initiatives such as the Impact Reporting Investment Standards (IRIS) have developed extensive taxonomies of metrics, very few projects have sought to effectively combine such measures to cut through the complexity of tracking impact performance.
Two similar organisations independently choose a different set of measures: one decides metrics A, B and C are important, the other choses B, D and E
In the absence of such established indices, and of a common understanding of what combinations of metrics matter most when and where, organisations seeking to understand their impact have tended to separately pick a selection of individual metrics. This has caused two problems. First, a lot of wasted time. Individual organisations providing similar (or even identical) products or services craft their own theories of change and come up with bespoke collections of performance measures. Which has led to the second problem: a lack of comparability. Two similar organisations independently choose a different set of measures: one decides metrics A, B and C are important, the other choses B, D and E. As a result, we’re left with two organisations attempting to solve the same social challenge, with little way to understand how their results compare, nor what one can learn from the other.
Building an index: what goes in?
For the data geeks among us, there are some well-established criteria for how to craft an effective index: validity (the indicators chosen all measure a meaningful part of the greater whole); reliability (results can be consistently reproduced under the same conditions), uni-dimensionality (limited, or better yet no, inter-relation between the underpinning measures); and weighting (are all measures equal). These may sound a little complicated, but they needn’t be so.
For example, the easiest way to check if an indicator is valid is to simply ask the people who experience change to describe what aspects of impact are most material to them. This approach was taken in the construction of the UK government’s national equalities framework in 2017. The authors spoke to members of different groups (categorised according to age, gender, racial diversity, religion, sexual orientation and disability), asking them to describe the elements of wellbeing that were material to their lived experience. There was variation, of course, but the yearlong process led to a Wellbeing Index that all groups could support. This Index included measures hitherto not incorporated into national wellbeing measures, such as personal autonomy, self-esteem and absence of workplace discrimination.
Read more from Jeremy Nicholls: Stakeholder engagement: if it’s not transferring power, don’t bother
Indices equals benchmarking equals decision making
What is so important about indices is their ability to provide comparison across multiple (potentially competing) phenomena. Behavioural studies have shown that the ability to contrast and compare is vital for decision making. As Gayannée Kedia and Thomas Mussweiller, psychology professors from the University of Cologne put it, “all judgments and evaluations are relative in nature and hence rely on comparisons.” After all, it is only possible to describe yourself as tall if you know the average height of others.
It is only possible to describe yourself as tall if you know the average height of others
Sticking with height for a moment, history tells us that if we can’t make like-for-like comparisons, we can easily derive false conclusions. Napoleon is famous for two things: being a peerless general and being unusually short. It turns out only the former is true. The reason for the confusion is that at the time the English and French measurements of an inch varied slightly. For the French, an inch was 2.7 cm, while the English inch (the one universally used today) was shorter, at 2.54 cm. Measured the French way Napoleon came to a diminutive five foot two inches; using the English approach he was an average five foot five. Of course the English were all too keen to jump on the French interpretation to make the all-conquering general seem a little less ferocious, and the myth endures to this day.
And yet, like 19th century British generals, so much judgement of social impact seems to happen without useful and accurate comparators. The great socially-focused minds at companies, investors, foundations, development programmes – all aiming to create impact – often measure in isolation. Worse still, impact measurement is typically done using different metrics, different measurement approaches, different survey questions, and different methodological constructs. This is a recipe for a lack of quality decision making.
If you’re concerned about generating as much impact as possible (and especially if you’re a stakeholder that experiences that impact) this is a cause for significant alarm. To that end, the dissemination and adoption of carefully designed social indices hold the key to more effective impact measurement and, ultimately, better informed and more thoughtful decisions within the sector.
- Read: Data-driven, deal-ready: partners aim high with impact management programme for 300 social ventures
Different sectors, different indices
Since the beginning of 60 Decibels, our focus has been on listening to customers, beneficiaries or employees. First, so that they could describe the impacts that they experienced in their own words, and second, so that we could measure changes against these.
This listening remains a north star, and as we have done so again and again, we have seen consistency within sectors and business models with respect to the sorts of impacts end-beneficiaries highlight as most important to them, most of the time.
- Read more from 60 Decibels: The risk of leaving farmer vulnerability undetected: How remote surveys can measure changes in poverty
On the back of this work of listening to more than 200,000 beneficiaries across multiple sectors, we have begun to build sector-based indices. For example, the social wellbeing of a smallholder farmer is a function of their incomes, ability to withstand climate- and other weather-related shocks, food security, as well as less tangible but equally important measures such as their perceptions of the fairness of their trading relationships.
Similarly, for customers of off-grid energy in previously unelectrified communities, we have found that the benefits of such energy include changes in expenditure on expensive and dirty fuels, carbon savings, health improvements and improved perceptions of personal safety and asset security due to presence of light where there was previously darkness. How does one compare social value across so many measures at once? Indices hold the key.
We featured the first of these in our first major Impact Performance Report in the off-grid energy sector Why Off-grid Energy Matters; the second (recently published) and even more ambitious is our 60 Decibels Microfinance Index, where we were able to effectively rank the social performance of over 70 organisations using a standardised index. Among other lessons, our MFI index has shown that business success is top of mind for MFI clients (over funding household expenses or generally increased income); that women MFI clients report only marginally larger household gains compared to men; and that offering non-financial services doesn’t necessarily mean clients are doing better.
We believe that the future of impact measurement will be sectoral and thematic indices, adopted at scale
The future: indices at scale
Those of us in the social sector should always be on the lookout for ways to be more efficient with our dollars and our efforts to ultimately improve lives. Indices hold huge promise in helping us to better identify the efficacy of social solutions. Instead of multiple solutions measuring different types of impact, we believe that the future of impact measurement will be sectoral and thematic indices, adopted at scale. This will not only greatly reduce the confusion around what metrics matter, but also increase the comparability of data and the transparency of performance.
For individual organisations, this will illuminate strengths and opportunities for improvement. More broadly, social impact indices will enable improved decision making, programme development and investments, bringing us closer to our collective goal of improving lives and wellbeing for people around the world.
Tom Adams is the co-founder and chief strategy officer of 60 Decibels; Madeline Copp is strategy manager at 60 Decibels.
Top image courtesy of BRAC
Thanks for reading our stories. As an entrepreneur or investor yourself, you'll know that producing quality work doesn't come free. We rely on our subscribers to sustain our journalism – so if you think it's worth having an independent, specialist media platform that covers social enterprise stories, please consider subscribing. You'll also be buying social: Pioneers Post is a social enterprise itself, reinvesting all our profits into helping you do good business, better.