Whose perspective is it anyway?

Whose perspective is it anyway?

Measurement requires objective data. This statement does not seem very original, and yet most analysists do not spend much time understanding what this truly means. While data may be objective, deciding how and what data to collect is not.

A few years ago, a group of academics asked fellow researchers to conduct an analysis using the exact same data. The question asked seemed to be very simple and straightforward: are football referees more likely to give red cards to players with darker skin than to those with lighter skin? (https://www.nature.com/articles/526189a)

Given that the data is objective, the conclusions should have been very similar.

The result, however, was very surprising, with 69% of the teams concluding that there was a clear direct correlation between the red cards and the colour of the skin and 31% concluding that there was no direct effect.

In this example, using a basic rule of probabilities, where a large number of trials gives a more accurate result, we can conclude that there seems to be a trend between the colour of the player’s skin colour and red cards shown. Considering that 31% of the teams did not find any evidence of this indicates that although true, this result is to be taken with caution.

Most measurement exercises – especially those with limited funding – do not have the luxury of having several teams independently analyse the same data. They might therefore have misleading results – that might in turn spawn bad decisions.

If this is true for analysts when they have the full range of “objective” data, it is even more so for those who design the data collection exercise. In particular, when data collection is standardised to fit the researcher’s analysis model. Beneficiaries of social programmes might be given limited choices in impact measurement questionnaires which might not truly represent them, forcing them to select the options they feel less uncomfortable with. One example is the question on gender as it usually only allows selecting “male” and “female”, excluding every person that does not identify with any specific gender. In order to be more inclusive, the question might need to allow for more options. While the most inclusive option would be to allow for only open text answers (i.e. Please write your gender) it would dramatically complicate the data analysis process. It is thus necessary to evaluate how inclusive the analysis should be and if leaving a percentage of the target population out of the measuring exercise would dramatically impact the end results.

It is therefore crucial that decision makers who rely on data analysis are fully aware of the unconscious biases that might influence the “objective” evaluation and collection of data. When reporting on the conclusions, reference should be made on these influencing factors as a caveat for the final decisions.

Impact measurement should be as objective as possible but this is not an easy task. The measurement process therefore needs to be transparent and highlight any factors that might influence the final conclusions. The study should then be reviewed by different stakeholders with a varied range of perspectives before the final results are accepted. In the absence of multiple research teams working on the same set of data, having different groups of people independently review the conclusions is a more cost-effective alternative than having multiple research teams.  

Asier Achutegui, Senior Manager Microinsurance Network, is one of the InFiNe.lu scholarship programme recipient 2021. This is his second article on the Oxford Impact Investing Programme in 2022 he attended thanks to an InFiNe.lu scholarship.