Governance Matters: What are the origins of randomised controlled trials (RCTs) and what are their benefits?

Shobhini Mukerji: RCTs are borrowed from the field of medical science. They are a way of measuring the impact of a specific policy. At its simplest, participants are randomly split into one group that receives an intervention and one group – known as the control group – that does not. Researchers are then able to use the results to determine the effects of the intervention. Of course, this can also be adapted to create more groups subject to different types of intervention in order to study various factors at once.

In a medical setting, for example, you might have three groups of patients who all suffer from chronic headaches. One would be left alone as the control, another might be given a certain drug, and another group could be given that drug alongside another intervention such as reduced screen time. The differences, or lack of difference, between the groups’ symptoms would enable the scientists to discern what might be an effective intervention for chronic headaches.

The methodology delivers reliable results and also has a very broad application. RCTs have most often been used for health and education policy, but it is increasingly being used to tackle policy problems in other fields. J-PAL uses RCTs to test interventions across 11 sectors including the environment, labour, political economy and governance, health, and gender. Another reason RCTs are useful is that they can compare outcomes in a dynamic situation. This enables us to tell the difference between natural progression and the impact of the intervention.

‍

How does this work in practice when it comes to social policy?

It is very similar. At J-PAL, for example, we have done many RCTs in India to support a programme called “Teaching at the Right Level”. This programme emphasises that teaching should start at the level of the child and groups children by learning level rather than by grade or age.

We spent 10 years assessing this via RCTs in India. One thing we examined was whether it worked better with a teacher or a volunteer. Such a study would involve randomly splitting a cohort of children into two groups, one being taught as usual by a teacher and another by a volunteer, with the two groups’ results compared after a certain period of time. We also used RCTs to test how cost-effective the programme was and how best it could be generalised and integrated within government schools. Helped by the findings of many different studies, more than 60 million children across India and Africa are learning this way.

‍

Where are the most innovative uses of this methodology now?

Increasingly, we are looking at areas such as market design, environmental policies, industrial policies, labour market outcomes, and skills and training. In one example, we partnered with the Government of Gujarat in India to assess the efficacy of a new Emissions Trading Scheme (ETS) that was introduced for particulate emissions in the city of Surat. In the past, the challenge of manually collecting data, inaccurate reporting, and rampant collusion between auditors and regulators had made research difficult.

At the time, the Government was rolling out new technology in the form of Continuous Emissions Monitoring Systems (CEMS) across the state. Using this accurate reporting system, we were able to establish a sample of 321 factories. Half were chosen at random to participate in the first phase of the new ETS and half were left under the existing regulations. We found that the ETS market led to a 20-30% reduction in emissions relative to the control group. This was really helped by the adoption of the CEMS. As big data advances, it creates a lot of opportunities for us.

This actually followed a previous RCT we had conducted in Surat which examined pollution audits. There was a lot of corruption because the auditors would examine the same set of factories and over time would develop relationships with the factories’ managers. This increased the risk of them being bribed to report lower levels of pollution. A new policy was proposed that would see the auditors randomly allocated to different factories and then periodically rotated across the industry to stop them building those relationships. We conducted an RCT comparing the different systems and found the new policy led to an 80% drop in collusion and corruption.

‍

How can RCTs be complemented by other methods to support industrial strategy and planning?

People tend to think RCTs are highly quantitative and they are often correct. Running a randomised evaluation involves extensive, rigorous data collection. We have been able to enhance this by making use of multiple data sources. Governments have increasing amounts of administrative data that we are able to use as part of a hybrid model alongside our own findings.

There is also an overlooked qualitative element that draws in other methodologies. We have to do a lot of work to ensure that our insights are localised, and that means understanding context and culture – we have a large field staff working across India doing just this. The findings of an RCT can offer a clear picture of whether something is effective or not – but we also put in a lot of behind-the-scenes work to establish why we got that result. Ultimately, the impact estimate is not very useful if we cannot understand the underlying levers.

Classic issues involve delays in payments and in introducing benefits. This is low-hanging fruit for whoever is responsible for the policy and can be very easy for us to identify during the RCT process provided we have the right methodology. It is why we seek to understand and unpack preferences, analyse limitations, and conduct interviews alongside our quantitative data collection.

‍

Is it possible to transfer insights from one country to another? What should practitioners be wary of when considering RCTs from a different context?

We have conducted more than 1,640 randomised evaluations worldwide across J-PAL’s seven regional offices and the 95 countries we work in. We also offer a teaching course that can guide practitioners to apply these principles, which allows us to spread the tradecraft behind it and hopefully help even more people. When we are talking about translating research into action, achieving scale is important to us and we want to shape a culture of evidence-informed decision-making.

We have had a lot of success in transferring findings from one city or country to another. That said, it is important to note that we do not always expect the size of the impact and the cost-effectiveness of the programmes to translate directly into new contexts. We focus on mechanisms and general lessons about human behaviour to judge whether a programme is likely to work in a new setting.

‍

How do you approach helping governments?

When we work with governments, we consider what they really require and how they could adopt a more evidence-informed approach. We share insights into what works and what does not, collaborate to scale insights that are effective, and, more importantly, bridge the gap between research and practice. We also partner with governments to help build internal monitoring and evaluation capacities as it is ultimately they who must implement and keep track of the programmes. We are not management consultants, we are in it for the long term. We have been working with the Government of Gujarat and Government of Tamil Nadu in India for almost a decade now. We want to help governments build the systems and processes for ongoing success.

Governments often approach us to share their plans for a certain programme and ask us to evaluate it using a suitable RCT. Given the complexities of social policy, this often evolves into a multi-department or multi-study relationship. One good example comes from Tamil Nadu where we were asked to evaluate the pension scheme. We realised that there were many factors at play.

We helped the Department of Statistics and the Department of Health conduct a large longitudinal survey where we trained government officers using tablet computers, and not the traditional pen and paper, for this massive census. It found that less than 16% of the eligible households were getting elderly pensions. The Department of Health took anthropometric measures, such as blood tests, and found a high incidence of non-communicable diseases, such as hypertension and diabetes, and very acute levels of depression and loneliness. Many were unaware that they had cataracts, for instance.

The Government is now tracking 5,000 households over seven to eight years and enrolling more elderly people into the pension scheme in each wave. It has also enacted a range of policies to address the deficiencies that were found, that we are in turn evaluating through RCTs. It is wonderful when we can come in to support governments, grapple with the details, and leave a lasting impact.

‍

Many countries establish technical training institutes to equip employees with relevant workforce skills. Can RCTs help governments to ensure that technical training is designed and delivered effectively?

We have done more than 700 randomised evaluations specifically related to skills and training programmes around the world. We have tested training programmes of varied duration, with different target populations, and various designs, content, and cost. We have analysed the supply and demand sides, from the perspective of trainers and trainees, to determine what is most cost-effective, and the downstream impact of a programme. An analysis of 28 evaluations of technical and vocational education and training programmes showed mixed effectiveness – there is no silver bullet and no one model that works.

‍

We have established that training programmes increased people’s employment rates modestly and increased earnings. The most effective model is to include an internship or apprenticeship, which leads to better job placement and higher earnings thereafter.

One element we have observed generally is that there is a huge mismatch between what employers want and what job seekers have in terms of qualifications, skills, and aspirations. Another barrier concerns social networks, which can work to keep people in the same kinds of jobs. How do you break out of this and access new opportunities? A big focus of our research now is assessing policies to help job seekers overcome these problems.

Discrimination against women in many skilling programmes is also observed. Some barriers include their home environments and orthodox views on gender roles that can lead to low participation rates. Unless there is an empowerment programme or a soft skills training element specific to women, the outcomes are usually lower. This translates to lower earnings, discrimination within the industry, lower likelihood of obtaining a formal work certificate, and lower bargaining power. In general, we have seen that programmes designed specifically to support women, factoring their constraints into programme design, have been found to be more beneficial.