Three Year RCT In Liberia evidences improved learning outcomes
The Liberian Education Advancement Program (LEAP) is a public private partnership with eight non-state actors designed to improve school management in order to bring about improved learning outcomes for students across the government school system. In parallel, the Ministry of Education commissioned a three year Randomised Control Trial (RCT) designed to study and measure whether LEAP did indeed improve outcomes. The Ministry’s ultimate vision is to transform learning for all 2,619 public primary schools across the country.
In 2016, 62% of primary school aged children were not enrolled in Liberia. Thirty-five per cent of women and 21% of men could not read a single sentence. LEAP had a singular goal, to improve learning outcomes for Liberian children. Uniquely, LEAP has thrived under two different administrations having being implemented as Partnership Schools for Liberia (PSL) under President Sirleaf and continued as LEAP under the leadership of President Weah. Strong political leadership and a clear focus on improving schools for Liberia’s children trumped the struggles signature policies often face with regime change.
In year one, the initial RCT results revealed that learning gains had increased by 60% in the programme overall and by 100% in Bridge supported schools. The metric that the Government wanted to measure, gave early indications that the programme could be very impactful. However, alongside a muted acknowledgement of improved learning, researchers raised concerns about the impact of government policy decisions and raised doubts about the long term financial sustainability of the programme. Three years on the final RCT is now complete and the results are startling for the fourth poorest country in the world.
To fully understand the learning outcomes detailed, it’s important to understand how the data was analysed by the researchers. The report analyses learning outcomes of children by answering two distinct research questions. Intention-To-Treat (ITT) estimates the treatment effect of a child being assigned to a LEAP school in 2015-16. Treatment-On-Treated (TOT) estimates the treatment effect of a child having being taught at the school. ITT includes all children originally included in the study irrespective of whether they whether they continued to attend a LEAP school, dropped out of school or migrated with their families over the three-year period. The latter group (TOT) only includes those children that spent some time at a LEAP school. The Ministry wanted to measure learning outcomes for those who had a LEAP education as opposed to those who did not and so it makes sense that we look primarily at the outcomes of children who have benefited from attending a LEAP school throughout.
The RCT results for those children that attended a LEAP school show how it could be possible for successful LEAP providers at scale to prevent the generational transfer of poverty in Liberia. Overall, the LEAP programme increased learning by 0.26 (TOT) standard deviations (SD’s) in English and 0.35 (TOT) SD’s in maths; combined this equates to more than a year of additional learning. In Bridge-supported schools student learning increased by 0.62 (TOT) SD’s; which is the equivalent of 2.5 years of additional learning. This means that children in Bridge-supported schools effectively achieved 5.5 years of schooling in just 3 years.
The improvement in learning outcomes are truly transformative. Stanford economist, Eric Hanushek believes that the long-term returns to educational improvement are significant. Mr. Hanushek estimates that even a reform plan that takes 20 years to improve by 0.5SD will increase GDP by 5% in 32 years. According to the RCT data, these LEAP students—especially if their education continues on the trajectory it is now—will radically improve the quality Liberia’s workforce and certainly bring about Hanushek’s projected gains in GDP.
Alongside learning outcomes prioritised by the government, the RCT researchers chose to prioritize two other indicators – access and sustainability. Let’s take access first. The researchers frame their data to suggest that the programme was detrimental to children’s access to school. Access is understood as pure presence in a classroom, irrespective of whether learning was taking place or teachers were present. Those characterised as being ‘denied access’ includes children who chose to leave a LEAP school and those who chose not to continue their education after finishing LEAP school by going to secondary school. At the outset of LEAP, The Liberian Ministry of Education made two policy decisions which removed students from the most overcrowded schools. First, they lengthened the school day in LEAP schools. This meant that schools which operated two shifts, one in the morning and one in the afternoon, would no longer be able to do so. Second, they limited class size to 65 students overall. This reduced the total number of students in overcrowded schools. The consequence of these government policy decisions meant that some children had to move to a ‘nearby’ school – ‘nearby’ is a variable term in Liberia. It is not unlikely that children may have dropped out if they were having to travel long distances and not learning anything once in the classroom. Families rightly have the opportunity cost to consider in such situations. However, it was not in this group that children are identified as dropping out by the RCT instead it is older students and sadly, girls. It is surprising that the authors would fail to contextualise these access findings within the Ministry’s broader policy framework.
Turning to sustainability; it should be recognised that the programme to date is largely reliant on the support of philanthropists who are committed to seeing whether the LEAP model can be an exemplar for SDG4 by both improving learning outcomes and achieving financial sustainability. LEAP can only be successful in the long term if it can be delivered within the government budget. Relying on philanthropic support does not provide the long term sustainability that the government designed the programme to achieve or that schools require. The researchers prioritise sustainability and weigh it equally alongside learning outcomes in the RCT in assessing the success of LEAP. Fortunately, the RCT reveals extremely positive news. It shows that the cost of LEAP has substantially reduced and is now being delivered for $119 per child a year; close to the government’s planned for education budget of $100. The path to sustainability is illustrated effectively by Bridge which more than doubled its student enrolment between year 1 and year 3 from 9,000 to 20,000, while its total expenditure dropped from $373 per student (excluding start up costs) in year 1 to $163 in year three. Supporting the same number of government schools, from Year 2 to Year 3 Bridge reduced program costs by 25%, and continued to deliver significantly increased learning outcomes for students. As anticipated at the outset, the programme is well on the path to sustainability. This sustainability will be realised even more quickly when the initially designed programmatic scale is reached.
The RCT has drawn much attention and its publication has been eagerly anticipated by a myriad of stakeholders; by the Government that seeks to make policy decisions based on the learning outcomes it evidences; by philanthropists keen to understand whether their support has enabled impact; by the global development community who are concerned that SDG4 will be unattainable without tested innovations that can deliver at speed and at scale; by governments who are increasingly seeing education as the route to prosperity and growth; and, of course by the providers themselves who have spent four years in some of Liberia’s most impoverished communities trying to give children a brighter future. That is a lot of pressure to sit on the shoulders on one RCT. Ultimately the question that everyone wants answered, is after all the work, funding and debate: Did LEAP significantly improve learning outcomes? The answer is, yes.
