Moving from Data to Evidence in Education Innovation

Evidence generation in education innovation has presented unique opportunities for EdTech founders to shift their focus from solely scaling to answering the critical question: “How can we determine if learning is taking place?” This emphasises the need for evidence-driven generation approaches prioritising educational outcomes over metrics alone. The North Star lies in understanding what constitutes evidence in education innovation and knowing how to track and measure it effectively. 

On the 24th of October 2024, Injini hosted an EdTech Roundtable in Cape Town. This gathering brought together key stakeholders to explore how data can advance education in the Western Cape. Panellists included Jonathan Clark, educational consultant at JET Education Services, Lyndsey Petro from Grow ECD, Ian de Vega from the Western Cape Education Department, and Alexei McGregor, MEL Head at Injini. The discussion revolved around identifying the types of evidence the government, schools, and other education stakeholders need to integrate EdTech solutions into schools and the key indicators EdTech founders must track to generate meaningful evidence.

The Evidence Gap in Education Innovation 

The World Economic Forum reported that one of the challenges in education innovation is the disparities in standards for evaluating learning evidence between the Global North and the Global South. Measuring evidence of what solutions improve learning outcomes usually involves using randomised controlled trials (RCTs), and the RCT standards used in developed countries are not always applicable in developing countries, especially for solutions designed for a specific social and educational context. 

In addition, the evidence gaps are also due to the lack of evidence-generation standards in the education sector. The healthcare sector in South Africa has standards in place to ensure that impact is measured effectively; these include patient rights, clinical support services, patient safety, clinical governance and care, and many more (Department of Health). Ian De Vega, who has previously worked for the Western Cape Department of Health, indicated that the healthcare standards in the Department of Health make it easy to evaluate if the initiatives put in place are effective. Similarly to other sectors, there need to be standards in the education innovation sector to measure the impact effectively and fill the evidence gap. 

The evidence generation gap is not unique to the South African context; Alexei McGregor highlighted that the United Kingdom is working towards appointing an EdTech Evidence Board, which will be responsible for testing education products’ quality against a set of criteria. This indicates a worldwide gap in evaluating the effectiveness of EdTech products (Fittes, 2024). This has become even more of a challenge in the South African context, where the education system is complex and requires a nuanced approach incorporating dynamic data collection methods so that the data speaks to each need. Evidence generation should be grounded in the realities of the local context, offering a genuine picture of what works—and what does not. 

Understanding what works, for whom, and under what conditions is a step towards filling the gap. It is crucial to develop evidence-generation standards that focus on the type of evidence generated and how it is gathered (World Economic Forum). This move will require policymakers to adopt “methodological purity,” which entails methodological and ethical transparency in evaluating existing EdTech products. 

Ian De Vega emphasised that improving the foundations of learning is one of the core priorities for the Western Cape Department of Education. This ambition is linked to the 2030 National Strategy For Early Childhood Development Programmes in South Africa. Reaching the goal of improving foundations of learning requires impact measurements that can assist in tracking what works in early childhood development. Lyndsey Petro explained that Grow ECD is working towards this goal by incorporating Center Assessment Tools and Learner Assessment Tools that help track their solution’s impact. Through these different tools, Grow ECD can evaluate what type of support is required for both the centre and the learners they serve. 

Dr Jonathan Clark indicated that the potential for negative impact is an often overlooked aspect. There is a risk in continuing interventions even when evidence suggests they are ineffective.  Amongst other things, products can generate a negative impact when the product is distracting to learners and confusing to teachers (Loble, 2024). Founders must resist the temptation to inflate their impact for stakeholder approval. Metrics like user numbers alone do not equate to improved learning outcomes, and overstating impact undermines trust and long-term success. The impact of the solution starts from the usability and accessibility of the product for learners and teachers.

What Evidence Should Be Generated, and for Whom?

EdTech founders face the challenge of demonstrating their solutions' impact to diverse stakeholders, including investors, government, board members, and users. Each stakeholder has distinct objectives and expectations for evidence. Rather than viewing this diversity as an obstacle, founders can create a cross-cutting set of indicators adaptable to various audiences. 

The top metrics that EdTech founders need to measure constantly are outlined by (Post, 2023), with the following questions to keep in mind: 

• Usage metrics: how often and when are users engaging with the product?
• Retention rates: how many users are continuing to use the platform and why:
• Are users meeting learning goals, and how can your EdTech solution improve learning outcomes?
• How quickly and widely are users adopting the EdTech product?
• How much learning progress are users making, and what is the long-term impact of EdTech implementation?
• How well is your product meeting the needs of different user groups? 
• What is the return on investment for your implementation?

Understanding each stakeholder's pain points is crucial when generating evidence for particular stakeholders. The information below highlights some of the key pain points of different stakeholders.  

1. Users are interested in the innovation of the product and solutions that will assist in improving learning outcomes. Whether the end-user is a school teacher or learner, their main goal is improving learning outcomes (Loble, 2024). ‍
2. Impact investors are looking for solutions that have been tested, evaluated, and designed with the problem in mind. As a result, they are looking for solutions with users who have already bought into the concept. It is also essential for the data to speak not only to the impact but also to user engagement and frequency. 
3. The Western Cape Department of Education is interested in how the intervention improves learning outcomes and understanding of what is happening on the ground, such as learner attendance. There should be a feedback loop for monitoring and evaluating the initiatives on the ground to assist in influencing policy and filling the existing gaps. 

The Path Forward

Moving from data to evidence in education innovation is a technical challenge and a strategic imperative. By embracing a localised, realistic approach to evidence generation and aligning it with stakeholders’ needs, EdTech founders can create solutions that genuinely make an impact. While this is the goal, there is still a need to develop standards for evaluating EdTech products in South Africa and beyond. It is essential for policymakers, education stakeholders, and EdTech innovators to work towards generating standards that can be applicable in the African education context. 

The panellists emphasised the importance of EdTech founders immersing themselves in the social context and staying attuned to the evolving landscape of African education innovation. EdTech founders must continuously revisit their Theory of Change to maintain a clear focus on the core problem their solution aims to address and the specific stakeholders it seeks to serve. This iterative approach ensures that their solutions remain relevant, impactful, and responsive to the unique challenges of the African education system.

References

Department of Health. 2011. National core standards for health establishments in South Africa. Available: https://static.pmg.org.za/docs/120215abridge_0.pdf [2024, December 10].

Fittes, E.K. 2024. How the UK plans to assess EdTech products and train teachers in AI. Available: https://marketbrief.edweek.org/regulation-policy/how-the-uk-plans-to-assess-ed-tech-products-and-train-teachers-in-ai/2024/06 [2024, December 10]. 

Loble, M, 2024. Evidence in EdTech: Matters of importance. The Technology Horizons in Education. Available: https://thejournal.com/Articles/2024/11/14/Evidence-in-Ed-Tech-A-Matter-of-Importance.aspx?Page=1 [2024, December 11].

Post, T. 2023. Top 8 EdTech metrics you should be monitoring (and why). Available: https://dashboardfox.com/blog/top-8-ed-tech-metrics-you-should-be-monitoring-and-why/ [2024, December 11]. 

World Economic Forum. 2023. Why EdTech must prioritise research and innovation from the Global South, May 2023. Available: https://www.weforum.org/stories/2023/05/to-advance-edtech-evidence-we-must-prioritise-research-and-innovation-from-the-global-south/ [2024, November 18]. 

‍