Independent Evaluation of the OTSS+ approach to quality assure malaria service delivery

The Issue

PMI’s Impact Malaria project, worked across 13 countries to improve malaria service delivery by supporting National Malaria Control Programmes to engage with Outreach Training Support Supervision Plus (OTSS+) to improve the quality of malaria service delivery. The objective of this independent evaluation was to generate evidence and analysis on the approach effectiveness, lessons learned and best practices.
Tropical Health’s independent evaluation covered 11 countries implementing the OTSS approach, with more focused in-depth data analysis in four countries, Cameroon, Ghana, Niger and Zambia.

Our Approach

We used a theory-based evaluation design, that involved a mix of quantitative and qualitive methods to answer five evaluation questions: OTSS effectiveness, OTSS success factors, contributions from the OTSS tool digitisation, OTSS sustainability and lessons learned to date. The focus countries covered had varying OTSS exposure and experience.
The evaluation team included a global team of technical experts, with country teams established for the four focus countries. Methods included an online survey, advanced secondary quantitative data analysis, documents review, key informants interviews, health facility observation and focus group discussions.
Our team synthesised the complex data and learnings into a comprehensive final report, quantitative and qualitative manuscripts published in peer-review journal, a learning brief, and a webinar.

Our Findings

Based on our review of nearly 15 years of OTSS+ implementation, the OTSS approach appears to have provided an effective framework for continuous improvement of malaria health service delivery, with an association between return OTSS visits and improved performance found on almost all OTSS outcome indicators assessed. The main success factors of the approach were identified as: 1) relevance of the framework, 2) delivery of the package of components, 3) generation of real-time data and 4) shared accountability between supervisors and supervisees. OTSS users preferred the digital tool over the paper-based system. OTSS has been adopted by national malaria programmes in all countries studied; their participation in the approach design and regular reviews facilitated that buy-in. However, OTSS was still perceived by national stakeholders to be mainly an ‘add-on’ donor driven and resourced process and approaches to strengthen its full domestic instituationalisation should be the focus of future investments in the approach.
Moving forward, there was little evidence to suggest that it is necessary to streamline the appproach in terms of malaria services covered or capacity strengthening approaches used. Instead, operational and strategic considerations reflecting country context, should be used to make OTSS more efficient and effective. These are fully explained in the report.
The Tropical Health team’s findings were presented to the global Roll Back Malaria Partnership Case Management Working Group in April 2023 and at the ASTMH conference in October 2023.