Lessons Learned From Implementing Prospective, Multicountry Mixed-Methods Evaluations for Gavi and the Global Fund

Emily Carnahan; Nikki Gurley; Gilbert Asiimwe; Baltazar Chilundo; Herbert C. Duber; Adama Faye; Carol Kamya; Godefroid Mpanya; Shakilah Nagasha; David Phillips; Nicole Salisbury; Jessica Shearer; Katharine Shelley; for the Gavi Full Country Evaluations Consortium; and Global Fund Prospective Country Evaluation Consortium

doi:10.9745/GHSP-D-20-00126

Figures & Tables

Figures

Tables

Download figure
Open in new tab
Download powerpoint
FIGURE.
Centers for Disease Control and Prevention Framework for Evaluation in Public Health¹⁷

Tables

Figures

View popup

TABLE 1.

Approaches Taken by the Evaluation Teams to Engage With the Donor Organizations to Monitor Program Developments

Gavi Full Country Evaluation	Global Fund Prospective Country Evaluation
Weekly calls with the Gavi M&E team (GEP, CEP) KIIs with Secretariat staff throughout the year, with a concentration of KIIs during an annual in-person visit to Geneva (GEP) Semi-annual touchpoints with Gavi Senior Country Managers (GEP, CEP)	Weekly calls with the TERG Secretariat (GEP) Engagement with Secretariat staff at tri-annual TERG meetings (GEP, CEP) One-off phone calls with rotating Secretariat teams scheduled by the TERG Secretariat (GEP, CEP) Semi-annual touchpoints with Global Fund Country Teams (GEP, CEP)

Abbreviations: CEP, country evaluation partner; GEP, global evaluation partner; KII, key informant interview; M&E, monitoring and evaluation; TERG, technical evaluation reference group.

View popup

TABLE 2.

Examples of Changing Prospective Country Evaluation Questions Due to Shifts at the Country and Global Levels

Responsive to Country-Level Shift

Responsive to Global-Level Shift

In Uganda, there was an unanticipated upsurge in malaria cases in 2019, so the Prospective Country Evaluation team added an evaluation question on whether and how Global Fund policies and structures enabled the country to respond.Findings indicated that several flexible aspects of the Global Fund business model, including modifications to procurement and supply chain plans, facilitated the country’s response to the malaria upsurge.

In 2020, the Grant Portfolio Solutions team at the Global Fund requested inputs about challenges related to Global Fund monitoring and reporting processes and opportunities for improvement.The Prospective Country Evaluation was able to quickly incorporate new evaluation questions into the evaluation scope and shared cross-country findings to inform the Secretariat’s revised reporting guidance.

View popup

TABLE 3.

Pros and Cons of a Flexible Evaluation Design

Pros	Cons
Is responsive to changing stakeholder needs, thereby increasing stakeholder buy-in and the likelihood findings will be used. Has the ability to adjust to unanticipated implementation delays to refocus on the most timely, relevant evaluation questions.	Can take months to get stakeholder consensus on priorities. Requires carefully balancing those stakeholder inputs while remaining objective. May mean that evaluation teams are developing evaluation tools in parallel to prospectively tracking a process that has already started. This may undermine the planning required for intentional mixed methods approaches.

View popup

TABLE 4.

Global Fund Prospective Country Evaluation Strength of Evidence Rating

Rank	Rationale
1	The finding is supported by multiple data sources (good triangulation) that are generally of strong quality.
2	The finding is supported by multiple data sources (moderate triangulation) of lesser quality, or the finding is supported by fewer data sources of higher quality.
3	The finding is supported by few data sources (limited triangulation) of lesser quality.
4	The finding is supported by very limited evidence (single source) or by incomplete or unreliable evidence. In the context of this prospective evaluation, findings with this ranking may be preliminary or emerging, with active and ongoing data collection to follow up.