Lessons Learned From Implementing Prospective, Multicountry Mixed-Methods Evaluations for Gavi and the Global Fund

Lesson 1: Include an Inception Phase to Engage Stakeholders

For multistakeholder evaluations of complex interventions, the evaluation team and donors should include an inception phase to focus on stakeholder engagement and evaluation design. Support by high-ranking government officials and donor organizations during the inception phase can facilitate early stakeholder engagement.

The FCE and PCE were each designed with an inception phase of 4 and 6 months, respectively. Given the complex nature of the evaluations, the inception phases were crucial to have dedicated time for a consultative and collaborative approach to engage stakeholders in developing a comprehensive understanding of the programs to be evaluated and refining the evaluation priorities and approach. Engaging stakeholders can improve evaluation design and relevance, facilitate data collection, and increase the likelihood that evaluation findings are used.^18–20

In the inception phases, we first relied on CEPs’ knowledge of the local context, reviews of relevant technical documents, and the organizational structure of key institutions to identify relevant stakeholders. We held face-to-face meetings with individuals or small groups of stakeholders to introduce the evaluation, which was essential to get buy-in from key government officials. These meetings were followed by half- or full-day kick-off meetings in each country with a wide range of stakeholders representing ministries of health and finance, implementing partners, technical partners, civil society, and Gavi/Global Fund. In many cases, the kick-off meetings were attended or endorsed by high-ranking stakeholders such as senior government officials who encouraged support for the evaluation. For example, the Permanent Secretaries of the Ministry of Health opened the FCE inception phase stakeholder meetings in Uganda and Zambia and the Minister of Health in Senegal presided over the opening ceremony of the PCE. Support from high-ranking officials paved the way for easier access to other government officials and partners for evaluation data collection and contributed to the sense of legitimacy of the evaluation, thereby improving the likelihood that findings would be used.

Support from funders also helped facilitate support for the evaluation. At the outset, a formal letter from the Gavi CEO was shared with country governments to endorse the FCE. During the inception phase, the Gavi M&E team and the evaluation team jointly met with key Ministry of Health personnel and other stakeholders, signaling Gavi’s support for the evaluation. The PCE did not have the same level of engagement from the Global Fund Secretariat in the inception phase, in part because the PCE was commissioned independently by the TERG. The limited early engagement from the Global Fund Secretariat resulted in early challenges for stakeholder buy-in, with downstream consequences in terms of accessing information, aligning the evaluation findings with decision making timelines at the Secretariat level, and ensuring widespread dissemination and use of synthesis findings.

Lesson 2: Align on the Approach to Embedding the Evaluation in the Program Implementation

In a prospective process evaluation, the donor and evaluation team should align on the degree to which the evaluation is embedded in the program implementation; a quasi-embedded approach can balance objectivity and learning. Expectations for program stakeholders’ engagement in the evaluation should be clearly communicated by the evaluation team.

The continuum of potential evaluation designs ranges from a purely external evaluation that is entirely independent of the program implementation to a fully embedded evaluation that is internal to the implementation team.²¹ Purely external approaches may be more objective, but they have limited ability to understand changing program implementation, thereby potentially limiting the usefulness of the evaluation. A more embedded approach allows for more collaboration and feedback loops between the evaluation and program teams to adapt the evaluation to shifts in context, programs, or priorities.²¹ In evaluations of complex adaptive program implementation approaches, some degree of embeddedness to understand these shifts is appropriate. Process evaluation in particular requires collaborative and trusting relationships with stakeholders involved in the program implementation to facilitate access to information.⁴ And, as noted in lesson 1, engaging stakeholders in the evaluation can encourage uptake of evaluation findings.^18–20 The FCE and PCE took a quasi-embedded approach to preserve evaluation objectivity while collaborating closely with stakeholders to support evaluation relevance, data access, and use of findings.

The quasi-embedded approach can encourage timely learning through feedback loops between the evaluation teams and programs—and messaging the evaluation in this way, as a “learning platform,” helped increase stakeholder buy-in. During the initiation of the FCE and PCE, there were concerns that country stakeholders who were the most familiar with independent outcome evaluations would be resistant to the evaluation if they felt like they were being audited. Gavi anticipated this concern and emphasized to stakeholders that the FCE was not an evaluation of country programs per se, but of Gavi’s policies and processes, which helped to increase stakeholder buy-in. For the PCE, we shifted our framing to explain the evaluation as a learning platform that could provide support to stakeholders, help answer their priority evaluation questions, and provide evidence or recommendations to improve their program implementation. Many PCE stakeholders were initially unfamiliar with or had never engaged with a prospective evaluation in practice, so the concept of a learning platform was more intuitive. This framing also facilitated buy-in by differentiating the PCE from past Global Fund evaluations that some country stakeholders perceived as top-down audits rather than learning opportunities. The learning platform positioned CEPs and country stakeholders as partners in learning and opened the door for a collaborative relationship. This approach is in line with calls for more participatory and collaborative models for learning and evaluation in the international development field.^22–24

This initial messaging of the FCE and PCE as partnerships focused on learning set the stage for a more collaborative relationship between evaluators and program implementers. The CEPs established close relationships with country stakeholders who were able to share documents and data, extend meeting invitations, and serve as key informants. Over time, the CEPs have become increasingly embedded in country programs, for example, being added to standing program meeting invites, which also meant they were on the email distribution list to receive meeting minutes and other key documents. While this involvement has enabled CEPs to track the unfolding processes in real time, gain access to essential documents and data sources, and share back emerging findings to improve program implementation, it has also made it challenging for CEPs to maintain evaluation independence (or a perception of independence). In some contexts, CEPs joining meetings solely as observers was not acceptable—they were expected to contribute if they wanted to keep being invited—thus, they shifted into participant observers.²⁵ The Zambia FCE team highlighted their approach to provide meeting notes as a way of adding value,¹⁵ and across all CEPs, evaluator reflexivity was used to balance independence and embeddedness.^11,26–28 Over time, the CEPs and stakeholders established shared expectations for engagement. This establishment of shared expectations—between evaluators and stakeholders, as well as evaluators and donors—should be discussed at the outset and revisited throughout the evaluation life cycle.

Although the FCE and PCE have used a quasi-embedded approach to balance objectivity and learning, there has been an ongoing tension in how to strike this balance in ensuring use of findings. Evidence uptake and knowledge translation rarely occur spontaneously and must be supported through a combination of “push,” “pull,” and “exchange” activities.²⁹ As a result of the messaging of the evaluation as a learning platform and the embeddedness of evaluators, the evaluation team was perceived as being well positioned to engage stakeholders in these knowledge translation activities; however, encouraging the uptake of recommendations also risked compromising evaluation independence. To create accountability with stakeholders for acting on evaluation findings, while preserving evaluator independence, the Gavi Alliance provided an annual “management response” to the FCE findings and recommendations. The management response reported how Gavi had used each finding/recommendation.³⁰ This approach could potentially be expanded to PCE country stakeholders or the Global Fund Secretariat to create more accountability for the use of findings, while maintaining independence of the evaluators. Ultimately, it is critical for the evaluation team and donor to align on the nature of collaboration and the role of the evaluation team at the outset of the evaluation because it has implications for how evaluators are received and how data are collected, as well as the adoption of findings.

Lesson 3: Continuously Monitor Changes and Adapt the Evaluation

In evaluations of complex interventions in which the programs, organizations, or contexts are constantly evolving, the evaluation team needs to continuously monitor changes and adapt the evaluation. The evaluation plan should be designed with enough flexibility to adjust evaluation questions and approaches to respond to changes; to support this, buy-in from the donor organization is essential.

During the inception phases, the evaluation teams directly engaged the intended users of the evaluation to inform the evaluation focus, priorities, and evaluation questions (consistent with evaluation best practices^20,21). The initial terms of reference for both the FCE and PCE provided overarching strategic evaluation questions, which the evaluation teams translated into a list of country-specific and cross-country evaluation questions responsive to the organizational context during the inception phase. However, as learning institutions, Gavi and the Global Fund frequently update processes and policies, which may affect evaluation context, objectives, and priorities throughout the course of the evaluation. (For example, Global Fund’s Operational Policy Manual³¹ undergoes numerous revisions throughout the year.) Thus, understanding the program and designing a responsive evaluation was not limited to the inception phase but required an ongoing assessment as to how the program was evolving, more consistent with developmental evaluation approaches.^32,33

The quasi-embedded approach of the CEPs facilitated program monitoring at the country level, as did collaborative relationships with the Gavi M&E team and Global Fund TERG at the global level. Table 1 summarizes the approaches taken by the FCE/PCE teams to maintain direct access to stakeholders who could provide insights on the changes to Gavi/Global Fund policies and processes. Weekly calls with the Gavi M&E team and TERG Secretariat were helpful to regularly solicit updates related to policies, processes, or strategies, and buy-in from the donor organization is critical in supporting the evaluation team to fully engage with its staff (e.g., the Gavi Secretariat and Global Fund Secretariat).

As context and priorities shifted throughout the course of the evaluation, the evaluation questions had to be updated to reflect these changes, identify emerging questions the evaluation could help to address, and ensure evaluation questions are useful. Buy-in from and engagement of the donor organization and other Secretariat staff was essential to ensure relevance of the updated evaluation questions, and the approaches summarized in Table 1 served as an opportunity to validate revised evaluation questions. This ongoing monitoring of the program context and discussion of priorities resulted in the adaptation and revision of evaluation questions, and ultimately a more flexible evaluation design. Two examples of how the PCE adapted evaluation questions based on shifts at the country level and global level are included in Table 2. Although it was necessary to design the evaluation to respond to the changing program context and priorities, we experienced pros and cons associated with designing the evaluation to encourage flexibility and adaptation over time (Table 3).

As we have strengthened relationships with stakeholders, and stakeholders have a better understanding of the scope of the evaluations, this process of adaptation has become more organic, with stakeholder inputs on evaluation questions shared more proactively and ad hoc. The GEPs/CEPs have also become more adept at identifying evaluation priorities through ongoing process tracking, including areas of cross-country synthesis that are most relevant in informing changes to Gavi/Global Fund policies or processes.

Lesson 4: Include People With Mixed-Methods Expertise on the Evaluation Team

To successfully mix methods in a complex evaluation, evaluation teams should ideally include individuals with experience across methods, or at minimum co-locate individuals with various methods backgrounds. Tools and approaches, such as collaborative data review meetings, root cause analyses, and Tableau dashboards, can help to bridge any divide between quantitative and qualitative methods expertise.

To foster mixed-methods analysis, we learned that our teams (GEP and CEP) worked best when team members encompassed various disciplinary and methods backgrounds, were co-located, and used collaborative approaches to data interpretation and synthesis. Without conscious attention to team composition and processes, we found that mixing of methods and paradigms was difficult to achieve.

Across the consortia, a range of staffing models were represented. Some CEPs had separate quantitative modeling and process evaluation teams, while others had integrated multidisciplinary teams. Aiming for a multidisciplinary team, preferably with multiple transdisciplinary staff that had “crossover” between methods expertise proved most successful. In cases in which CEP teams were divided methodologically, co-locating team members helped to ensure more regular full team meetings to review and triangulate emerging evidence, if not full mixing of methods.

To achieve true mixing of methods and paradigms, it is necessary to have both a well-integrated team with diverse expertise, as well as established procedures and processes for dialogue and analysis. While conducting a mixed-methods evaluation has been an ongoing challenge for some teams, the evaluations have adopted tools and approaches to help bridge the gap between quantitative and qualitative approaches. Using collaborative and interactive processes is valuable in facilitating mixed-methods analysis of the data and interpretation of findings. The PCE held joint CEP-GEP data review conference calls (approximately bimonthly) to share updated quantitative analyses, discuss data quality issues and resolutions, and identify opportunities for further triangulation with process evaluation evidence or the need for additional data collection. To further facilitate collective analysis, GEP and CEP held joint in-person analysis and report writing workshops 2 or 3 times per year, in addition to cross-country synthesis workshops at least once per year. The limitation in a more collaborative analysis process is the time and cost of engaging all evaluation partners—it is a dynamic and (potentially) nonlinear process that is best served by face-to-face interaction and may take substantive time.

In terms of tools, root cause analysis was a particularly effective analytic tool as it encouraged participants to incorporate all the available data—qualitative and quantitative—and iteratively explore hypotheses collaboratively. (Example FCE root cause analyses have been shared elsewhere.^11,15) Similarly, Tableau dashboards were a useful tool to support interpretation of quantitative data among team members with a range of quantitative data skillsets; all team members had access to the dashboards and would look at the quantitative results to generate questions for qualitative follow-up. For example, the PCE visualized quantitative data from Global Fund grant revisions to understand budgetary shifts, and then generated key informant interview questions to understand why the shifts occurred and how they were affecting implementation activities.

Finally, we also learned that mixed-methods approaches can be more intentionally incorporated by starting from the evaluation question phrasing. Over time, we shifted to evaluation questions that encouraged mixed-methods data collection and analysis, such as “whether, why, and how does X outcome occur.” For example, the Uganda FCE team asked the question: “What is the effectiveness, efficiency, and country ownership of national immunization partnerships and their contribution to program performance?” This encouraged a mixed-methods approach that included social network mapping, document review, and qualitative interviews to understand the structure and added value of the partnership working on the Gavi HPV vaccine application.³⁴

Lesson 5: Contextualize Recommendations and Clearly Communicate Strength of Evidence

In evaluating complex adaptive interventions, the heightened need for attention to feasibility and context of recommendations means evaluators should clearly communicate the strength of evidence underlying each finding and should consider engaging stakeholders in the process of refining findings and recommendations.

Existing best practices focus on enhancing credibility of conclusions by ensuring data are analyzed and systematically interpreted, findings are directly linked to evidence and informed by stakeholder standards, and resulting recommendations are contextualized and actionable.¹⁷ While the evaluator’s role is to justify the evaluation conclusions, engaging stakeholders in the process presents a potential opportunity to further contextualize the findings and facilitate evidence use.¹⁸ The FCE/PCE teams shared preliminary findings with stakeholders for review to ensure we were reporting full and accurate information. Occasionally, these reviews would motivate stakeholders to share additional evidence to be incorporated. In determining when to share emerging findings, the evaluation team must balance the opportunity to gather additional insight from stakeholder reviews with the potential risk of sharing early findings with insufficient evidence that could undermine evaluators’ credibility.

Additionally, it is important to convey the strength of evidence underlying evaluation conclusions so stakeholders trust the findings and associated recommendations. This is particularly true in a mixed-methods evaluation in which each finding relies on multiple data sources with varying quality. Moreover, in some settings stakeholders perceived findings based solely on qualitative evidence to be less rigorous than quantitative evidence. To clearly signal the strength of evidence, we developed a rubric informed by GRADE and other evidence rating systems³⁵ that rated the evidence along a 4-point scale.¹⁵ However, while the GRADE rubric considers study design and rates randomized trials highly, our scale was limited to the types of evidence used in the FCE/PCE, so randomized trials were not feasible or fit-for-purpose. Our rubric considered the extent of triangulation between data sources and the quality of the sources. Table 4 shows the strength of evidence rating used in the PCE, and Simuyemba et al.¹⁵ shared the rubric used in the FCE. Each finding was published with a rating to communicate our confidence in the conclusion, accounting for data quality and triangulation.

During the FCE, the evaluation team independently generated recommendations that were shared with global and country stakeholders. The PCE has taken the same approach, but in some PCE countries we have used annual dissemination meetings as an opportunity to iteratively refine the recommendations with stakeholders. This approach has been well received and may prove a promising practice to generate buy-in for acting on the recommendations.