Using Responsive Feedback in Scaling a Gender Norms-Shifting Adolescent Sexual and Reproductive Health Intervention in the Democratic Republic of Congo

INTRODUCTION

Public health interventions are designed and implemented in complex and dynamic systems across cultures and time. A responsive and responsible implementation strategy will seek to maximize improvements in the lives of people for whom the interventions are designed. However, dominant models of global health intervention programs do not account or allow for the complexity and unpredictability inherent in implementation in dynamic environments.¹ This is in part due to donor-driven metrics and timelines, an overreliance on randomized control trials as the gold standard for evaluation, and the weakness of clinical empirical models to assess social change efforts.² In recent decades, numerous thought leaders and organizations have recognized the limitations of the standard development paradigm and have called for new strategies that engage global and local stakeholders via timely data use and other learning strategies to systematically adapt and improve programs.^1,3,4 Stemming from these emerging perspectives, a number of complementary approaches have been established to guide implementers in how best to respond and adapt to complexity.

One such approach, responsive feedback (RF), is a systematic process for linking ongoing implementation learnings to modifications in project design that promotes reflection and discussion between project designers, implementers, researchers, and decision-makers to provide timely assessments and actionable feedback, thereby enabling programmatic course corrections needed to achieve intended outcomes and maximize health improvements.¹ Another similar framework known as Collaboration, Learning, and Adapting calls for using a set of tools developed by the U.S. Agency for International Development and their implementation partners to aid in learning and adapting throughout the project life cycle.³ Common elements to the RF and Collaboration, Learning, and Adapting frameworks include the following program characteristics: (1) an agile and flexible design; (2) adaptive to context and situation; (3) openness to test and change; (4) inclusive of and responsive to needs of implementers and decision-makers; (5) use of actionable data.¹ The capacity and capability to gather evidence to fill gaps in knowledge is foundational to the RF approach.⁵ The learning agenda that guides evidence generation is set by a program's theory of change (TOC), which helps program implementers and researchers prioritize “what” to observe and “when” to observe by explicitly documenting assumptions in cause-effect relationships between activities and outcomes.^1,6 This allows implementing staff, researchers, and others involved in the program to critically appraise these assumptions, with support from responsive feedback mechanisms (RFMs)—tools that support the learning process and adaptive thinking.¹

Despite growing recognition of the value of using RF loops to enable course corrections, published evidence documenting the benefits and strategies used to successfully apply RFMs is scant. Viswanath et al. discuss the multiple challenges associated with applying RFMs, including organizational structure and culture, insufficient resources, and lack of capacity.¹ However, they also assert that these barriers can be overcome with sufficient stakeholder commitment. Use of feedback during the design phase of a pilot to develop a digital service business for Nigerian patent and proprietary medicine vendors is explained by Wright et al.'s article highlighting the importance of a TOC to identify pathways and markers of course correction.⁷

The use of RF is also critical to the successful scale-up of proven interventions. Although many definitions of scale-up exist, the World Health Organization/ExpandNet Consortium describes scale-up as “deliberate efforts to increase the impact of…innovations successfully tested in pilot or experimental projects so as to benefit more people and to foster policy and program development on a lasting basis.”⁸ To successfully scale a proven pilot program, several requirements must be met. First, scale-up depends on adaptation, as it is impossible to reproduce any activity or program without accommodating the contexts to which it will be spread or in which it will be institutionalized.⁹ This adaptation must occur while retaining fidelity to outcomes and “core” program mechanisms. However, the process for achieving these outcomes may change. A second requirement is that of adaptive management—a style of activity management that emphasizes continuous collection of information to flag needed improvements and facilitate adaptation.¹⁰ The data collection and monitoring approaches needed for scale-up must provide information that yields a real-time dynamic picture of progress to guide efforts.¹¹ During the scale-up period, the focus is on measuring processes to ensure the innovation is implemented with fidelity at an acceptable pace and achieves desired coverage.¹² A final requirement is an awareness of complexity—at the contextual, temporal, and interpretive levels—to consider how the environment and implementation process shape intervention outcomes; how interventions evolve over time due to new understandings, constraints, opportunities, and priorities; and the many viewpoints and perspectives held by the various stakeholders.¹³ Each of these requirements—adaptation, adaptive management, and complexity awareness—necessitates the application of RF approaches.

We document the RF approaches applied from piloting through scale-up and the corresponding results from the Growing Up GREAT! (GUG) intervention (known locally as Bien Grandir!) in Kinshasa, Democratic Republic of the Congo (DRC). In this article, we describe the RFMs used to adapt, pilot, and scale this gender norms–shifting intervention for very young adolescents (VYAs) aged 10–14 years over 6 years; discuss results of the application of RFMs to implementation; and draw conclusions about the value of applying RF to program adaptation and scale-up efforts.

GROWING UP GREAT! INTERVENTION DESCRIPTION

A social norms exploration conducted with local staff and stakeholders identified a related set of inequitable norms linked to girls' education, chore burden, and free time at home; access to adolescent sexual and reproductive health (ASRH) services; and violence. GUG was designed as a scalable gender norms–shifting¹⁴ approach to foster a critical examination of these local gender norms, increase ASRH knowledge and assets, and cultivate gender-equitable and nonviolent attitudes and behaviors among VYAs. From September 2017 to May 2018, Save the Children, in partnership with 8 local community-based organizations (CBOs), implemented the multilevel intervention by engaging VYAs, their caregivers, communities, teachers, and health providers (Figure 1). More than 2,300 VYAs participated in school and community-based adolescent groups that used a toolkit with interactive, age-appropriate materials to learn about ASRH and explore related social norms during weekly mixed-sex group sessions. More than 2,000 parents and caregivers participated in 6 hour-long reflective group discussions prompted by testimonial videos featuring positive norms-driven behaviors. Additionally, more than 2,200 community members, including traditional and religious leaders, joined discussions about ASRH and norms as part of the community game during community-wide sessions. At the education systems level, teachers received training to integrate GUG materials into the national Family Life Education (FLE) curriculum; this reinforced information for school club members but also reached VYAs who were not members of school clubs. Importantly, it also provided a path to scale-up and sustainability within government systems. At the health systems level, facility-based health providers received training to provide adolescent-friendly services and hosted facility visits for VYAs.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 1

Growing Up GREAT! Theory of Change

Abbreviations: ASRH, adolescent sexual and reproductive health; FP, family planning; GBV, gender-based violence; GEAS, Global Early Adolescent Study; GUG, Growing Up GREAT!; IPV, intimate partner violence; IS, in-school; MOE, Ministry of Education; OOS, out-of-school; SRH, sexual and reproductive health; VYA, very young adolescents.

^a Outcomes not measured by the GEAS.

^b Outcomes fully measured by the GEAS.

^c Outcomes partially measured by the GEAS.

Since the beginning of the 7-year project life cycle (2016–2023), GUG has also engaged researchers, scale-up experts, and implementation scientists to (1) build the evidence base on VYA-specific outcomes, including the formation of gendered attitudes via longitudinal data from a 5-year cohort study as part of the larger Global Early Adolescent Study^15,16; and (2) institutionalize locally prioritized intervention elements within Congolese policies, systems, and structures. Over the final 3 years of the project life cycle, scale-up efforts accelerated, resulting in vertical scale-up achievements, such as integrating GUG into the FLE in-service training for teachers and including GUG programmatic elements into the Ministry of Health (MOH)'s National Program for Adolescent Health (Programme National de Santé de l'Adolescent) 5-year strategic plan (2021–2025).¹⁷ The project is described in greater depth elsewhere.¹⁸

ENABLING CONDITIONS FOR RESPONSIVE FEEDBACK

From project inception, the GUG team purposefully established values, systems, and processes to support a learning culture. This culture was grounded in systems thinking, which recognized the complex, interconnected social and institutional factors that influenced project activities and outcomes.

RESPONSIVE FEEDBACK MECHANISMS

Building on stakeholder engagement, a learning culture, and the TOC, the GUG team applied several RFMs during the project life cycle. We used simple and coordinated monitoring tools, developed a learning agenda, developed a learning matrix to document lessons learned, and commissioned rapid research on specific intervention elements. During scale-up, we adapted the World Health Organization/ExpandNet's Implementation Mapping Tool (IMT)²² to support and document learning and adaptation related to institutionalization.

RFMs were operationalized in each phase of the project life cycle to maximize opportunities for learning (Figure 2). Phase 1 of the program (2016–2017) included rapid formative research, drawing on workshops with youth, parents, and Ministry stakeholders to adapt a previously validated intervention (albeit from a rural Ugandan context) to urban Kinshasa. Phase 2 (March–August 2017) was known as the learning lab, in which a rapid pilot test occurred to determine how to adjust the intervention before implementation. During this 6-month phase, regular activity monitoring began, and learning meetings were initiated. These meetings brought together implementers to review available data, discuss observations, and recommend adaptations and improvements as necessary. Phase 3 (2017–2018) was the official pilot, conducted in 2 communities (Masina and Kimbanseke) and assessed by a pre- and post-test control group design outcome evaluation as part of the Global Early Adolescent Study.¹⁵ Project monitoring and documentation efforts that started during the learning lab (Phase 2) were strengthened and continued during this phase. Implementation learning and evaluation findings pointed to a handful of needed revisions to the intervention, especially the video discussion guides for caregiver and community sessions. The guides were revised to include more content on open and respectful communication with VYAs and nonviolent discipline, a topic frequently requested by participants during implementation. Phase 4—preparing to scale—began at project inception and was informed by all other phases. A preliminary scale-up plan was developed during a workshop in August 2016 in Kinshasa. This plan focused on generating credible, actionable evidence; engaging stakeholders; and developing a scalable intervention. This and subsequent workshops deepened scale-up understanding and planning and built the project team's ability to troubleshoot problems as they arose. After the pilot (Phase 3), the team held a workshop to review evaluation results and discuss implementation experience, informed in part by monitoring data and documentation. The goal of these data-informed discussions was to decide whether the results justified further implementation and scale and to determine whether the intervention met the CORRECT (Credible, Observable, Relevant, Relative advantage, Easy to install, Compatible, Testable) criteria for scale-up.²³ The decision was made to continue to the scale-up phase, although learnings from the workshop prompted a number of changes to the intervention to ensure maximum potential for integration and sustainability. Thus, the implementation learning was important not only for adjusting the intervention during Phase 3 (pilot) but also for informing adaptation before planned scale-up, which began in 2019.

• Download figure
• Open in new tab
• Download powerpoint

FIGURE 2

Timeline of Growing Up GREAT! Implementation

RESULTANT PROGRAMMATIC ADAPTATIONS

Development of a learning culture and application of RFMs resulted in concrete program adjustments that enhanced the effectiveness and scalability of GUG. These changes included increased program acceptability within the community and intervention sites; expansion to include engagement of male caregivers; revisions to implementation materials; improved facilitation quality; and the ability to respond rapidly to changes in the environment.

KEY LEARNINGS IN USE OF RESPONSIVE FEEDBACK

This case study from an ASRH program in Kinshasa provides important learning about how RF can be established and cultivated for the pilot to scale-up life cycle of a community-based health program. We found that creating a culture of learning that embraces and learns from failure starts with staff who model this approach. Our core team internalized this culture and created open, transparent, and 2-way communication with local partners and stakeholders. This process required both an initial investment and continued support.

Initial investment centered on establishing strong processes and structures for meeting regularly to examine and apply learning. Identifying and engaging key stakeholders—including implementing partners, parents, teachers/school staff, and adolescents via the Youth Advisory Council—to participate in these processes was also important. Implementing partners were unaccustomed to playing such a central and trusted role informing program evolution and required coaching and encouragement following initial training and learning meetings. We used some of the same critical reflection and discussion techniques used in the intervention to encourage sharing and spent time reviewing actions that resulted from partner observations so they could see how their input was being applied practically. We also used their feedback to improve RF tools and systems (like monitoring processes) along the way. This commitment to including implementer perspectives and participation in RF was not only an important opportunity for capacity-building but also motivated partner engagement and ownership.

Each RFM had different advantages and challenges. Application of more formal research, such as the outcome evaluation results, to improve program implementation was difficult due to the length of time research firms needed to collect, analyze, and share the data. Similarly, although the learning studies were meant to be rapid to provide real-time data, in practice, the time from conceptualizing the study to sharing the results for application was too slow to be maximally useful. Nevertheless, the GUG project team prioritized sharing results from the outcome evaluation, Global Early Adolescent Study cohort survey, and learning studies among stakeholders. This provided additional opportunities to engage in dialogue with stakeholders and provided important confirmation of issues emerging from monitoring and practice-based data sources. Confirmation with the research study data was critical because, although the monitoring data were available almost immediately to stakeholders, challenges with initial data collection and manual compilation in some cases resulted in data quality issues, leaving questions about the monitoring data's reliability. One example of this was data on exposure to the intervention, which was reported as very low by the outcome evaluation. By contrast, self-reported monitoring data from CBOs showed high and consistent participation throughout the intervention; a subsequent data quality audit confirmed that evaluation measures were likely accurate. In addition, implementer observations were readily available and the most detailed source of information, but they were not always generalizable to all intervention zones.

Finally, RF itself must remain flexible and change over the program cycle. Certain aspects of our RF approach, such as learning meetings and our commitment to collecting and triangulating multiple sources of data, were consistent throughout all phases of program implementation, but others changed along with program needs. Monitoring tools such as the quality benchmarks were added to existing tools as a need arose for more detailed data on facilitation quality, and learning tools changed as well. We traded our simple learning matrix for the more systematic IMT as we entered the scale-up phase. Monitoring tools and indicators were also simplified for scale-up and a new set of benchmarks developed to assess scale-up progress.

In the future, we would advise improving monitoring by tracking participation of youth and their parents in different program components to allow linked, real-time analysis of intervention depth and coverage. This would require a unique code for each child-parent dyad and a tracking system such as stickers, bar codes, or a participation card. This information would enable analysis of the effect of intervention fidelity on outcomes. In addition, given the issues with the quality of monitoring data and the burden collecting this data puts on program staff, we recommend adopting automated approaches for rapidly gathering, collating, and presenting monitoring data, such as using tablets or digital approaches through interactive voice response programs or smartphones (resources permitting). Finally, to understand and improve multilevel norms-shifting interventions, such as GUG, monitoring should be strengthened at all levels of the socioecological system implicated in the program's TOC. In the case of GUG, that would mean gathering monitoring data from caregivers and children and tracking the diffusion of new ideas to the broader community.

We hasten to note that donor support and relationships were critical in allowing us to pursue our RF approach. GUG was supported by 2 donors equally committed to the program vision, who worked collaboratively to provide technical and financial resources. They placed a high amount of trust in our ability to document learning and implement adaptations for increased impact and provided us with latitude to experiment, fail, and learn to improve. Their commitment to RF and adaptive programming were vital to our success.

Programmatic adjustments that resulted from RF are documented in the GUG Implementation Guide—a step-by-step resource for organizations that wish to adopt the intervention.¹⁸ It provides users with guidance; tested tools; and materials for planning, implementing, supervising, and monitoring this successful norms-shifting intervention.

CONCLUSIONS

The GUG team of implementers, researchers, donors, and stakeholders realized that success was more likely if the team developed a learning culture, worked closely with stakeholders to develop consensus on a TOC, and gathered and applied information to implementation decisions. RF resulted in an environment where stakeholders had access to the data, skills, and processes to determine whether GUG met scalability criteria, as well as provide critical insight into how to improve the approach for scale. The GUG team collected, analyzed, interpreted, and applied multiple sources of information. This process of triangulation was fundamental to decision-making. Many implementation challenges were first identified by implementers. Their early feedback was almost always confirmed by other sources of data, such as monitoring data or rapid study results. In several instances, we would not have understood the full context of the quantitative data without the accompanying qualitative or “soft” data (e.g., anecdotes). The ability to confirm emerging learnings from multiple perspectives and data sources made it easier to make informed programmatic decisions.

As this case study demonstrates, RF approaches are vital to norms-shifting programs such as GUG, which seek to catalyze change within complex, dynamic systems. In the GUG experience, RF approaches were critical to actively monitor and adjust norms-shifting program processes and effects without delay. We paid close attention to how new ideas and information spread or diffused through communities, what actions and reactions were occurring within communities, how change happened, and which actors were involved. As compared to traditional research efforts, these RFMs supported a more immediate identification of and responses to both the expected and unexpected consequences of program activities. These systematic and timely course corrections ensured programmatic feasibility and acceptability in Kinshasa and helped foster policy support for scale-up efforts and project sustainability. The lessons learned from this project can be extended to other country settings.