Conducting research that focuses on questions about implementing evidence-informed interventions

Questions about implementation are often concerned with how to produce expected results in a particular setting from an intervention that has been previously shown to work elsewhere.1 2 7–9 Such questions may address the fit of an intervention to new settings, how to introduce, scale-up or sustain the intervention among broader populations or address real-time operational issues that may occur in the process of implementation.7 8 IR studies that focused on scale-up and sustainability are particularly essential for understanding the large-scale impact of health interventions. However, less than 5% of 791 Implementation Research (IR) studies reviewed for this paper addressed objectives concerning issues such as scaling up (n=32) or sustainability (n=25). The BetterBirth project,38 which focused on implementing a Safe Childbirth Checklist (SCC) to promote safe childbirth practices and reduce maternal and neonatal mortality in Uttar Pradesh, India, is an example of a research project that did address questions around implementing an evidence-informed intervention. Using an iterative learning process, researchers involved in the BetterBirth project adapted the SCC intervention to operational issues that occur in the process of introducing the SCC to health workers in different settings from where it was originally tested. Over two learning phases, they added a peer-to-peer coaching strategy to support the introduction of the checklist, in part to respond to leadership challenges and health worker’s lack of motivation that were identified during an initial implementation phase. They used physician coaches to motivate health facility leaders and provide critical leadership for the implementation and used nurse coaches to motivate behavioural changes in birth attendants to use the checklist. The study was able to demonstrate large increases in the correct use of the SCC intervention over time and provide evidence for a set of implementation strategies to support the scale-up of the SCC.

Whereas the focus of most IR studies are around evaluation, key implementation questions in most settings are concerned with how to scale up or sustain an intervention within a practice area or population.49 Issues around scaleup of interventions and their sustainability occur within complex health systems that are constantly evolving.8 9 The lack of studies addressing issues on scale-up and sustainability may perhaps point to the practical difficulties in how to conduct research that addresses complexity. Indeed, the capacity of public health researchers to address questions examining complex and adaptive processes has been limited until recently.50 There are, however, examples of implementation models for describing and studying scale-up and innovations for sustaining implementation activities and outcomes from other fields that are being extended to IR in public health.51–53 For example, Aarons and colleagues describe one model for implementing public sector services and included four different implementation phases (exploration, adoption/preparation, implementation and sustainment), highlighting specific factors that are important at each phase.51 These included factors affecting the outer context (service environment, interorganisational environment, consumer support/advocacy), the inner context (intraorganisational characteristics, individual adopter characteristics), innovation characteristics and the fit with existing systems.51 There are additional methods related to systems thinking that are also useful for examining questions of implementation within complex systems.50 54

Conducting research on implementation of evidence-informed interventions under real world conditions

Whereas what constitute ‘real-world’ conditions for a set of implementation activities may be debatable, criteria such as whether implementation was led by usual implementing agencies and without additional funding (apart from usual budget) or management support for implementation during the study1 2 8 may be useful to gauge the extent to which conditions for implementing an intervention depart from normal routine. Where provision of additional resources is used as an implementation strategy, then this should be clear, and the study should indicate how such provisions will be sustained and integrated into the system. This principle—to conduct research on implementation under real world conditions—is perhaps one the most distinguishing characteristics of IR studies.2 8 10 Unfortunately, most IR studies are conducted under more controlled settings, which limits the extent to which learning from those studies could be applied to commonly found conditions.2 8–10 Our review of the literature also suggests that this critical approach may be lacking in published IR studies in LMICs. In examining management support, funding and who is doing the implementation, only 26% of IR studies reviewed did not have additional funding or provided plans for sustaining any additional funding provided as an implementation strategy, and implementation of interventions of these studies was identified as being led by the usual implementing agency in about 44% of the studies (table 3).

Some authors have suggested that implementation efforts and relevant IR studies should be led by dedicated implementation teams (including key personnel enacting the implementation strategy) under routine conditions.10 55 56 Such teams have been espoused as critical for maintaining the implementation activities under real-world practices and also aiding the scale-up and dissemination of such practices to other settings.10 55 56 The formation of teams is critical also for addressing systemic issues and contributing to sustainability. One example is the Population Health and Implementation Training (PHIT) Partnership in Mozambique, which has built a network of institutions to strengthen primary healthcare and is integrated into the Ministry of Health (MOH) at the subnational level.21 Partners include provincial health government, academic institutions (local and external), NGOs, and the research arm of the MOH.21 Each PHIT partner has a specific technical assistance and/or research role to support provincial health authorities design and implement activities. Another example is the proof of concept study in Liberia that assessed integration of family planning and immunisation. In this case, the MOH worked with an implementing partner NGO and USAID to design a project that would generate lessons to inform implementation at-scale. The three partners worked together to engage with a broader stakeholder group, identify intervention study areas, pilot intervention and share results (see Cooper in table 1).33

Research under real-world conditions may also incorporate readiness assessments or involve organisational and system level changes for optimising the benefit and impact of evidence-informed interventions.56 For example, the BetterBirth project recognised leadership gap at health facility levels in Uttar Pradesh, India, and the implication of this for the successful rollout of the SCC intervention.38 The project engaged with district health level leadership and an application-based technology to motivate organisational and system level changes that provided real-time data feedback to district teams on observations and the availability of essential birth supplies at health facilities.38 Using the data, the district health leadership were able to strengthen the real-time availability of supplies necessary for essential birth practices.41 The PHIT Partnership in Mozambique has integrated targeted operations research studies to address systems bottlenecks on a continuous basis; these are implemented in partnership with local level managers.21 Examples of their studies include assessments of the effect of introducing shift work to extend outpatient care clinic hours and quality improvement activities for reducing loss to follow-up among paediatric patients with HIV.21 These strategies involved organisational changes to improve work processes to support implementation of evidence-informed interventions.

IR conducted under real-world conditions also allows for identification and potentially the modification of incentives faced by different actors. A project that expanded community mobilisation for maternal and newborn health through women’s groups in rural Bangladesh faced challenges of frequent staff turnover of group facilitators due to unexpected constraints on facilitators’ movement or offers of better employment, which were compounded by recruitment requirements for minimum levels of literacy. Project implementers were aware of how important familiarity with local communities and customs was to effective facilitation, so they adapted the recruitment criteria and processes to be more flexible and responsive to their context.25 Another example is the STRETCH trial in South Africa, which was an intervention aimed at supporting nurse-initiated and managed antiretroviral therapy. As part of the trial, clinical support to improve quality of care and self-confidence was offered to nurses administering antiretroviral therapy through several mechanisms, including training, telephone and in-person assistance and management and logistics support. Unfortunately, clinical support was insufficient and nurses felt that STRETCH trainers were unable to fill the gap.24

IR studies should provide evidence for ‘how’ and ‘why’ interventions led to health impact through use of implementation outcome variables and a pragmatic research paradigm

The primary objective of IR studies is often about answering how and why interventions produce their desired effects on health in a given setting.8 9 There is almost always the assumption that some evidence exists a priori on what interventions have worked for a particular health problem in some (often controlled) setting; this is why they are selected as ‘evidence-informed’ interventions. What is often missing, however, is an understanding of the implementation pathway for these interventions; how they are carried out in real-life contexts to address existing health problems and how they may be used to achieve sustainable impact at scale.7 One way to describe the implementation pathway is to link changes in implementation outcome variables and contexts to specific implementation strategies.7 Implementation outcome variables include measures of implementation fidelity, acceptability of interventions by potential beneficiaries, uptake of the interventions, reach or coverage and cost.7 8 The changes in implementation outcome variables and contextual factors could in turn be linked to measures of programme effectiveness and impact.7 For example, an evaluation of the Brazil’s Bolsa Familia programme—a conditional cash transfer programme to reduce poverty and improve health outcomes among poor families—showed that an increase in the reach of the programme (an implementation outcome variable) was associated with increased utilisation of preventive child health services, and this in turn was associated with a decline in postneonatal deaths and infant mortality over a 5 years period.27 The study was able to show the implementation pathway leading to impact for the evidence-informed intervention by explicitly linking changes in implementation outcome variables to indicators of programme impact.

Our review, however, found that the majority of IR studies from LMICs neither described changes in implementation outcome variables nor described how these changes are related to implementation strategies and desired outcomes in a given setting (table 3). While such descriptions may not be necessary for all types of IR studies, they are essential for IR studies that yield evidence of large-scale impact. Without such links, IRDS studies perhaps become undistinguishable from more traditional efficacy and effectiveness studies57 and the utility of IRDS for fulfilling the dual role of being useful for generating knowledge while addressing health problem in real-time may not be realised.57

Use of a pragmatic research paradigm and relevant study designs such as implementation-effectiveness hybrid designs (which combines measurement of outcomes of clinical effectiveness and implementation), mixed methods designs (which strategically combines qualitative and quantitative methods) and systems science methods (which characterises nonlinear processes within complex phenomena) allow IR researchers to model the iterative nature of implementation efforts without compromising the rigour of such studies.8 11 12 Such research designs are also useful for modelling changes in the intervention as well as implementation outcome variables and programme effectiveness in the same study. Our review suggested that mixed methods were indeed commonly used in IR health studies in LMICs; more than one-third of IR papers reviewed used some form of mixed methods. However, when we examined these mixed method IR studies closely, very few described the explicit mixed method design strategies used,58 which have implication for the rigour and validity of findings from such studies. Furthermore, only 14 of the relevant 791 IR papers we reviewed used other designs informed by a pragmatic research paradigm, such as implementation-effectiveness hybrid designs or system science methods. One example involved studying the adaptations to the implementation of the Seguro Popular de Salud programme in Mexico (a health financing reform package to accelerate universal health coverage and financial risk protection) using a system dynamics approach to inform a better understanding of real-life implementation variations in the programme in response to different incentives.59 While the impact of the reform on reducing catastrophic health expenditure was previously established,60 the application of a system dynamics approach to the evaluation data provided useful lessons on how the reform was tailored to various contextual factors and the features of organisational and system level changes that contributed to the impact observed under various settings.61

Conducting IR studies should be fit to purpose and balance the need to address the immediate implementation problems and support broader and longer term learning

Whereas IR studies conducted under real-world conditions may be more responsive to immediate implementation problems, lessons learnt from such studies may not always be readily generalisable to support broader and longer term learning, resulting in a common trade-off of IR studies.61 Suggested approaches for balancing this trade-off include use of specific research designs (as described above) that allow IR studies to adapt and study particular real-world conditions while still ensuring that research parameters are studied and analysed objectively.8 11 12 Others have suggested that well-designed studies conducted within multisetting and multicountry contexts help to balance this trade-off and improve the generalisability of IR study findings.56 61 For example, a feasibility study to evaluate the potential for establishing a platform for bringing together relevant health data from the public and private sectors to promote data use at the district health level was initially implemented in five different districts of India, Nigeria and Ethiopia.62 This study captured potential facilitators and barriers to the introduction of the platform and provided generalisable learning on the lack of standardised processes for data-based decision-making at district level across all three countries.62 Less than 10% of studies we reviewed were conducted within multiple settings and multicountry contexts. This may in fact be reflective of the financial and logistical constraints of undertaking such complex research efforts.4 While it may be difficult, and perhaps inappropriate, to answer relevant IR questions using multiple settings and countries, the need to optimise the trade-off between internal and external validity of IR studies is another reason to use of adaptive research designs.8 11 12 56

IR requires multistakeholder collaboration and partnerships

Collaboration and partnerships among multiple stakeholders (such as academics, implementers, policy-makers and donors) across various influence domains (research, programme, policy and funding) is important for any IR enterprise to achieve large-scale impact.1 2 4 5 8 10

Aligning research, programme, policy and funding cycles has been suggested as critical to address health problems at scale using IR, especially in LMICs.2 Such alignments are often more easily accomplished when there is collaboration and partnership to facilitate decisions and cross-learning among actors involved in these various cycles.2 One example is a community mobilisation project of women’s groups in Bangladesh, which made considerable efforts to engage and build partnerships with local government and NGOs, and also with community stakeholders involved in the women’s groups, such as traditional birth attendants and community health workers.25 These efforts were made in recognition that trust was needed to ensure an effective intervention, but also to eventually address service gaps and influence policymaking.25 Similarly, an intervention to enhance social capital in Nicaragua worked with governmental, non-governmental and community stakeholders at the project’s outset to identify, based on their collective experience, the communities which would be targeted in the programme, but further interactions between the stakeholders was unclear.40 In the integrated family planning and immunisation project in Liberia, project team recommendations were supplemented by additional recommendations from the MOH, suggesting strong engagement from policy-makers.33 The degree of alignments may also reflect the extent of embeddedness of such IR activities within context-specific programme, practice or policy and could infer the policy and practice relevance of findings from such IR activities.34 Some teams develop relationships, either by design or that build organically over time, that put researchers in a position to support implementers’ roles and activities. For example, Ngana and colleagues point out how by implementing a participatory action research model to address health reporting challenges, health staff developed problem-solving skills that allowed them to better respond and adapt to systemic challenges.30 Under the STRETCH trial in South Africa, the trial coordinator contributed to improving communications between clinical staff to strengthen relationships between physicians, nurses and other staff, which were critical to successful task-shifting.24 However, our review showed that efforts to involve key stakeholders across various domains of influence are still infrequent, with only 16 out of the 28 articles included for in-depth review reporting on the policy or practice implications of their findings (table 3).

Reporting of key characteristics of IR studies

Our review uncovered the lack of consistent reporting of implementation characteristics, including the adequate descriptions of the context, the intervention itself, including deviations from planned interventions and changes in implementation variables. Several authors have noted that implementation of evidence-informed interventions in health has been hampered by the lack of adequate reporting of IR studies.16–18 As a result, implementers have difficulty in translating the IR publication findings into practice, and the findings of such research cannot be easily used outside of their original settings.16 17 Adequate reporting of implementation strategies has been particularly emphasised in the literature because of its centrality to implementation science.17 Indeed, some authors have referred to implementation strategy as the ‘intervention’ in IR studies,8 11 and several guidelines have been proposed on how to report implementation strategy and IR studies.16–18 All the guidelines recommend that an implementation strategy should be described in details and the implementation outcome variable that describes the proximal effect of such strategy should also be described.16–18

Other key aspects that have been proposed for reporting include description of the background (including implementation gaps and supporting evidence for selecting an intervention), the settings and contexts (including relevant factors within the internal and external environment of the implementing organisations and beneficiaries) and how these factors changed over time to facilitate or hinder implementation activities.16 Such information may be required to inform practitioners on how to adapt the strategy or implementation activities to their own local contexts. It is possible that issues, such as journal word limits and writing requirements, can restrict what is included in publications.16 Reporting of IR studies could be improved if deliberate attention is paid to highlighting key aspects of IR given journal constraints. Our review suggested that articles published in certain journals (eg, Global Health: Science and Practice, PLoS One, Implementation Science) had higher probability of reporting more completely on the implementation descriptors assessed under this study. This goes to suggest that journal constraints may be a much bigger factor in determining extent to which key IR characteristics are reported. Our review further suggested that involvement of multiple stakeholders in an IR study was associated with more complete reporting on the implementation descriptors assessed. For example, articles involving academics, CSO, government and donors had the highest percentage of reporting on implementation outcome variables (69%), much higher than observed with single stakeholder or other multistakeholder groups for example, academics/CSO/government (51%), academic/CSO (50%), academics/government (57%), academic/donor (67%).