The country context

The maternal mortality ratio for Nigeria was estimated at 576/100 000 live births for the period 2006–2013 and is likely to be higher in the north-east since associated health indicators such as uptake of antenatal, delivery and postnatal care are poorer in this zone.25 The neonatal mortality rate in north-east Nigeria in this period was estimated at 43/1000 live births. There was no community-based health outreach in rural areas and ‘Community Health Extension Workers’ provided reproductive healthcare in community health centres. Only 49% of women received antenatal care from a skilled provider and 20% delivered in a facility. Thirty-two per cent of mothers and 10% of newborns had a postnatal check within the recommended 2 days of birth.25

In Ethiopia, the pregnancy-related mortality ratio estimate for 2009–2016 was 412/100 000 live births, and the neonatal mortality rate in the period 2012–2016 was 29/1000 live births.26 The greatest burden fell on the rural poor and health posts were established in each subdistrict, or kebele, staffed by two ‘Health Extension Workers’ who provided antenatal, delivery and postnatal care and were responsible for appropriate and timely referral where needed to Primary Health Centres, the next tier of the health service.27 Over the same period, in rural areas 58% of women had any antenatal care from a skilled provider, 20% delivered in a facility and 13% of mothers and 10% of newborns had a postnatal check within the recommended 2 days of delivery.26

In Uttar Pradesh, between 2009 and 2012, the maternal mortality ratio estimate was 258/100 000 live births and the neonatal mortality rate was 49/1000 births.28 Eighty-five per cent of women received some antenatal care.28 In the period 2011–2016, 68% of women delivered in a facility and 59% had a postnatal check within the recommended 2 days of delivery.29 In contrast, 29% of newborns were reported to have been checked in their first 2 days of life. Three cadres were responsible for community-based reproductive healthcare. Accredited Social Health Activists and Anganwadi Workers undertook outreach and home-based counselling and record-keeping while Auxiliary Nurse Midwives offered facility-based care.

The portfolio of projects and innovations

All nine projects sought to support government efforts to improve MNH and stimulate adoption of health-enhancing behaviour. During this study, they were implementing 61 innovations which were either designed and developed for a specific context, such as ‘Links with pastoralist or remote communities’ of the SFH-MNH project in north-east Nigeria,20 or adapted to address local needs from an existing tool, such as the ‘Safe Childbirth Checklist’ of the Better Birth project in Uttar Pradesh.23

The innovations had been developed in line with an overarching theory of change from the funder, the Bill & Melinda Gates Foundation (figure 1).30

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Theory of change with agreed characterisation framework questions. MNH: maternal and newborn health.

This theory proposed that innovations should enhance routine contacts between front-line workers and service users by making them more frequent, better quality and more equitable. These enhanced routine contacts were expected to increase coverage of evidence-based, life-saving interventions such as immediate and exclusive breast feeding or appropriate and safe administration of antibiotics, and thereby to improve the survival of mothers and newborns.3 The theory of change also included a scale-up loop, which described adoption of innovations beyond the project areas. Here we focus on implementation in the project areas. Evaluation of project outcomes as well as scale-up of innovations are reported elsewhere.23 31–39

Glossary

Key terms used in this paper are defined in box 1.

Characterisation step 1: develop a characterisation framework

Characterisation teams were formed for each project, comprising a group of two to five project personnel at management level with specialist implementation or evaluation knowledge of the project, and two from the study team.

Through an iterative process, a common set of five questions was agreed that would describe innovations and how they were designed to enhance MNH. This set of questions, known as the characterisation framework, was mapped onto the theory of change (figure 1). The characterisation framework provided a mechanism to add specificity to the theory of change. The questions captured detail about each innovation: where, when and how it was expected to enhance contact with the health system, and how this enhancement was expected to lead to an improved health outcome.

Characterisation step 2: populate the characterisation framework

Characterisation teams drew on project documents and critical discussion to agree a set of innovations that represented their work. Critical discussion was also essential to fill gaps in project documentation and agree a definition of each innovation. Projects with external evaluation partners, such as Better Birth and Manthan in Uttar Pradesh, had well-defined innovations, while other projects such as the Uttar Pradesh Community Mobilisation Project (UP-CMP) had not previously considered distinct innovations within its overall project.

The characterisation teams from the nine projects identified 61 innovations (online supplementary appendix table A1) across the study period. These had varied intended outcomes such as enhanced MNH care seeking, access and provision of MNH services and adoption of healthy MNH-related practices in the home and community.

Each innovation was described using standardised terms to address the five agreed questions in the characterisation framework (figure 1). An example of a completed characterisation of the SFH-MNH project in north-east Nigeria is given in online supplementary appendix tables A2 and A3. The five questions were as follows:

Characterisation step 3: collate information from all characterisation teams by geography

All innovations operating throughout the study are listed in online supplementary appendix table A1. We collated characterisations by geography to describe the anticipated combined population-level effects of all innovations in that geography on contacts during antenatal, delivery and postnatal care and on coverage of life-saving interventions.

The theory of change proposes that innovations enhance routine contacts between front-line workers and service users in each geography. To illustrate, we examined anticipated enhancement of skilled birth attendance, focusing on five dimensions: frequency; quality as timeliness; quality as content of care; equity of access; and facility readiness (table 1). Not all projects implemented innovations to address all five dimensions. In Ethiopia, all three projects included innovations which were contributing to enhanced frequency and timeliness of skilled birth attendance through their community-based innovations to enhance awareness of MNH and care seeking, as well as strengthening the work of community front-line workers. None was actively working on facility readiness. In Uttar Pradesh, only the Better Birth project sought to enhance facility readiness through its Safe Childbirth Checklist innovation. Better Birth and Manthan both sought to enhance quality of skilled birth attendance in Uttar Pradesh. The same innovation mapping was undertaken to describe antenatal and postnatal care.

Following the theory of change (figure 1), enhanced contacts between front-line workers and service users would result in increased coverage of life-saving interventions. The effect may be direct or indirect (see box 1 for definitions). To illustrate, we focus on the intrapartum period, described in table 2.

In north-east Nigeria and Ethiopia, the anticipated joint effect of the innovations was largely indirect, mainly comprising promotion of facility delivery, recognition of danger signs and appropriate timely referral, while in Uttar Pradesh, Better Birth and Manthan worked on enhancing coverage through engagement with front-line workers who were directly administering life-saving interventions and the facilities in which they worked.

An important finding of this step of the characterisation process was that the implementation of the innovations differed by time and geography, even within a single project. To examine the potential joint effect of the projects in any one geography, information was collected on the timing and geographical focus of each innovation. A high-level summary of timing of projects is shown in table 3. As the table shows, while the projects in north-east Nigeria and Ethiopia did overlap in terms of timing, in Uttar Pradesh, only UP-CMP and Better Birth operated in the same period. Examination of data on geographical focus suggested a mixed picture. In north-east Nigeria, both projects intended to work in the whole of Gombe State so there was an opportunity for synergy in the field, while in Ethiopia the three projects operated in different woreda (districts) so there was no such opportunity.

Characterisation step 4: annual update

The characterisations were updated annually to keep abreast of changes in project design and implementation and to inform evaluation. Substantial changes took place, for example, when SFH-MNH Nigeria embarked on their work in 2012 they anticipated implementation of 13 innovations in two states (online supplementary appendix table A1), but by the 2014 update they were implementing nine innovations in one state (online supplementary appendix table A2).

Value of using a theory of change

Mapping innovations onto an established overarching theory of change enabled stakeholders including donors, policymakers and managers and evaluators of projects to identify what innovations were implemented, where and when. At each step, we identified the contribution of every innovation within each project and the anticipated change resulting from the combined work of all projects. This enabled us to apply common measures, including frequency, quality and equity of targeted contacts, and coverage of targeted life-saving interventions (figure 1). We were also able to identify gaps in provision and areas of overlap, which were of interest to policymakers, the funder and the implementers in each geography.

A theory of change can be defined in many ways.14 40 It can be seen as an iterative and participatory tool to describe a programme fully, examine assumptions and foster learning through reflection and revision,41 or it can also be used more narrowly to describe a logical programme pathway from inputs to outcome and provide a framework for evaluation.42 We used an existing theory of change in this latter way to make sense of a portfolio of diverse innovations implemented within existing health systems in three geographies, and identified their anticipated individual and joint contribution to enhancing MNH.

Value for characterisation teams

Our study made real a rather abstract overarching theory of change. For some participating project staff, the exercise gave them an opportunity to engage in assessing how their work contributed to this overarching theory, thereby giving them a sense of common purpose within the donor’s broader strategy to improve MNH. Project teams were interested to learn from each other’s work at annual meetings, which enabled them to identify both commonalities and uniqueness, as well as considering the adoption of common indicators, which could be used to measure the combined effects of multiple innovations across each geography. The discussions stimulated the teams to examine assumptions about steps along the pathway from innovation to outcome and generated a new interest in process evaluation and in qualitative work to investigate the complexities of their work.

Defining innovations

Defining innovations was a challenge for some projects, which proposed many components as individual innovations, such as SFH-MNH’s front-line workers toolkit, while others proposed fewer components, such as Better Birth’s introduction of a Safe Childbirth Checklist in health facilities. Focusing on the feasibility of measuring the outcome of the innovation encouraged these projects to consider the purpose of each innovation and how it might interact with others to affect an outcome.

Time needed to complete the characterisations

Of the nine projects, two did not have a well-developed evaluation plan or external evaluation partners: they found it challenging to conceptualise their work in terms of discrete innovations with measurable outcomes. For these projects, characterisation meetings took up to 2 days. The other seven projects did have external evaluation partners, well-articulated innovations and measurable outcomes, and for these the exercise could be completed within 2 hours. The most efficient characterisation teams in terms of completeness, clarity and speed included three to five members of project staff who had an operational understanding of the details of the innovations and an overview of the purpose of their work.

Engaging with complexity

Our work characterised innovations by considering them as sets of bounded activities with interacting and mutually reinforcing components, developed for or adapted to the context. As such, they could be seen as complex interventions.43 However, in reality projects in the same geography operated at different times or different places, to different scales, thereby losing opportunities for synergy. For example, Manthan and Better Birth both worked on enhancing the quality of skilled birth attendance in Uttar Pradesh. However, Manthan worked with Auxiliary Nurse Midwives in one block in Bahraich District and Better Birth implemented the checklist in 60 hospitals in 24 districts in the State.

Systems thinking examines interactions between the innovation and the whole system, rather than between the innovation and one component of the system. It examines how the system adapts and measures outcomes at all levels, and considers how outcomes in different parts of the system affect one another and the mechanisms by which the effect of an innovation is amplified or diminished.44 This approach would be suited to a set of projects aligned and integrated in both space and time, and we were not able to adopt it here.

We engaged with the complexity of individual projects by examining how the innovations operated, and their effect on front-line workers and targeted improvements in the health system in different combinations and different contexts. The projects were dynamic and were reshaped in response to changing needs. We updated our characterisations annually to capture these changes. Further examination of complexity would involve a separate qualitative study.

Harmonised terms

Behaviour change interventions in the literature tend not to have commonly agreed terms and definitions,12 45 in contrast to natural science experiments.46 47 Terms and definitions can lack precision and standardisation so knowledge in the field of behaviour change in healthcare seeking and provision has been slow to evolve, hampering scale-up of effective initiatives.12 Particularly where there are numerous stakeholder groups, such as government policymakers, implementers, donors and researchers and evaluators, a common vocabulary has been essential. Those wishing to coordinate multiple efforts under a common set of criteria, harmonise and align support to government priorities9 or replicate initiatives need a lucid description of what has been tried and a way to compare different initiatives, including an assessment of their contribution to behaviour change.12 16 48–50 This clarity is particularly valuable where ministries of health and other health authorities deal with multiple donors and implementing partners. We used harmonised terms across projects to describe individual and combined efforts to improve MNH, whether and how they interact, their location, timing and what changes were made across all projects in a given geography. We would advocate for such an approach in any group seeking to compare implementation or effectiveness of a set of complex interventions.

Limitations

Our study had a number of limitations. First, we did not examine the extent to which the innovations were relevant for the purpose they set out to address. In part, this is because the projects were already under way when we undertook the characterisations.

Second, we included generalised material on the health system in each country to provide the context for the characterisations. On reflection, we could have described the context in a more granular way,13 including other substantive activities taking place in the same geographies, to enable us to consider how the innovations and the system adapt in different contexts and help explain findings in different contexts.

Third, there was risk of reporting bias in the characterisations, as characterisation teams may have downplayed programmatic challenges.51 Although we made efforts to ensure that the opinion of one team member did not dominate at the expense of others, we were not in a position to verify this objectively or to assess whether descriptions were accurate.

Fourth, as part of the characterisation process, more attention could have been paid to the length of time needed for an innovation to have a measurable effect on outcomes. Innovations such as SFH-MNH’s work to enhance supplies of clean delivery kit were likely to have an immediate and direct effect on health outcomes because of improved hand-washing and use of gloves during delivery. Others were only likely to have an effect on outcomes in the longer term, such as changing community attitudes through the mass media event.

Lastly, for projects without predefined innovations, discussions were lengthy and resulted in relatively large numbers of separate innovations. By focusing on capturing and describing discrete innovations, our approach may have missed synergistic aspects of combinations of these innovations.