Lessons Learned From the Capacity-Building and Mentorship Program to Improve Health Information Systems in 11 Districts of Ethiopia

Learning and Demonstration Sites

A subset of 74 CBMP program sites were selected by DUP partners and RHB collaborators to become “learning and demonstration sites.” These sites received additional resources and more intensive capacity-building support to develop and implement site-specific HIS strengthening plans. They also received more frequent and structured mentorship visits (i.e., every 2 months) and on-site needs-based trainings focused on improving HIS capacity and competency among staff. As an initial step of a long-term strategy to diffuse learnings to other districts and health facilities across the regions, these sites served as incubation and testing locations for data use interventions.

Study Design

This study assesses the effect of the more intensive CBMP HIS intervention package implemented in the learning and demonstration sites on HIS performance at the primary facility level. We measured HIS performance at 24 learning and demonstration sites in 2018, then compared these indicators against the same measures at the same sites in 2020. The 24 sites (6 hospitals and 18 health centers) were randomly selected from the 74 CBMP learning and demonstration sites. The selected sites represent 1 district within each of the 11 regions in Ethiopia.

The Performance of Routine Information System Management (PRISM) tools¹¹ were used to measure the effect of the CBMP intervention package on HIS performance in selected study sites. Specifically, the following PRISM assessment tools were adapted to the local context and used for this study: District Routine Health Information System (RHIS) Performance Diagnostic Tool; Facility Level RHIS Performance Diagnostic Tool; Management Assessment Tool; and Facility/Office Checklist. The PRISM tools provide a structured way to assess RHIS data quality and information use in health facilities and support the identification of strengths and weaknesses in these areas that can inform intervention refinements.¹²

The data collection team received an intensive 5-day training, guided by a structured training manual, led by experts who were familiar with the Ethiopian health system and PRISM tools. Data were collected using tablets loaded with SurveyCTO, an Android-based mobile data collection platform, by data collectors who had experience in similar activities and were familiar with the Ethiopian health system. Internal data quality validation rules were programmed in SurveyCTO to minimize data collection errors. The data were uploaded in real time and managed on a cloud server. Regular data checks were administered throughout data collection to ensure quality.

Data Analysis

In line with the World Health Organization’s (WHO) data quality assessment guideline, 7 tracer indicators were selected by the MOH to assess HIS data quality over a 3-month period in 2018 and 2020: (1) number of births attended by skilled birth attendant, (2) number of new and repeat contraceptive acceptors, (3) children younger than age 1 year who have received a third dose of the pentavalent vaccine (penta 3), (4) number of clients who tested positive for HIV, (5) number of children younger than age 5 years with pneumonia who received treatment (U5 pneumonia), (6) number of confirmed malaria cases using microscopy or rapid diagnostic testing, and (7) number of all forms of TB cases (Table 1).^11,13 The PRISM tools measure HIS data quality in 3 dimensions: (1) data completeness, (2) reporting timeliness, and (3) data accuracy.

1. Data completeness

   • Source document completeness (facility level): Number of facilities that completed primary source documents (e.g., registers and tally sheets).

   • Reporting completeness (district level): Number of expected reports compared to the number of reports actually submitted to the district health office.

2. Reporting timeliness (district level): Number of facility reports received by the district health office by the predetermined deadline.

Data accuracy: Numerical consistency between the data recorded in facility source documents and the data aggregated in monthly reports submitted to the district health office. Report accuracy was assessed among facilities providing services associated with the 7 tracer indicators (Table 1) reviewed in this study. In this article, we used the MOH and WHO’s “acceptability tolerance range” for discrepancies in data accuracy (i.e., within ±10%).^11,13

The PRISM tools were also used to assess HIS performance related to information use by measuring the use of data for planning and target setting; performance review and evidence-based decision-making; production of analytic reports; and data dissemination. The PRISM tools were also applied to assess the status of HIS data management and processing related to data quality assurance practices, data analysis and visualization, supervision, and feedback mechanisms.

Data were cleaned and analyzed using Stata version 14.0 and are presented as descriptive statistics based on MEASURE Evaluation’s PRISM analysis guideline.¹⁴

Ethics Approval

Ethical approval was sought from and granted by the Ethiopian Public Health Association EPHA/OG/203/20.

Changes in Data Quality

For consistent information use to occur, data need to be of good quality so that users are confident that the data they consume are complete, accurate, and available in a timely manner.¹⁷ Findings from this study indicate improvements in the generation and use of quality data over the 2-year intervention period. The percentage of health facilities that have a data accuracy ratio within the accepted tolerance range (±10%) improved across most tracer indicators.¹³ While encouraging, there is room for further improvement in data accuracy across all tracer indicators to ensure quality data are consistently available in the HIS. The inadequate data quality control practices and gaps in source document completeness that we observed have implications on health facility reporting accuracy. These findings are consistent with similar research conducted in other parts of Ethiopia, indicating poor quality assurance mechanisms in the country.^15,16,18,19 Data quality assurance mechanisms succeed when they are accompanied by continual feedback, which increases the visibility of any issues and builds health workers’ capacity in addressing them.²⁰ The results highlight the need for the CBMP to further build these necessary skills through tailored and targeted training, supervision, and mentorship activities. To address this gap, the program intends to institutionalize multiple levels of quality checks and feedback loops through the lot quality assurance sampling technique, routine data quality assessment, data quality desk review, and behavioral interventions that foster sustainable change in health facilities and districts.

The study results show improvements in primary health care facilities reporting aggregated routine service and disease data to higher-level facilities. Following the nationwide transition from a fragmented electronic health management information system (HMIS) to a unified, web-based District Health Information System 2 (DHIS2), intensive trainings were provided to health workers on HMIS indicator definition, data capturing, and reporting.^9,21 The deployment of a standardized electronic data aggregation and reporting system and accompanying capacity-building activities have facilitated observed improvements in reporting completeness. Other studies have also shown that data reporting improves after an electronic, web-based HMIS platform is introduced in-country. For example, evidence from Uganda and Bangladesh showed improvement in the reporting of certain health indicators after DHIS2 implementation.^22,23 The DHIS2 implementation in Uganda improved outpatient and inpatient reporting, as well as the reporting of health service coverage (e.g., penta 3, pregnant women who attended at least 4 antenatal care visits, and birth with a skilled birth attendant). As DHIS2 is further integrated throughout Ethiopia, we expect to see continued improvement in the reporting of health facilities within the CBMP areas.

Transforming to an Information Use Culture in Health Facilities

We observed both positive and negative changes in the development of a data use culture during the 2-year study period. PMT meetings are one of the data use platforms institutionalized at district health offices and primary health care unit levels in Ethiopia. Findings from this study indicate that PMTs are generally used to promote the use of routine data to track performance, identify performance-related problems, and create action plans to address issues that arise. The importance of such review forums to strengthen HIS activities has been highlighted in other studies.^20,24 While our findings show that PMTs have been established in health facilities, regular monthly PMT meetings declined over the 2-year study period. The need for social distancing because of the coronavirus disease (COVID-19) pandemic, poor Internet access to conduct meetings virtually, and shifting priorities to respond to COVID-19 may have contributed to this. For those facilities that were able to conduct meetings, encouraging progress was observed in the use of data for discussing and reviewing performance, problem identification, target setting, and planning.

This study also revealed improvements in data use for planning and target setting at health facilities. After 2 years of implementing activities under the CBMP, the use of routine health information for data presentation, root cause analysis, and action planning increased at the health facility level. This change was observed in the health facilities at the time of supervision and from success stories documented by some health facilities.^25,26 This indicates that progress toward a data use culture among health care administrators and providers may be unfolding over time, which could contribute to improved health service provision if it continues. Classroom training is not enough to effect change, since it is not expected that all staff will apply what they learned during the training. The practical nature of mentorship coupled with trainings generated more attention and capacity to use data for action.

A culture of supportive supervision is also necessary to promote data quality and support information use at all levels of the health system. This study acknowledges the inadequacy of HIS supportive supervision practices and use of data for supervision at the primary health care level. Supportive supervision practices must be enforced to ensure higher levels of data engagement and data quality across the health system. Strengthening existing supportive supervision (in terms of frequency, quality, and approaches) can contribute to improving health workers’ capabilities by encouraging their adherence to data quality assurance practices and use of data to address problems and service delivery issues.^16,27,28

Data Management and Processing in Health Facilities

HIS data management and processes are also crucial to the production and use of quality data. The study revealed that some HIS processes, such as feedback loops and data visualization, are already in place at the health facility level. For managers to effectively apply facility data for the daily planning and management of primary health care delivery, the data must be processed into a meaningful format through analysis. However, results indicate that data analysis practices are weak at the health facility level. These findings are consistent with previous studies’ observations of the limited data analysis, interpretation, and problem-solving skills among health workers and how this, in turn, hinders information use at the point of care.^17,21,28 This implies that capacity-building interventions are needed to improve the analytical skills of health workers and health facility managers. Integrating simple digital data analysis tools and dashboards as part of the CBMP intervention package, along with targeted skill development, would help address this gap.

The quarterly IR status assessment conducted by the university partners helped identify strengths and areas of improvement. Interventions were tailored based on the gaps in the assessment; behavioral interventions were also integrated, contributing to the improvements we observed. Moving forward, continual assessment and site-specific modifications are essential to further improve the effectiveness of the CBMP.

These study findings underscore the importance of maintaining regular coordination and review meetings with CBMP partners, providing focused support to facilities and district health offices, and establishing mechanisms for routine HIS improvement monitoring. Furthermore, the findings suggest that mentorship decentralization, particularly by creating and using a pool of local mentors, could increase the impact and sustainability of HIS strengthening interventions.

Limitations

Our results should be interpreted with caution because of certain study limitations. These include the pre-post design, which lacks a control group, and the small sample size, which limits its statistical power and generalizability. While we acknowledge that our sample size is relatively small and only represents 1 district per region, sharing the results is crucial to imparting lessons learned from implementation. A more rigorous statistical evaluation of the CBMP, which will not be beset by this limitation, is forthcoming.