http://orcid.org/0000-0001-5400-9383
Figures
Abstract
Introduction
Routine health information systems are critical for monitoring service delivery. District Heath Information System, version 2 (DHIS2) is an open source software platform used in more than 60 countries, on which global initiatives increasingly rely for such monitoring. We used facility-reported data in DHIS2 for Gombe State, north-eastern Nigeria, to present a case study of data quality to monitor priority maternal and neonatal health indicators.
Methods
For all health facilities in DHIS2 offering antenatal and postnatal care services (n = 497) and labor and delivery services (n = 486), we assessed the quality of data for July 2016-June 2017 according to the World Health Organization data quality review guidance. Using data from DHIS2 as well as external facility-level and population-level household surveys, we reviewed three data quality dimensions—completeness and timeliness, internal consistency, and external consistency—and considered the opportunities for improvement.
Results
Of 14 priority maternal and neonatal health indicators that could be tracked through facility-based data, 12 were included in Gombe’s DHIS2. During July 2016-June 2017, facility-reported data in DHIS2 were incomplete at least 40% of the time, under-reported 10%-60% of the events documented in facility registers, and showed inconsistencies over time, between related indicators, and with an external data source. The best quality data elements were those that aligned with Gombe’s health program priorities, particularly older health programs, and those that reflected contact indicators rather than indicators related to the provision of commodities or content of care.
Conclusion
This case study from Gombe State, Nigeria, demonstrates the high potential for effective monitoring of maternal and neonatal health using DHIS2. However, coordinated action at multiple levels of the health system is needed to maximize reporting of existing data; rationalize data flow; routinize data quality review, feedback, and supervision; and ensure ongoing maintenance of DHIS2.
Citation: Bhattacharya AA, Umar N, Audu A, Felix H, Allen E, Schellenberg JRM, et al. (2019) Quality of routine facility data for monitoring priority maternal and newborn indicators in DHIS2: A case study from Gombe State, Nigeria. PLoS ONE 14(1): e0211265. https://doi.org/10.1371/journal.pone.0211265
Editor: Alessandra N. Bazzano, Tulane University School of Public Health and Tropical Medicine, UNITED STATES
Received: July 10, 2018; Accepted: January 7, 2019; Published: January 25, 2019
Copyright: © 2019 Bhattacharya et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data for this study are available from the London School of Hygiene & Tropical Medicine public repository: http://datacompass.lshtm.ac.uk/229/.
Funding: This work was supported through a grant made by the Bill & Melinda Gates Foundation to the IDEAS project at the London School of Hygiene and Tropical Medicine. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: Mr Felix Habila and Mr Ahmed Audu are members of the Gombe State Primary Health Care Development Agency. This does not alter our adherence to PLOS ONE policies on sharing data and materials. The authors declare no other competing interests.
Introduction
Routine health information systems are critical for monitoring service delivery. One distinctive feature of routine health information systems is the availability of data at a frequency and level of disaggregation seldom possible through nationally representative household surveys such as the Demographic and Health Surveys and Multiple Indicator Cluster Surveys. [1–3]
Global initiatives including the Sustainable Development Goals and Countdown to 2030 emphasize the contribution of routine health information systems to monitor progress and enable course correction. [4–6] Two major maternal and newborn health initiatives, Ending Preventable Maternal Mortality and Every Newborn Action Plan, have identified strategies to achieve goals for reduced maternal and newborn mortality by 2030 to a global average of 70 per 100,000 live births and 12 per 1,000 live births, respectively. Both initiatives have identified priority indicators as signals for progress, with a vision that facility-based data should contribute to monitoring. [7, 8] The District Health Information System, version 2 (DHIS2), is a flexible open source electronic information system currently used in over 60 countries to manage and visualize routine health data, particularly facility-based data.[9] Here, we present a case study for Gombe State, north-eastern Nigeria, to examine the availability and quality of routine facility data in DHIS2 for this monitoring purpose.
A routine health information system is a sub-system of a national health information system’s effort to capture, process, report, and use information to support policymaking and program implementation. [10, 11] A facility-based information system is a further sub-system that includes data captured by health facility workers during their day-to-day activities. These facility-based data include paper-based and electronic-based medical records, service delivery registers, and aggregate service delivery reports. When facility-based data are of sufficient quality, they can be used at the facility level for effective clinical management, at the district-level to understand the extent to which their facilities are functioning as intended, and at the state- and national-levels to review policies and allocation of resources. [1, 12] At all levels of the health system, good quality facility-based data can contribute to reliable estimates of service delivery coverage to understand if communities are accessing and receiving needed services, such as the proportion of facility births attended by a skilled health worker. [1, 3, 12, 13].
While facility-based information systems are often unable to maintain the good quality needed for monitoring [14, 15], DHIS2 is considered an innovation for transmitting and aggregating data faster than paper-based information systems and for improving data quality by limiting errors in how data are transmitted and aggregated from the facility to higher levels of the health system. Further, DHIS2 has the potential to promote program monitoring because its digital platform increases the accessibility of data for health managers and stakeholders at the district-, state-, and national levels. [3, 9, 16]
With Nigeria having one of the highest maternal mortality ratios and newborn mortality rates in the world (576 maternal deaths per 100,000 live births in 2015 and 37 newborn deaths per 1,000 live births in 2017), the Government has developed action plans to reduce preventable deaths for mothers and newborns and has made considerable investment in strengthening information systems, including DHIS2, to support performance management and service delivery. [17–22]
The aim of this study was to determine the quality of routine facility-based data in DHIS2 to monitor priority maternal and neonatal health indicators in Gombe State, north-eastern Nigeria. Using the World Health Organization data quality review toolkit, we focused on metrics for the data quality dimensions of completeness and timeliness, internal consistency, and external consistency. [23] For data defined as poor quality by the toolkit, we discussed opportunities for improvement.
Methods
Ethical approval
Gombe State approval for the study was obtained from Gombe State Ministry of Health. Ethical approval was obtained from the London School of Hygiene & Tropical Medicine (reference 14091).
Study setting
Gombe State has a projected population of 2.9 million (2006 census: 2.4 million) and is located within north-eastern Nigeria, where maternal and newborn mortality are estimated to be higher than the rest of the country (1,549 maternal deaths per 100,000 live births in 2015 and 35 neonatal deaths per 1,000 live births in 2017). [22, 24–26] In 2017, Gombe State had a total of 615 health facilities across 11 Local Government Areas (LGA, equivalent to a district); each LGA has 10–11 political wards (114 wards, total). As in other states in Nigeria, Gombe facility staff generally complete 13 paper-based registers to document the services they provide. Every month, a subset of data in these registers are tallied and summarized in a paper-based report and sent to the LGA (district) health office to be entered into DHIS2.
Data sources
We accessed three data sources for this study: facility-reported data in DHIS2, an external facility survey, and an external household survey as described below.
In 2017, DHIS2 contained monthly reports for 615 Gombe public and private health facilities across 11 districts: 587 primary facilities offering basic preventative and curative services and 28 referral facilities offering specialized care. Of these, 471 of the 587 primary facilities had been appointed to provide antenatal care and postnatal care services, 460 of the 587 primary facilities provided labor and delivery services, and 26 of the 28 referral facilities were equipped to provide both types of services, in addition to specialized care. Therefore, in total, 497 facilities provided antenatal and postnatal care services and 486 facilities provided labor and delivery services. For these 497 and 486 facilities, respectively, monthly aggregated DHIS2 data for the reference year July 2016-June 2017 were downloaded at one time and included 15 maternal and newborn health-related data elements. Additionally, we downloaded data for July 2013-June 2016 as comparison years for assessing the consistency of data over time.
In July 2017, a facility-level survey was conducted in 97 primary and 18 referral facilities across Gombe to assess their readiness to provide maternal and newborn health services. Detailed methods are reported elsewhere.[27] Briefly, these primary and referral facilities were a state-wide random sample drawn from all government-owned primary health facilities and a census of all 18 government-owned referral health facilities. The facility survey protocol was similar to a Service Availability and Readiness Assessment, which included an inventory of equipment and supplies that were available and functioning on the day of survey; an inventory of staff employed at the facility, their cadre, training and whether they were present on the day of survey; and an interview with the in-charge of the facility about the services available at that facility and about recent supervision visits they had received. Additionally, this survey included data extraction from the facility’s paper-based antenatal and postnatal care register and the labor and delivery register (Nigeria health management information system, version 2013).[28] A trained third party data collection team tallied and recorded the register data for each month of the six-month period immediately prior to the survey: January-June 2017. We compared the facilities’ paper-based register data with the facilities’ data downloaded from DHIS2. These extracted data are shown in Table 1.
Also in July 2017, a household-level survey was conducted in catchment areas of the 97 primary facilities from the July 2017 facility survey to assess access to and quality of maternal and newborn services. [27] These catchment areas represented 79 enumeration areas: some facilities serving more than one enumeration area. All households in each enumeration area were surveyed (or in a segment of between 75 households from the enumeration area if more than 75 households were present). The household survey comprised of three modules. (1) A household module asked all household heads about characteristics of the household, ownership of commodities and registered all normally resident people in the household. (2) A women’s module asked all women aged 13–49 years and normally resident in the household about the health care available to them, their recent contact with frontline workers and their birth history in the two years preceding the survey. (3) A mother’s module asked all women who reported a birth in the last two years (identified in the women’s module) a detailed set of questions about their contact with health services across the continuum of care from pregnancy to postnatal care. Informed consent was obtained at the community leadership-level and at the individual-level for each respondent; all invited participants agreed be interviewed. Among 965 surveyed women who reported a live birth in the 12 months prior to the survey, 588 women had visited the facility at least once during their pregnancy and 377 women gave birth at a facility. For DHIS2 reported indicators that were also estimated in the household survey, we compared estimates from this household survey to those from the 79 matching facilities in DHIS2. Calculations of point estimates and their 95% confidence intervals were done using the svyset Stata command (StataCorp, College Station, USA) to adjust for clustering at the enumeration area-level.
Selection of priority maternal and newborn health indicators to assess in Gombe’s DHIS2
To determine globally-defined priority maternal and newborn health data in DHIS2, we referred to the Ending Preventable Maternal Mortality and Every Newborn Action Plan strategy documents which described priority indicators to monitor progress towards targets during the Sustainable Development Goals era. [7, 8] For content of care indicators that were referenced by these strategy documents, but not yet fully defined, we referred to indicators defined in Carvajal-Aguirre et al. [29]
We focused our data quality review on health services that should be received by all pregnant women and newborns accessing either primary or referral health facilities. Therefore, rare events and outcomes such as deaths, adolescent births, pre-term births, deliveries by caesarean section, and kangaroo mother care were excluded from our analyses.
For Gombe State, we identified 14 priority maternal and newborn health indicators that were captured at the facility-level by health care workers. (Table 1) These 14 indicators are made up of 17 distinct data elements contained within the paper-based facility registers, including three denominators to determine how many women and newborns have accessed these facilities for services: women who visited the facility at least once during their pregnancy; women who gave birth in a facility; and live births among the facility births.
For Gombe State, 15 of these 17 distinct data elements were reported in DHIS2; the data for women receiving oxytocin for the prevention of postpartum hemorrhage and newborns receiving essential care were captured in facility registers, but not reported in DHIS2. Therefore, the final set of data assessed included 15 data elements used to calculate 12 priority indicators.
Data quality assessment
We reviewed the quality of the DHIS2 data according to metrics of three routine data quality dimensions outlined by the World Health Organization data quality review toolkit: completeness and timeliness; internal consistency; and external consistency. [23] Table 2 outlines the data quality metrics assessed, the criterion for each metric, and the data sources used. A stratified analysis was performed by facility type for primary and referral facilities.
Results
We present the quality of 15 data elements which represented 12 priority maternal and neonatal health indicators included in DHIS2 for Gombe State.
Completeness and timeliness of facility reporting
For settings such as Gombe, the World Health Organization guidance defined 75% to represent satisfactory completeness of facility reporting, that is each facility annually submitted at least nine of the 12 expected reports. [23] In Gombe State, facilities providing antenatal and postnatal care services (n = 497 facilities) and labor and delivery services (n = 486 facilities) submitted, on average, 75% of the expected reports during July 2016-June 2017 (nine of 12 expected reports submitted per year, standard deviation: 2.9 reports). Of these, 84% of reports were submitted on-time, although referral facilities were less likely than primary facilities to submit their reports on time (p<0.01 for both antenatal and postnatal care facilities and labor and delivery facilities). Figs 1 and 2 present the completeness of facility reporting, alongside completeness of indicator data described below.
Notes: ANC = antenatal care; HIV = human immunodeficiency virus; IPT = intermittent preventive treatment of malaria in pregnancy.
Completeness of priority maternal and newborn data in DHIS2
To assess the completeness of indicator data (the extent to which health facilities reported for specific indicators), we observed that Gombe’s DHIS2 data did not distinguish between missing values and true zero values. For example, a remote facility may have been equipped to provide antenatal care services but had no clients for antenatal care during a review month (true zero value); in contrast, a facility may have provided antenatal care services to clients but did not include this in their monthly report (missing value). In Gombe’s DHIS2, both situations are presented as missing data.
The World Health Organization defined completeness of indicator data to be satisfactory when less than 10% of the expected data were missing values. In Figs 1 and 2, the priority data elements in DHIS2 with the least missing values were for the number of times pregnant women visited a facility for antenatal care (first antenatal care visits, four or more antenatal care visits), deliveries taking place in a facility (facility deliveries), the provision of tetanus toxoid vaccinations to pregnant women, and the provision of Bacillus Calmette-Guérin (BCG) vaccinations to newborns. Facilities reported a value for these data in at least 52% of the expected monthly reports and 65% of submitted reports. By contrast, the priority data elements with the most missing values were for the provision of screening tests for anemia and proteinuria as well as malaria intermittent preventive treatment. Facilities reported a value for these data in less than 25% of expected monthly reports and less than 33% of submitted monthly reports.
Differences in the completeness of indicator data were noted by facility type. Primary facilities were more likely than referral facilities to report that any woman and her newborn received early postpartum-postnatal care (early postpartum-postnatal care) (p<0.01), any newborn was given a polio vaccine at birth (p<0.01), and any mother reported exclusively breastfeeding her infant up to six months of age (p = 0.01). Referral facilities were more likely to report any pregnant women received a screening test for anemia (p = 0.03) and for proteinuria (p = 0.02).
Consistency over time
When assessing the extent to which a data element’s reported value was consistent over time, the World Health Organization guidance recommended that the reported value for the reference year be within ±33% of the mean value for the preceding three years, taking into consideration any expected changes in the patterns of service delivery. For Gombe State, facilities were more likely to report consistently, compared to the preceding three years, for 7 of 15 data elements (Table 3): first antenatal care visits; four or more antenatal care visits; women receiving at least two doses of malaria intermittent preventive treatment during pregnancy; facility deliveries; deliveries by a skilled birth attendant; newborns receiving BCG vaccinations; and mothers reporting exclusive breastfeeding through six months of age. Facilities did not report consistently, compared to the preceding three years, for women having a live birth in a facility (live births); early postpartum-postnatal care; antenatal care commodities and services provided, except for the provision of malaria intermittent preventive therapy; and newborns receiving the polio vaccine at birth.
We observed differences in consistency over time by facility type, with referral facilities more likely to report an inconsistent and higher number of events for July 2016-June 2017 compared to the mean of the previous three years for six data elements (for each data element: p<0.05): four or more antenatal care visits; facility deliveries; live births; deliveries by a skilled birth attendant; newborns receiving a polio vaccination; and newborns receiving a BCG vaccination.
Outliers in the reference year
When assessing indicator data for unlikely or extreme values (outliers) in the reference year, the World Health Organization guidance defined an individual monthly value of a given data element to be a moderate outlier if it was between two and three standard deviations from the mean value and an extreme outlier if it was more than three standard deviations from the mean value for the year.
For Gombe State, outliers were present during the reference year for nine of the 15 data elements (Table 4). Primary facilities were responsible for reporting all outliers, with the monthly outlier values being higher than the reported mean number of events for the year. Primary facilities reported moderate monthly outlier values for: first antenatal care visits; facility deliveries; deliveries by a skilled birth attendant; newborns receiving a BCG vaccination; and mothers reporting exclusively breastfeeding their infant up to six months of age. Primary facilities reported extreme monthly outlier values for pregnant women receiving a screening test for proteinuria; women whose HIV status was known or tested for; pregnant women given iron-folic acid supplementation; and pregnant women receiving at least two doses for malaria intermittent preventive therapy. Three extreme outliers were reported in October 2016, mostly due to one primary health facility’s reported values contributing 60% towards Gombe State’s aggregate value for women receiving at least two doses of IPT, 87% towards the aggregate value for women who were tested for HIV or with previously known HIV status, and 90% the aggregate value for women who received iron-folic acid supplementation.
Consistency between related data reported in DHIS2
When reviewing the extent to which data make sense with respect to each other (internal consistency between related indicators), the World Health Organization guidance recommended that pairs of data elements that we expect to be equal in value fall within ±10% of each other. For example, for Gombe State, it was expected that the number of facility births would equal the sum of live births and still births. Internal consistency between related data can also be examined by comparing the number of unique women who have accessed the facility for services (e.g., first antenatal care visits or facility deliveries) to the number of women who have received an individual service. If we find that not every woman has received the expected service, this could represent low service uptake or under-reporting. For example, the number of women tested for anemia could be compared to the number of first antenatal care visits; the use of partograph during delivery could be compared to the number of facility deliveries.
For Gombe State, related indicators that should have equal values did not meet the World Health Organization guidance: (i) the total number of deliveries (n = 65,312) did not equal the sum of live births and still births (n = 52,943) and (ii) the total reported number of women and newborns receiving early postpartum-postnatal care (n = 51,382) did not equal the sum of the visit categories (n = 34,686). (Fig 3 for antenatal and postnatal services, Fig 4 for labor and delivery services) Also in Figs 3 and 4, across all facilities, none of the priority data elements compared demonstrated the expected numerical relationship. For example, of the 143,786 first antenatal care visits, expected services provided during this first visit for anemia testing and proteinuria testing were reported for 65,732 women and 76,555 women, respectively. Primary facilities reported lower than expected numbers for 10 of the priority data elements which could be due to either low service uptake or under-reporting.
Consistency between original facility registers and reported data in DHIS2
When assessing the extent to which data match across sources (consistency of original facility registers and reported data in DHIS2), the World Health Organization guidance defined the data to be consistent when the reported value (e.g., in DHIS2) was within ±10% of the facility register’s value. This review of consistency, in part, reflected the capacity to tally and report service statistics as intended.
For the five data elements compared (Fig 5), reported data in DHIS2 consistently agreed with the original facility registers for the priority indicators’ three main denominators: first antenatal care visits, facility deliveries, and live births. In general, facilities submitted higher numbers to DHIS2 (over-reported) compared to their original facility registers by 50%-60% for deliveries by skilled birth attendants and early postpartum-postnatal care.
Notes: According to WHO guidance, ratios <0.9 or >1.1 indicate that reported data in DHIS2 were inconsistent with data extracted from the original facility register. For the 97 primary facilities where facility surveys and data extraction took place, five facilities offering antenatal and postnatal care services and seven facilities offering labor and delivery services were excluded as the facility registers were unavailable at the time of the survey.
By facility type, the frequency and magnitude of under-reporting was greater for referral facilities. While referral facilities’ data in DHIS2 consistently agreed with original facility records for early postpartum-postnatal care, referral facilities under-reported by more than 10% for the main denominators first antenatal care visits, facility deliveries, and live births and under-reported by more than 50% for deliveries by a skilled birth attendant. For primary facilities, data in DHIS2 consistently agreed with original facility registers for the abovementioned main denominators, but over-reported by more than 50% for deliveries by a skilled birth attendant and for early postpartum-postnatal care.
External consistency between household surveys and reported data in DHIS2
When assessing the extent to which the data in DHIS2 are consistent with estimates from external data sources (external consistency), such as household surveys, the World Health Organization guidance recommended that the value of the routine data lie within the confidence limits or be within ±33% of the survey result. [23]
Fig 6 presents a comparison of the data reported in DHIS2 to the estimates from household surveys for four data elements. Comparing women who had visited a facility at least once during their pregnancy in 79 matching facilities and catchment areas, the routine data in DHIS2 did not fall within the confidence limits nor within ±33% of the survey results for the three antenatal care services reviewed: four or more antenatal care visits, women receiving a screening test for anemia, and women receiving a screening test for proteinuria. Comparing women who had delivered in a facility, the routine data in DHIS2 did not fall within the confidence limits nor within ±33% of the survey results for deliveries by a skilled birth attendant. Further, no pattern emerged in which the routine data in DHIS2 consistently overestimated or underestimated the results from the household survey.
Notes: ANC = antenatal care. Household survey denominator for (i) four or more ANC visits, (ii) anemia testing during ANC, and (iii) proteinuria testing during ANC: number of women who had received at least one ANC visit while pregnant during the one year prior to the survey (n = 377 women). Household survey denominator for deliveries by skilled birth attendant: number of women who had given birth in a facility during the one year prior to the survey (n = 588 women).
Discussion
We assessed the quality of routine data in DHIS2 to monitor priority maternal and newborn health indicators in Gombe State, north-eastern Nigeria. Of 14 facility-based indicators reflecting services that every woman and her newborn should receive, data elements to estimate 12 priority indicators were included in Gombe State’s DHIS2. However, similar to other studies assessing routine data, the routine maternal and newborn health data in DHIS2 for Gombe State did not meet all defined criteria for sufficient quality. [30–48]
During the reference year July 2016-June 2017, the data in DHIS2 did not regularly reflect what was in the facilities’ service registers, were incomplete, and exhibited inconsistencies over time, between related indicators, and with an external data source. Nevertheless, the data quality metrics assessed were not equally poor across all priority indicators. This variability suggests high quality routine data is achievable.[49] Data were of better quality when aligned with Gombe’s health program priorities, particularly for older health programs; there were also differences in data quality by indicator type.
Contact indicators, which reflect attendance at a facility, had the highest overall data quality among the priority indicators: first antenatal care visits, four or more antenatal care visits, facility deliveries, live births, deliveries by a skilled birth attendant, and early postpartum-postnatal care. [29, 50] These are well-defined events to document, which may ease the tallying and reporting of these data. In particular, the main denominators—first antenatal care visits, facility births, and live births—had the highest completeness of indicator data rates in our study, were more consistent over time, lacked extreme outliers, and demonstrated the greatest level of agreement between facility registers and DHIS2 data. Further, these have been key denominators for local program planning because they track the number of women accessing antenatal or postnatal care and labor and delivery services at health facilities. These data have been prioritized for monitoring progress in previous global initiatives including the Millennium Development Goals and Countdown to 2015. [51, 52] Four or more antenatal care visits and deliveries by a skilled birth attendant are also long-standing priority indicators for these initiatives and had the same data quality characteristics as the three denominators above. However, the lower overall data quality for deliveries by a skilled birth attendant may in part be reflective of the Gombe context where the majority of facility deliveries in primary facilities are managed by community health extension workers, rather than more highly trained nurses or doctors, following the recent political instability there.[53] Finally, early postpartum-postnatal care, an acknowledged “neglected period for the provision of quality care” had the lowest data quality metrics within this type of indicator. [54]
Indicators related to the provision of a commodity or vaccination that every woman or newborn should have received had the next highest level of overall data quality. While these indicators’ overall data quality was not as consistent as the contact indicators, they had relatively high completeness, relatively low inconsistencies over time and between related indicators. While reporting for these indicators may reflect the ease of accounting for a dispensed commodity, most commodities tracked by these indicators have been a part of Nigeria’s routine immunization program where completeness of indicator data and agreement across data sources have been emphasized.[55]
The last type of indicator reviewed is related to screening or testing pregnant women for anemia, proteinuria, or HIV. These had much lower completeness of indicator data rates, lower consistency between related indicators, and more outliers. These indicators reflect a more complex encounter between the client and health care provider and have been gaining attention as the maternal and neonatal health program priorities expand to include the content of care. [56, 57] They also tended to exhibit less consistency over time, possibly reflecting the increased attention. [58]
The configuration and use of DHIS2 in Gombe State underscores government commitment to using data to improve service delivery and health outcomes. As this study was reviewed from a program monitoring lens, we did not examine all facility-level characteristics and factors associated with data quality, but outline below actions at multiple levels that could improve data quality.
Maximize the use of existing data
Data for two priority indicators of life-saving care are already captured at facility level, but are not included in monthly monitoring reports: women receiving oxytocin to prevent post-partum hemorrhage and newborns receiving essential newborn care. Monitoring these two indicators would align with recent efforts to focus on the content of care received at critical timepoints during labor and delivery. [59, 60]
Rationalize data flow
Comparing facility registers to data entered in DHIS2, referred elsewhere as “accuracy”, highlighted differences by facility type. Both primary and referral facilities were affected by challenges in data flow. At the primary level, client antenatal “treatment” cards were often kept at the facility, and data were later transferred to a register, which was the primary data source for reporting. If the data from treatment cards had not been transferred to the register when the monthly report was prepared, data was taken directly from the cards into the monthly report, resulting in apparent over-reporting. At referral facilities, the physical task of gathering data from “treatment” cards and facility registers dispersed across the hospital grounds was a challenge. The person filling in the summary form was relatively far from services provided and relied on possibly incomplete or unavailable registers, resulting in under-reporting.
Our study suggested that facility staff could strengthen accuracy and completeness of documentation by ensuring that the most complete data source, whether it is the client’s antenatal treatment card or the service register, be the primary source for tallying and summarizing the services provided in the facility’s monthly report. At the state and national-levels, another action could be to review the role that the client cards play in the data flow, given that they remain in the facility as a medical record; a simple job aid could be developed to help tally across the treatment cards, rather than intensive data transcription to service registers that may no longer be fit for purpose. A cohort register, based on month of first antenatal visit date, could be developed to combine the longitudinal information needs of a treatment card with the tallying and summarization needs of the register; however, this type of tool development may not be realistic in the near-term.
Routinize data quality reviews and feedback at all levels of the health system
Data quality review, feedback, and supervision are essential to optimize routine data for monitoring. [33, 35, 41, 43, 44, 46, 61–69] Studies specifically considering technology-based innovations, including DHIS2, noted that while innovations can make reviews of completeness and internal consistency more efficient, feedback and supervision remain essential to achieving and maintaining improvements in data quality. [33, 49, 66, 70] At the facility level, staff responsible for reporting should review the monthly reports for completeness and internal consistency, ensuring that related data elements have the expected numerical relationships, before submitting the report to the district level. This provides an important opportunity to review relationships between the number of clients and the services/commodities received to understand gaps in service uptake or gaps in data quality related to data capture or reporting; feedback can be provided to staff and supervised to address any gaps identified. At the district level, health management teams could take on these same practices and additionally structure their review and feedback to regularly allow for facilities to confront their own data and for comparison with neighboring or similar facilities in the context of where data quality metrics for completeness, timeliness, and internal consistency could be improved. [41, 43, 46, 62–64, 66, 67, 69, 71–73]
Optimize and maintain DHIS2
Many global initiatives are looking to the DHIS2 platform to promote better quality data and improve access for monitoring at all health management levels. DHIS2’s platform allows Governments to develop a responsive information system. [33, 74] Based on our study, it is difficult to determine to what extent those features have been used to control data quality. For example, at the time of this review, DHIS2 for Gombe contained inactive facilities and administrative units, duplicate entries for active facilities and data elements, and did not distinguish between missing data and true zero values. [75, 76] These required additional preparation for our analyses, suggesting that comprehensive data quality reviews could not take place in DHIS2 in its current form. An investment in DHIS2 should include ongoing reviews of its content to promote data quality and fitness for purpose.[11, 33, 35, 70, 74]
There were limitations to this study. Similar to other assessments, we did not validate the data through direct clinical observations [41, 43–45, 49, 64, 77] nor did we compare the paper-based monthly summary reports to their electronic versions in DHIS2. [32, 34, 41, 44, 61, 78] For the assessment of consistency, the facility-level and household-level surveys used in this study could not be considered a gold standard, but we did consider them to be relevant references for reviewing the consistency of routine facility-based data in DHIS2. Understanding consistency between multiple data sources is a perennial problem for health managers who frequently have to make sense of different estimates. The surveys were conducted similarly to the Demographic and Health Surveys and Multiple Indicator Cluster Surveys at the household-level, and the Service and Readiness Assessment at the facility-level, where estimates of priority maternal and newborn coverage and service delivery indicators have been obtained.[28, 79, 80] Despite close attention to quality control, these surveys might still be susceptible to errors in data recording, including incorrectly tallying the number of events in the original facility registers for comparison with data in DHIS2. Further, for some maternal and newborn health events, household survey measures may not provide a valid representation of care provided in health facilities.[81] Acknowledging these short-comings highlights the importance of working to improve the utility of routine data sources. We did not review rare events or outcomes such as deaths or complications and extra care for women and their newborns, as our primary interest was in the contribution of DHIS2 data to routine program monitoring. Lastly, this study reviewed the quality of routine data for maternal and neonatal health and may not be representative of indicators for the planning and service provision for other health programs.
Our study adds new evidence showing the potential of data in DHIS2 for local, real-time monitoring of maternal and newborn health services. While the quality of data in DHIS2 could be strengthened, the data quality metrics for priority indicators were not universally nor equally poor. Coordinated action at multiple levels of the health system is needed to maximize reporting of existing data; rationalize data flow; routinize data quality review, feedback and supervision; and ensure the ongoing maintenance of DHIS2.
Acknowledgments
The authors thank the study team for the IDEAS Project, who conducted the facility and household data collection.
References
- 1. AbouZahr C, Boerma T. Health information systems: the foundations of public health. Bulletin of the World Health Organization. 2005;83(8):578–83.pmid:16184276
- 2. Amouzou A, Kachaka W, Banda B, Chimzimu M, Hill K, Bryce J. Monitoring child survival in ‘real time’ using routine health facility records: results from Malawi. Tropical Medicine & International Health. 2013;18(10):1231–39.
- 3. Maina I, Wanjala P, Soti D, Kipruto H, Droti B, Boerma T. Using health-facility data to assess subnational coverage of maternal and child health indicators, Kenya. Bulletin of the World Health Organization. 2017;95(10):683–94. pmid:29147041
- 4.
Nations United. A/RES/70/1—Transforming our world: the 2030 agenda for sustainable development. New York: United Nations, 2015.
- 5. Victora C, Requejo J, Boerma T, Amouzou A, Bhutta ZA, Black RE, et al. Countdown to 2030 for reproductive, maternal, newborn, child, and adolescent health and nutrition. The Lancet Global Health. 2016;4(11):e775–e6. pmid:27650656
- 6. Boerma T, Requejo J, Victora CG, Amouzou A, George A, Agyepong I, et al. Countdown to 2030: tracking progress towards universal coverage for reproductive, maternal, newborn, and child health. The Lancet. 2018;391(10129):1538–48.
- 7.
World Health Organization. Strategies towards ending preventable maternal mortality (EPMM). Geneva: World Health Organization, 2015 9789241508483.
- 8.
World Health Organization, United Nations Children’s Fund. Every newborn: an action plan to end preventable newborn deaths. Geneva: World Health Organization, 2014.
- 9.
District Health Information System. District Health Information System (Version 2) Overview [August 11, 2017]. https://www.dhis2.org/overview.
- 10.
Lippeveld T. Routine health information systems: the glue of a unified health system. Keynote address at the Workshop on Issues and Innovation in Routine Health Information in Developing Countries, Potomac, March 14–16. Washington, D.C.: JSI, 2001.
- 11.
Hotchkiss D, Diana M, Foreit K. How can routine health information systems improve health systems functioning in low-resource settings? Assessing the evidence base. MEASURE Evaluation Special Report. Chapel Hill, North Carolina: MEASURE Evaluation, 2012.
- 12.
Nutley T. Improving data use in decision making: an intervention to strengthen health systems. Chapel Hill, North Carolina: MEASURE Evaluation, 2012.
- 13. Nisingizwe MP, Iyer HS, Gashayija M, Hirschhorn LR, Amoroso C, Wilson R, et al. Toward utilization of data for program management and evaluation: quality assessment of five years of health management information system data in Rwanda. Global Health Action. 2014;7:25829. http://dx.doi.org/10.3402/gha.v7.25829.pmid:25413722
- 14. Rowe AK, Kachur P, Yoon S, Lynch M, Slutsker L, Steketee R. Caution is required when using health facility-based data to evaluate the health impact of malaria control efforts in Africa. Malaria Journal. 2009;8(1):209.
- 15. Rowe AK. Potential of integrated continuous surveys and quality management to support monitoring, evaluation, and the scale-up of health interventions in developing countries. The American Journal of Tropical Medicine and Hygiene. 2009;80(6):971–9. pmid:19478260
- 16.
Routine Health Information Network. Electronic health management information systems (eHMIS) [February 3, 2017]. http://www.rhinonet.org/rhis-data-sources/ehmis/.
- 17.
Every Woman Every Child and Partnership for Maternal Newborn & Child Health. Progress in Partnership: 2017 Progress Report on the Every Woman Every Child Global Strategy for Women’s, Children’s and Adolescents’ Health. Geneva: World Health Organization, 2017.
- 18.
Trends in maternal mortality: 1990 to 2015: estimates by WHO, UNICEF, UNFPA, World Bank Group and the United Nations Population Division. Geneva: World Health Organization, 2015 9789241565141.
- 19.
United Nations Inter-agency Group for Child Mortality Estimation. Levels and trends in child mortality. 2015 Report. Estimates developed by the UN inter-agency group for child mortality estimation. New York: United Nations Children’s Fund, 2015.
- 20.
Federal Ministry of Health of Nigeria. Nigeria Every Newborn Action Plan: A plan to end preventable newborn deaths in Nigeria. Abuja, Nigeria: Federal Ministry of Health, 2016.
- 21.
Federal Ministry of Health of Nigeria. National health information system strategic plan, 2014–2018. Abuja, Nigeria: Federal Ministry of Health, 2014.
- 22.
National Population Commission of Nigeria. Nigeria demographic and health survey 2013. Abuja, Nigeria: NPC/Nigeria and ICF International, 2014.
- 23.
World Health Organization. Data quality review: a toolkit for facility data quality assessment. Geneva: World Health Organization, 2017.
- 24.
Gombe State Government. History of Gombe State, Nigeria 2018 [March 15, 2018]. http://gombestate.gov.ng/history-2/.
- 25.
Izugbara CO, Wekesah FM, Adedini SA. Maternal health in Nigeria: a situation update. Nairobi, Kenya: African Population and Health Research Center, 2016.
- 26.
National Bureau of Statistics and United Nations Children’s Fund. Multiple Indicator Cluster Survey 2016–17, Survey Findings Report. Abuja, Nigeria National Bureau of Statistics and United Nations Children’s Fund, 2017.
- 27.
Marchant T. Change in maternal and newborn health care. Interactions between families and frontline workers—their frequency, quality, and equity—and coverage of interventions for mothers and newborns. Report from Gombe State, Nigeria, 2012–2015. IDEAS Project Report. London School of Hygiene and Tropical Medicine, 2016.
- 28.
World Health Organization. Service Availability and Readiness Assessment (SARA): an annual monitoring system for service delivery. Reference Manual, Version 2.2. Geneva: World Health Organization, 2015.
- 29. Carvajal-Aguirre L, Amouzou A, Mehra V, Ziqi M, Zaka N, Newby H. Gap between contact and content in maternal and newborn care: an analysis of data from 20 countries in sub–Saharan Africa. Journal of Global Health. 2017;7(2):020501. pmid:29423178
- 30. Ohiri K. An assessment of data availability, quality, and use in malaria program decision making in Nigeria. Health Systems & Reform. 2016;2(4):319–30.
- 31. Yourkavitch J, Zalisk K, Prosnitz D, Luhanga M, Nsona H. How do we know? An assessment of integrated community case management data quality in four districts of Malawi. Health Policy and Planning. 2016;31(9):1162–71. pmid:27162235
- 32. Nicol E, Dudley L, Bradshaw D. Assessing the quality of routine data for the prevention of mother-to-child transmission of HIV: an analytical observational study in two health districts with high HIV prevalence in South Africa. International Journal of Medical Informatics. 2016;95:60–70. pmid:27697233
- 33. Kiberu VM, Matovu JKB, Makumbi F, Kyozira C, Mukooyo E, Wanyenze RK. Strengthening district-based health reporting through the district health management information software system: the Ugandan experience. BMC Medical Informatics and Decision Making. 2014;14(1):40. pmid:24886567
- 34. Mlambo MG, Peltzer K, Koivu A. Prevention of mother-to-child transmission of HIV data completeness and accuracy assessment in health facilities of the Nkangala District. Health SA Gesondheid. 2014;19(1):a774.
- 35. Hahn D, Wanjala P, Marx M. Where is information quality lost at clinical level? A mixed-method study on information systems and data quality in three urban Kenyan ANC clinics. Global Health Action. 2013;6:21424. pmid:23993022
- 36.
Makinde OA, Enemuo J, Adeleke O, Ohadi E, Osika J. Assessment of the routine health management information system in Imo State, Federal Republic of Nigeria. Bethesda, Maryland: Health Systems 20/20, Abt Associates Inc, 2012.
- 37.
Makinde OA, Onazi M, Adeleke O, Ohadi E, Dieng A, Osika J. Assessment of the routine health management information system in Delta State, Federal Republic of Nigeria. Bethesda, Maryland: Health Systems 20/20, Abt Associates Inc, 2012.
- 38.
Makinde OA, Onigbanjo-Williams A, Adeleke O, Ohadi E, Dieng A, Osika J. Assessment of the routine health management information system in Oyo State, Federal Republic of Nigeria. Bethesda, Maryland: Health Systems 20/20, Abt Associates Inc, 2012.
- 39.
Makinde OA, Umar C, Adeleke O, Ohadi , Dieng A, Osika J. Assessment of the routine health management information system in Niger State, Federal Republic of Nigeria. Bethesda, Maryland: Health System 20/20, Abt Associates Inc, 2012.
- 40.
Makinde OA, Adebayo SB, Adeleke O, Ohadi E, Dieng A, Osika J. Assessment of the routine health management information system in Taraba State, Federal Republic in Nigeria. Bethesda, Maryland: Health System 20/20, Abt Associates Inc, 2012.
- 41. Mphatswe W, Mate KS, Bennett B, Ngidi H, Reddy J, Barker PM, et al. Improving public health information: a data quality intervention in KwaZulu-Natal, South Africa. Bulletin of the World Health Organization. 2012;90(3):176–82. pmid:22461712
- 42.
World Health Organization, Resource Centre and Quality Assurance Department of the Uganda Ministry of Health. Assessment of health facility data quality. Data quality report card Uganda, 2010–2011. Geneva: World Health Organization, 2011.
- 43. Bosch-Capblanch X, Ronveaux O, Doyle V, Remedios V, Bchir A. Accuracy and quality of immunization information systems in forty-one low income countries. Tropical Medicine & International Health. 2009;14(1):2–10. pmid:19152556
- 44. Mate KS, Bennett B, Mphatswe W, Barker P, Rollins N. Challenges for routine health system data management in a large public program to prevent mother-to-child HIV transmission in South Africa. PLOS ONE. 2009;4(5):e5483. pmid:19434234
- 45. Makombe SD, Hochgesang M, Jahn A, Tweya H, Hedt B, Chuka S, et al. Assessing the quality of data aggregated by antiretroviral treatment clinics in Malawi. Bulletin of the World Health Organization. 2008;86(4):310–4. pmid:18438520
- 46. Ronveaux O, Rickert D, Hadler S, Groom H, Lloyd J, Bchir A, et al. The immunization data quality audit: verifying the quality and consistency of immunization monitoring systems. Bulletin of the World Health Organization. 2005;83(7):503–10. pmid:16175824
- 47. Chilundo B, Sundby J, Aanestad M. Analysing the quality of routine malaria data in Mozambique. Malaria Journal. 2004;3(3). pmid:14998435
- 48. Roomaney RA, Pillay-van Wyk V, Awotiwon OF, Nicol E, Joubert JD, Bradshaw D, et al. Availability and quality of routine morbidity data: review of studies in South Africa. Journal of the American Medical Informatics Association. 2017;24(e1):e194–e206. pmid:27357829
- 49. Puttkammer N, Baseman JG, Devine EB, Valles JS, Hyppolite N, Garilus F, et al. An assessment of data quality in a multi-site electronic medical record system in Haiti. International Journal of Medical Informatics. 2016;86:104–16. pmid:26620698
- 50. Marchant T, Tilley-Gyado RD, Tessema T, Singh K, Gautham M, Umar N, et al. Adding content to contacts: measurement of high quality contacts for maternal and newborn health in Ethiopia, north east Nigeria, and Uttar Pradesh, India. PLOS ONE. 2015;10(5):e0126840. pmid:26000829.
- 51.
Bryce J, Requejo J, Countdown to 2015 Working Group. Countdown to 2015: tracking progress in maternal, newborn, and child survival. The 2008 report. New York, USA: UNICEF, 2008.
- 52.
World Health Organization. Countdown to 2015: a decade of tracking progress for maternal, newborn and child survival, Final Report. Geneva: World Health Organization, 2015.
- 53.
World Health Organization. Making pregnancy safer: the critical role of the skilled attendant. A joint statement by WHO, ICM and FIGO. Geneva: World Health Organization, 2004.
- 54.
World Health Organization. WHO recommendations on postnatal care of mother and newborn, October 2013. Geneva: World Health Organization, 2013.
- 55.
Federal Ministry of Health of Nigeria. National Routine Immunization Strategic Plan 2013–2015. Nigeria: Federal Ministry of Health, National Primary Health Care Development Agency, 2013.
- 56. Carvajal-Aguirre L, Vaz LME, Singh K, Sitrin D, Moran AC, Khan SM, et al. Measuring coverage of essential maternal and newborn care interventions: an unfinished agenda. Journal of Global Health. 2017;7(2):020101. pmid:29423175
- 57. Moran AC, Marchant T. Measuring coverage of essential maternal and newborn care interventions: an unfinished agenda to define the data matrix for action in maternal and newborn health. Journal of Global Health. 2017;7(2):020307.pmid:29423176
- 58. Akachi Y, Kruk ME. Quality of care: measuring a neglected driver of improved health. Bulletin of the World Health Organization. 2017;95(6):465–72. pmid:28603313
- 59. Moran AC, Jolivet RR, Chou D, Dalglish SL, Hill K, Ramsey K, et al. A common monitoring framework for ending preventable maternal mortality, 2015–2030: phase I of a multi-step process. BMC Pregnancy and Childbirth. 2016;16(250).
- 60. Moxon SG, Ruysen H, Kerber KJ, Amouzou A, Fournier S, Grove J, et al. Count every newborn; a measurement improvement roadmap for coverage data. Special Issue: Every woman, every newborn. 2015;15(Suppl. 2):S8.
- 61. Gimbel S, Micek M, Lambdin B, Lara J, Karagianis M, Cuembelo F, et al. An assessment of routine primary care health information system data quality in Sofala Province, Mozambique. Population Health Metrics. 2011;9(12).
- 62. Gimbel S, Mwanza M, Nisingizwe MP, Michel C, Hirschhorn L. Improving data quality across 3 sub-Saharan African countries using the Consolidated Framework for Implementation Research: results from the African Health Initiative. BMC Health Services Research. 2017;17(Suppl 3):828. pmid:29297401
- 63. Muthee V, Liku N, Puttkammer N. Institutionalization of a routine data quality assessment (RDQA) procedure for improved data quality of electronic patient medical records in Kenya. Annals of Global Health. 2016;82(3):460.
- 64. Wagenaar BH, Gimbel S, Hoek R, Pfeiffer J, Michel C, Manuel JL, et al. Effects of a health information system data quality intervention on concordance in Mozambique: time-series analyses from 2009–2012. Population Health Metrics. 2015;13(1):9. pmid:25821411
- 65. Ledikwe JH, Grignon J, Lebelonyane R, Ludick S, Matshediso E, Sento BW, et al. Improving the quality of health information: a qualitative assessment of data management and reporting systems in Botswana. Health Research Policy and Systems. 2014;12:7. pmid:24479822
- 66. Mutale W, Chintu N, Amoroso C, Awoonor-Williams K, Phillips J, Baynes C, et al. Improving health information systems for decision making across five sub-Saharan African countries: implementation strategies from the African Health Initiative. BMC Health Services Research. 2013;13(2):S9. pmid:23819699
- 67. Admon AJ, Bazile J, Makungwa H, Chingoli MA, Hirschhorn LR, Peckarsky M. Assessing and improving data quality from community health workers: a successful intervention in Neno, Malawi. Public Health Action. 2013;3. pmid:25767750
- 68.
World Health Organization, Department of Planning and Health Information of the Cambodia Ministry of Health. Assessment of health facility data quality. Data quality report card Cambodia, 2012. Geneva: World Health Organization, 2013.
- 69. Brouwer HJ, Bindels PJE, Weert HC. Data quality improvement in general practice. Family Practice. 2006;23(5):529–36. pmid:16868006
- 70. Maokola W, Willey BA, Shirima K, Chemba M, Schellenberg JRMA, Mshinda H, et al. Enhancing the routine health information system in rural southern Tanzania: successes, challenges and lessons learned. Tropical Medicine & International Health. 2011;16(6):721–30.
- 71. Hotchkiss DR, Aqil A, Lippeveld T, Mukooyo E. Evaluation of the Performance of Routine Information System Management (PRISM) framework: evidence from Uganda. BMC Health Services Research. 2010;10:188. pmid:20598151
- 72. Nicol E, Bradshaw D, Phillips T, Dudley L. Human factors affecting the quality of routinely collected data in South Africa. Studies in Health Technology and Informatics. 2013;192.
- 73. Stausberg J, Nasseh D, Nonnemacher M. Measuring data quality: a review of the literature between 2005 and 2013. Volume 210: Digital Healthcare Empowering Europeans. Studies in Health Technology and Informatics. 2015 p. 712–6.
- 74. Matheson AI, Baseman JG, Wagner SH, O'Malley GE, Puttkammer NH, Emmanuel E, et al. Implementation and expansion of an electronic medical record for HIV care and treatment in Haiti: an assessment of system use and the impact of large-scale disruptions. International Journal of Medical Informatics. 2012;81(4):244–56. pmid:22361158
- 75. O'Hagan R, Marx MA, Finnegan KE, Naphini P, Ng’ambi K, Laija K, et al. National assessment of data quality and associated systems-level factors in Malawi. Global Health Science and Practice. 2017;5(3):367–81. pmid:28963173
- 76. Makinde OA, Azeez A, Bamidele S, Oyemakinde A, Oyediran KA, Adebayo W, et al. Development of a master health facility list in Nigeria. Online Journal of Public Health Informatics. 2014;6(2):e184. pmid:25422720
- 77. Glele Ahanhanzo Y, Ouendo E-M, Kpozehouen A, Leveque A, Makoutode M, Dramaix-Wilmet M. Data quality assessment in the routine health information system: an application of the lot quality assurance sampling in Benin. Health Policy and Planning. 2015;30(7):837–43. pmid:25063699
- 78. Omoleke SA, Tadesse MG. A pilot study of routine immunization data quality in Bunza Local Government area: causes and possible remedies. The Pan African Medical Journal. 2017;27:239. https://dx.doi.org/10.11604/pamj.2017.27.239.11875.pmid:28979641
- 79.
Demographic and Health Survey. The Demographic and Health Survey (DHS): survey process Calverton, Maryland: ICF. https://dhsprogram.com/What-We-Do/Survey-Process.cfm.
- 80.
United Nations Children’s Fund. Multiple Indicator Cluster Surveys New York: UNICEF. http://mics.unicef.org/surveys.
- 81. Munos MK, Stanton CK, Bryce J, Core Group for Improving Coverage Measurement for M. Improving coverage measurement for reproductive, maternal, neonatal and child health: gaps and opportunities. Journal of global health. 2017;7(1):010801-. Epub 2017/06/04.