Early Lessons From Ethiopia in Establishing a Data Triangulation Process to Analyze Immunization Program and Supply Data for Decision Making

Health managers in Ethiopia used a data triangulation tool and process to prompt decision making and meaningful actions to improve immunization services. Immunization managers interested in incorporating data triangulation analyses into their current data review systems will need to determine practical, feasible means for operationalization, preferably through a collaborative, iterative process with users.


BACKGROUND
S trengthening data use and quality is critical to achiev- ing high, equitable immunization coverage, both globally and in Ethiopia, which ranks among the top 10 countries with the most under-and unvaccinated children. 1 Streamlining data systems, promoting evidencebased decision making, improving data visibility, and strengthening feedback systems are all efforts that have been noted as key to progressing on global and country-level strategies, including Gavi's 5.0 strategy, 2 the World Health Organization's (WHO) Immunization Agenda 2030, 3 and Ethiopia's Health Sector Transformation Plan. 4 Data triangulation-the synthesis of 2 or more existing data sources to address relevant questions for program planning and decision making 5 -is being increasingly recognized as a means to improve data use and quality in public health programs, including immunization, and to improve program decisions.Evidence has shown that data quality can improve through programs actually using data, 6 so by triangulating several pieces of data, even if they are imperfect, decision makers can gain an additional layer of insight and understanding to improve their basis for decision making. 5One common way to triangulate data is by comparing health program data with product/ commodity supply data.For example, in Laos, data on rates of malaria cases and testing is compared with stock data to understand what health facilities' needs are for malaria commodities. 7hile there are many potential ways to triangulate immunization data, documentation is limited on the specific techniques or processes undertaken by ministries of health.
In Ethiopia, administrative data for immunization-the data collected and reported through the government health system-has ongoing quality challenges, including timeliness, completeness, and accuracy. 8,9For example, administrative data can differ considerably from survey estimates of immunization coverage, with administrative data often skewing higher (e.g., administrative coverage for the third dose of the pentavalent vaccine in 2019 was 96%, while the WHO/UNICEF estimate was 69%). 10 In addition, some indicators relevant to programmatic decision making are reported through different systems to different responsible departments, such as immunization coverage data that are reported up through District Health Information System 2 (DHIS2) and vaccine supply related indicators, which are reported through a logistics management data system called mBrana.Coordination is limited between program staff at the Expanded Programme on Immunization (EPI) within the Ministry of Health (MOH), who generally focus on DHIS2 data, and supply chain staff within the Ethiopian Pharmaceutical Supply Agency (EPSA), who generally focus on mBrana data.Limited coordination has meant that regional EPI officers do not consistently incorporate supply chain data into their decision-making processes, and review or use of data overall for program improvement is not always consistent.
JSI, through the Universal Immunization through Improving Family Health Services (UI-FHS) project, supported the MOH over the past decade to improve the EPI, including providing technical assistance to 5 of Ethiopia's 11 regional health bureaus (RHBs) since 2018.The RHB in Benshangul Gumuz (BG) region requested support in 2019 to improve the use of immunization program and supply data for programming and decision making.We describe JSI's experience and lessons learned from introducing a process and tool for triangulating immunization program and supply data for use at the regional level to support regular feedback and action at the district level.It contributes to the literature on and practical experiences with triangulating data for data use and decision making.

DEVELOPMENT OF DATA TRIANGULATION TOOL AND DATA REVIEW PROCESS
JSI held a consultative workshop with BG RHB stakeholders to develop a practical approach and structured process for using and analyzing triangulated program and supply data on a regular basis.The group discussed specific indicators that could be compared or analyzed together to provide greater insight into program performance, at what level this data triangulation process should occur (at the regional level on a monthly basis to bring together cross-departmental staff who could take feedback to the districts), what review processes should look like, and how to merge and analyze the data.Since time and technical capacity to integrate disparate datasets were limited, stakeholders determined that developing a tool to automate analysis would be critical to facilitating regular review of triangulated data.
In collaboration with RHB stakeholders, JSI codeveloped a Microsoft Excel-based Immunization Data Triangulation Tool (IDTT) (Supplement 1).The tool was designed to triangulate service statistics indicators from DHIS2 and supply chain indicators from mBrana, the logistics management data system to provide a comprehensive picture of immunization program and supply chain data in a userfriendly way.Several indicators were compared against each other to produce ratios that provide insight into data quality issues between stock delivered and consumed or programmatic issues such as consistent understock; more detail on the indicators By triangulating several pieces of data, even if they are imperfect, decision makers can gain an additional layer of insight and understanding to improve their basis for decision making.
The Immunization Data Triangulation Tool was codesigned with RHB stakeholders to provide a comprehensive picture of immunization program and supply chain data.used is included in Supplement 2. Coverage for all antigens was also included in the IDTT, as well as WHO Reaching Every District categorization scores, which classify performance based on pentavalent vaccine coverage and dropout. 11An IDTT user guide (Supplement 3) was also developed, which provided information on the rationale and process for conducting data triangulation, provided guidance on the data review process, and instructed users in a step-by-step manner on the use of the IDTT Excel tool.
Using color coding, IDTT dashboards highlighted districts that require programmatic attention and those that are performing well.The IDTT provided decision-support features, such as scoring based on performance and suggested actions managers should take based on scores.The tool was thus meant to trigger actions by managers across departments to support changes at the district level, such as examining discrepancies between vaccine doses supplied and administered.The Figure outlines the structured data review process that was used to facilitate data use, provide a collaborative forum through which staff from multiple departments could identify action points, and track improvements and potential outcomes based on action taken.
JSI rolled out the IDTT and data review process in 2 management units: at the RHB level in BG region, and the zonal health department (ZHD) level (a management level between regional and district) in Kembata Tembaro (KT) zone within the Southern Nations, Nationalities, and Peoples Region (SNNPR).Both were within the project's overall scope geographically, and they were chosen to implement this activity because of their expressed interest and the desire to test the tool/ process at 2 different management levels.Rollout included several activities.Upon finalizing the tool's development, JSI provided an initial indepth orientation and distributed the IDTT user guide.Then, ongoing follow-up support was provided throughout implementation, eliciting users' feedback on the tool through individual and group discussions and helping facilitate monthly data review meetings led by the RHB and ZHD.Monthly meetings took place from October 2020 to May 2021 for KT ZHD and March to May 2021 for BG RHB.

PROCESS DOCUMENTATION OBJECTIVE AND METHODS
The process documentation aimed to understand the feasibility of the IDTT's application as a decision-making tool by examining its use during review meetings.The learning particularly focused on examining the IDTT's use in crossdepartmental problem solving, the decisions made based on the data analysis, the actions taken (e.g., supportive supervision conducted), and the outcomes of those actions (e.g., improved availability or quality of data).This information would be used to share lessons learned on if and how the IDTT tool and process could address problems within the immunization program.
The process documentation applied mainly qualitative methods to understand the feasibility, benefits, and challenges regarding the data triangulation process and use of the tool.We conducted key informant interviews (KIIs) with representatives of the RHB and ZHD, EPSA, health management information system, and JSI project staff involved in this activity.A team of experienced local researchers conducted 14 KIIs (7 in each region, BG and SNNPR) in Amharic using semistructured openended guides.KIIs were transcribed and translated into English, and a simple text analysis was conducted following a list of themes and codes aligned with the study's objective.In addition, we examined the data uploaded in the IDTT and meeting minutes over the period of implementation to track the data being uploaded and the decisions that were being made as a result of data review.

Ethical Approval
We obtained ethical clearance from JSI's institutional review board and additional clearances to conduct KIIs from both regions' RHBs.Before the interviews started, all participants gave their verbal consent.

FINDINGS 1. Perceptions of Early-Stage Effectiveness
While the early stage of introduction differed in each region, stakeholders from both regions recognized the value of using triangulated program and supply data.
The KT ZHD demonstrated faster uptake of the tool and approach, leading to more regular use of the tool for decision making, consistent data review meetings, and increasing ease of using the tool.Members attending the triangulation data review meeting on a monthly basis included EPI staff; broader maternal, newborn, and child health (MNCH) staff; and logistics staff (who communicate with regional-level EPSA staff).Respondents reported that during IDTT data review meetings, The KT ZHD demonstrated faster uptake of the tool and approach, leading to more regular use of the tool for decision making, consistent data review meetings, and increasing ease of using the tool.
committee members interacted and examined the data holistically.For example, logistics staff responsible for reviewing immunization supply and stock data communicated directly with EPI staff, whose usual concerns were coverage data from DHIS2.
Respondents reported that simultaneous review of immunization coverage and supply data

FIGURE. Conceptual Framework for Data Triangulation Data Review Process in Ethiopia, Including Illustrative Actions and Outcomes
Abbreviations: DHIS2, District Health Information System 2; EPI, Expanded Programme on Immunization; EPSA, Ethiopian Pharmaceutical Supply Agency; IDTT, Immunization Data Triangulation Tool. 1 Stock on hand and vaccine issue data from mBrana logistics management system. 2 Immunization coverage data from DHIS2.
was a new process; previously, the process had focused mainly on reviewing coverage data.Based on the decisions made in the data review meetings, follow-ups for the action items were conducted during supportive supervision and/or by directly communicating with districts, and updates were shared in subsequent data review meetings.
Committee members also reported increased comfort and ease with using the tool-such as mastering data import, navigating dashboards, and understanding the outputs from analyses-as they continued to use it over time.
The BG RHB experienced a slower roll-out.Since the process documentation exercise occurred 3 months after the IDTT tool was rolled out, the activity in BG was still in a nascent stage.Despite the slow start, respondents reported beginning to populate the tool and meeting as a committee.Some key discussion points focused on missing supply data (region-wide issue) and missing program data (from districts experiencing security issues).For the districts experiencing security issues and missed reporting, meeting minutes showed the intention to develop plans based on the affected districts' needs.However, due to the coronavirus disease (COVID-19) and ongoing security threats, these plans had not yet been executed.Although full use of the tool was constrained by lack of stock data, respondents expressed interest in continuing to use the IDTT with improved supply data.
Many respondents in both regions reported clear benefits to using the tool and data review process, including respondents in BG who had only recently initiated the activity.Respondents from both regions noted the following benefits.
The tool allowed for easy comparison and analysis of data from different sources in a single platform; data were more accessible, and the tool eased data management for decision making, thereby reducing workload.
Synthesis of data and information helped identify districts that were functioning better and those that were not, and regular updating of the tool helped monitor progress over time.
The tool and data review process helped improve and maintain data quality, which is an important policy priority for the MOH.
Color coding used in the tool's dashboards helped visualize data and eased interpretation of data.
The trend of triangulating supply chain data and the service data was not known previously.We can now monitor monthly improvements in the immunization program and take immediate actions to address the gaps.Outbreaks have reduced in our zone.-SNNPR Public Health Emergency Management representative

Barriers to Effective Implementation During Roll-Out
BG experienced greater challenges with introducing the tool.EPSA, a critical partner responsible for ensuring the timely supply of vaccines, was unable to participate due to competing priorities.Recurring conflict/ethnic clashes and competing priorities such as COVID-19, limited the RHB staff's availability to engage in IDTT implementation and affected the availability of data.These issues affected the functionality and performance of the region's immunization program as a whole and caused a delay in the data triangulation activity's implementation, which started in March 2021.
In both regions, mBrana often had incomplete supply chain data, which led staff to focus more on indicators/analysis based on program data, while at the same time raising the issue of incomplete or missing vaccine supply data for discussion and action at higher levels.KIIs indicated that committee members discussed data gaps and developed action plans collaboratively.
We download the DHIS2 [and mBrana] data, and then one of us displays the results over a projector. . .We then discuss each finding to identify if there is low coverage or dropout rates for each woreda (district).We then provide feedback to the woredas based on findings; if good, we just write feedback to keep up the good deeds; or if poor performance, we provide action points for improvement depending on the problem.-EPI focal person Respondents from BG RHB reported more challenges than respondents from KT ZHD.This aligns with the finding that familiarity and comfort with the tool increased over time and the fact that the activity was still in an early stage in BG.Overall, respondents reported indicator-or analysis-specific challenges or operational hurdles.Respondents reported some of the following challenges.Some respondents desired additional indicators (e.g., raw numbers of unimmunized children in addition to percent coverage) or for indicators to be summarized in a different manner (e.g., quarterly summary in addition to monthly analysis).
In both regions, mBrana often had incomplete supply chain data, which led staff to focus more on indicators/ analysis based on program data.
The tool was perceived as somewhat complex; interpreting output values takes time and practice to understand.
The tool requires some knowledge of Microsoft Excel and computer skills.

Demonstrated Potential for Impact
One important recognition that came out of the early stages of introduction was the importance of having data available, particularly supply data.Before commencing the activity in BG, the RHB and EPSA took steps to improve data completeness in mBrana, and to some extent in DHIS2, as reporting to these systems had lapsed considerably in the previous 6 months due to conflict and other issues.This initial step was needed to ensure adequate data were available to begin to triangulate and analyze.While implementing the IDTT and data review process in both regions, improving the availability of vaccine supply data continued to be a focus to strengthen data systems.
One documented outcome was an expansion of outreach (community-based) immunization sites.By using the IDTT tool, the KT ZHD team identified lower-than-expected coverage in a town despite adequate vaccine supply.The team communicated with EPI officers in the town and identified the reasons, which were poor planning and a shortage of outreach sites.As a result, the ZHD, in collaboration with the district health office, decided to launch new outreach sites.
Another documented outcome was the redefining of a target population.The data review process revealed that a small town in a more urban area demonstrated poorer performance compared to its neighboring communities.The KT ZHD team investigated the cause and found that some neighboring rural areas with low coverage had been recently rezoned into the town, which resulted in a high number of unimmunized children.The EPI team took immediate measures and adjusted its plan to redefine the target population.

LESSONS LEARNED
This experience provided lessons for establishing a systematic and recurring process to use triangulated data in immunization and for introducing tools and processes into health systems.
Establishing Regular Processes for Using Triangulated Data in Immunization 2. Regular availability of data is important to fully leverage the benefits of triangulation: To optimally triangulate data, it is essential to ensure regular reporting of data to the data systems that will be used for triangulation analysis; otherwise, the indicator measures could fall short of achieving desired objectives.Efforts to improve and sustain data availability/completeness should be prioritized.
3. Subnational managers could use data triangulation to monitor issues at the district level: Given that data triangulation can be complex, having subnational managers use data triangulation as a management tool at the regional level to address problems happening at the district level showed some success.This takes ongoing practice and support.

Introducing Tools and Processes Into Health Systems
1. Design with and for the users: The stakeholders were engaged from the beginning to codesign a tool with features that would address their gaps/needs.In addition, the process was designed to encourage crossdepartmental collaboration yet remain feasible to implement for regional staff.This helped promote uptake, as stakeholders recognized the value and potential from the beginning.

WAY FORWARD AND RECOMMENDATIONS
Despite limitations in data availability, the process documentation showed some evidence that the use of the IDTT and data review process improved decision making and spurred actions that improved the management of immunization services at the district and health facility levels.Respondents also noted the value of regularly using triangulated data.KII respondents provided the following recommendations on the data triangulation tool and process.
National-level MOH officials should institutionalize the tool for use at regional and zonal management levels.
Formalize clear tasks/responsibilities of every data triangulation committee member from across departments.
Improve the availability of vaccine supply data; more human and financial resources are needed to support this process.
The IDTT and review meeting process described in this article represents an approach to introducing a regular data triangulation process for immunization programs.These early findings and lessons related to introducing the tools and establishing regular processes show promise in the ability of EPIs to successfully use triangulated data to address challenges.The IDTT process facilitated collaboration and decision making, as well as breaking down of silos; more data use and attention to data availability and quality; and some demonstrable actions that improved the management of immunization services.More work needs to be done to document this kind of work, including at more advanced stages of implementation occurring over a longer period.If data triangulation processes advance and scale, they should be integrated into current health information systems, such as DHIS2, to be properly institutionalized and sustained.