ABSTRACT
Background:
Vaccination coverage is widely used to assess immunization performance but, on its own, provides insufficient information to drive improvements. Assessing the performance of underlying components of immunization systems is less clear, with several monitoring and evaluation (M&E) resources available for use in different operational settings and for different purposes. We studied these resources to understand how immunization system performance is measured.
Methods:
We reviewed peer-reviewed and gray literature published since 2000 to identify M&E resources that include national-level indicators measuring the performance of immunization systems or their components (governance, financing, regulation, information systems, vaccine logistics, workforce, service delivery, and demand generation). We summarize indicators by the system components or outcomes measured and describe findings narratively.
Results:
We identified 20 resources to monitor immunization program objectives and guide national strategic decision-making, encompassing 631 distinct indicators. Indicators for immunization program outcomes comprised the majority (124/631 [19.7%]), largely vaccination coverage (110/124 [88.7%]). Almost all resources (19/20 [95%]) included indicators for vaccine logistics (83/631 [13.2%]), and those for regulation (19/631 [3.0%]) and demand generation (28/631 [4.4%]) were least common. There was heterogeneity in how information systems (92/563 [14.6%]) and workforce (47/631 [7.4%]) were assessed across resources. Indicators for vaccination coverage in adults, data use in decision-making, equity and diversity, effectiveness of safety surveillance, and availability of a public health workforce were notably lacking.
Conclusions:
Between the resources identified in this review, we identified considerable variability and gaps in indicators assessing the performance of some immunization system components. Given the multitude of indicators, policymakers may be better served by tailoring evaluation resources to their specific context to gain useful insight into health system performance and improve data use in decision-making for immunization programs.
INTRODUCTION
Successful immunization programs are strongly associated with lower childhood morbidity and mortality and are an important marker of health systems performance.1–3 As an essential health service, immunization performance is routinely measured by vaccination coverage.1 Despite the increased breadth of protection through new vaccine introduction, coverage rates of vaccines given in infancy have stagnated in the last decade and remain lower than the World Health Organization’s goal of 90%.4,5 Immunization successes gained globally between 2010 and 2019 were adversely impacted by the COVID-19 pandemic.6,7 In 2020, coverage rates dropped to 83% for the third dose of diphtheria-tetanus-pertussis vaccine (DTP3) and 84% for the first dose of measles-containing vaccines8 and then even further in 2021 (DTP3: 81%; first dose of measles-containing vaccine: 81%).9 The number of zero-dose children, defined as infants who did not receive any doses of the DTP vaccine, was reported as high as 18.2 million in 2021.10
Measuring vaccine and zero-dose coverage alone is insufficient to drive improvements in immunization. An immunization system contains health system components, including all the organizations, institutions, resources, processes, and activities involved in the delivery of vaccines.11 Vaccines of assured quality must be available at the point of service, trained health workers must administer vaccines, and the community must demand vaccines and be able to access them.1 Program delivery is underpinned by dedicated financing and governance structures and surveillance mechanisms.1 High-quality and timely data inform where systematic weaknesses exist and how resources should be invested.1,12 System components need to work together effectively to achieve high vaccine coverage.1,13 In rural Madagascar, for example, interventions to improve infrastructure, staff training and availability, and procurement systems, underpinned by data from a new health information system platform, increased full vaccination coverage among children aged 12–23 months from 34.6% in 2014 to 63.6% in 2018.14,15 Initiatives like these demonstrate the importance of assessing immunization system components to target and prioritize strategies to improve overall performance.
Unlike immunization coverage, there is no clear consensus on how to measure the performance of immunization system components. Several monitoring and evaluation (M&E) resources exist to measure the performance of immunization systems or programs and are used for different purposes.16–18 In planning for the integration of COVID-19 vaccination programs into immunization systems, there is a strong focus on integrating not just health services but also health governance functions and on improving the availability and reach of health services across the life course.19 There are opportunities to capitalize on the investments made during the COVID-19 pandemic, such as through leveraging technological advancements, enhancing disease surveillance and monitoring of adverse events following immunization, enumerating and upskilling workforce, and strengthening supply chains for vaccines and other medications.19 Examining the current performance of immunization systems can help countries understand how best to achieve integration and leverage the investments made for the COVID-19 response. This creates a timely opportunity for the review of existing M&E resources. In this study, we examine how immunization system performance has been measured in the past and where gaps exist in measuring the performance of specific components of the system.
Examining the current performance of immunization systems can help countries understand how to achieve integration and leverage COVID-19 response investments.
METHODS
We reviewed existing M&E resources used for immunization systems globally to understand how immunization system performance is measured. We were guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.20
Conceptualization of an Immunization System
We applied the World Health Organization’s (WHO) health system framework to categorize indicators by the components of immunization systems that they intend to measure. The framework describes health systems according to 6 building blocks21 and is frequently used to examine system-wide impacts of initiatives like the introduction of a new vaccine.22,23 The 6 building blocks include workforce, information systems, supply chains, financing, governance, and service provision. Additionally, we included safety and regulation of vaccines and demand generation in our framework because the published literature recognizes these components as critical.24,25
We defined the outcomes of immunization systems as the immediate and short-term effects achieved through the collective and synchronized functioning of immunization system components, namely, to vaccinate populations against a broad range of vaccine-preventable diseases. Impacts of immunization systems were defined as health effects and changes in disease burden associated with vaccine-preventable diseases and overall effects on the health of populations. This conceptualization was the basis for the immunization system framework developed as part of this study (Supplement 1).
Search Strategy
We searched for M&E resources on immunization systems in the peer-reviewed (OVID MEDLINE) and gray literature (Google Scholar) using search terms related to immunization systems, evaluation, indicators, measurement, and surveillance. The detailed search strategy is available in Supplement 2. Articles were also identified by snowballing through references of included publications. Searches of the gray literature were limited to M&E resources published by global partner agencies. We excluded international donors and funders not directly involved in implementing programs due to the differing priorities and motivations of these agencies and potential/perceived conflict of interest. Searches were conducted between March (MEDLINE) and May (Google Scholar) 2022.
Inclusion and Exclusion Criteria
M&E Resources of Immunization Systems
We included contemporary M&E resources published in 2000 or later that examined system-wide immunization performance. Our selection process is illustrated in the Figure, and the inclusion and exclusion criteria are summarized in the Box. We considered a resource eligible if it (1) incorporated national or country-level indicators that measured the overall performance of the immunization system and at least 1 of its components or (2) examined the performance of multiple components of the system. Resources that included only outcome (e.g., vaccination coverage) or impact-level indicators (e.g., disease incidence or mortality) or that included indicators only at the global, regional, or subnational levels were excluded. We excluded resources that focused on a single aspect of the immunization system (i.e., did not examine overall performance or other components of the system) and studies that evaluated a specific intervention or targeted initiative that did not examine system-wide performance (e.g., implementation of a new logistics management system that only evaluated the direct benefits of supply chain management improvement, without consideration of service provision or coverage of vaccines). Similarly, we excluded M&E resources that were focused on evaluating a disease-specific program and did not include indicators examining the impact of the program on the immunization system. Titles and abstracts were screened by 1 author (CP) using Rayyan QCRI.26 Ambiguous resources were discussed with MS. Two authors (CM and RF) who are content area experts also reviewed the list of resources identified to ensure it was comprehensive.
Search Results and Included Resources on Immunization Monitoring and Evaluation
Inclusion and Exclusion Criteria for Review of Monitoring and Evaluation Resources Measuring National Immunization System Performance
Inclusion criteria
Examines system-wide performance of immunization system or the system-wide impact of a disease-specific immunization program or initiative
Includes performance indicators on overall system performance and at least 1 of the system components or examines the performance of multiple components of the system
Incorporates national-level indicators
Published in 2000 or later
Exclusion criteria
Included only outcome (e.g., vaccination coverage) or impact-level indicators (e.g., disease incidence)
Did not include national-level indicators (e.g., resources measuring performance at the subnational level only)
Focused on a single component of the immunization system without examining system-wide performance
Evaluated a disease-specific immunization program, intervention, or targeted initiative without examining its impact on system-wide performance
Immunization System Indicators
Following extraction of all indicators from the included M&E resources, we included all indicators where the value was numeric or categorical (e.g., yes/no or high/medium/low responses). We included indicators measuring performance at a global level that could be modified to the national level, as data for these indicators are typically collected nationally and then aggregated to obtain a global estimate (e.g., “proportion of countries with DTP vaccine coverage ≥80%” was extracted as “national DTP vaccine coverage is ≥80%”). We excluded indicators that were not applicable at the national level related to goals requiring global effort and coordination, such as development of new vaccine platforms. In our examination of distinct indicators, we removed indicators that were duplicated across resources.
Data Extraction and Analysis
We tabulated data on the M&E resources, including the tool’s author, year of publication, purpose, context or location where it was used, and the number of indicators in each tool. We used methods previously used to summarize and compare performance indicators, particularly those used to analyze indicators for health system performance.27,28 We extracted and classified indicators into 10 broad domains aligning with system impacts, system outcomes, and the 8 system components. We further categorized the indicators within each domain into measurement areas to better understand how performance within each domain is assessed (see Supplement 1 for definitions). A formal quality assessment of resources and indicators was not done.
Data were extracted using Microsoft Excel and imported into NVivo 12 for coding. Two authors (CP and NR) independently coded the indicators. Discrepancies were resolved through discussion and consensus. We categorized each indicator into a single domain and measurement area. Where indicators could be categorized under 2 or more system domains, we used contextual information from the source document to understand which domain the indicator was intended to measure. For example, “the proportion of the immunization budget dedicated to advocacy activities” was categorized under demand generation rather than financing because it was used in the source document to measure commitment to engaging with communities to build demand for vaccines.
We used summary statistics to calculate (1) the number of M&E resources, including indicators for each domain and measurement area, and (2) the number of indicators in each domain and measurement area. We summarize our findings narratively.
RESULTS
After screening 3,707 titles in the peer-reviewed literature and 1,000 in the gray literature, we identified 20 M&E resources that met the inclusion criteria (Table 1); 13 from the gray literature (of which 1 was an unpublished draft document)13,17,18,29–38 and 7 from the peer-reviewed literature.1,39–44 M&E resources from the gray literature were published by global partner agencies, including the World Health Organization (WHO) (n=8, 1 in collaboration with UNICEF);13,29–31,33,35,37,38 Gavi, the Vaccine Alliance (n=2);17,36 and the U.S. Agency for International Development (n=2).18,32 The remaining resource from the gray literature was developed by an independent group.34
Summary of Included Publications Reporting Immunization Program Monitoring and Evaluation Resources
Table 1 describes the identified M&E resources across key categories, including purpose of the resource, context where the resource is intended to be used, and the number of indicators. All resources were designed to monitor progress toward immunization program objectives and collate data to guide national-level strategic and programmatic decision-making to achieve goals. Three resources assessed the extent to which specific initiatives (namely, measles and rubella elimination programs and polio eradication programs) improved routine immunization.34,37,42 Three resources examined the extent to which the objectives of global development organizations were achieved.13,17,44 One resource (WHO’s New Vaccine Post-Introduction Evaluation Tool) evaluated the introduction of new vaccines into national immunization programs.33 One resource established for use in the United States40 included indicators that could be applied in other countries but is likely to be of limited use in low-and-middle-income countries (LMICs) due to the differing priorities and degree of maturity of immunization systems compared with high-income countries.
The 20 included resources captured a total of 740 indicators, of which 631 distinct indicators (85%) were retained following exclusion of ineligible and duplicate indicators (Figure and Box). Table 2 presents the number of resources and corresponding distinct indicators covering each domain. Of the 631 distinct indicators, 47 (7.4%) measured system impacts, 124 (19.7%) measured system outcomes, and 460 (72.9%) measured the performance of immunization system components. Three resources—the Immunization Agenda 2030 (IA2030) Monitoring and Evaluation Framework,13 the WHO-UNICEF Joint Reporting Form,29 and the guide for conducting an Expanded Programme on Immunization review31—included indicators that covered all 10 domains. Indicators from these 3 resources comprised almost half of all 740 indicators identified (112, 169, and 85, respectively, a total of 366/740 [49.5%]). A summary of which resources included indicators that measured performance in each domain is in Supplement 3.
Number and Proportion of Included Monitoring and Evaluation Resources and Distinct Indicators Measuring Performance in 10 Indicator Domains
Performance Indicators for Immunization System Impacts and Outcomes
Table 3 summarizes the resources and indicators measuring the outcomes (n=8) and impacts of immunization systems (n=16). Nine resources (45.0%) included indicators measuring system impact, with the majority (39/47 [83.0%]) of indicators measuring incidence, mortality, or disability-adjusted life years attributable to specific vaccine-preventable diseases. In our review, indicators for immunization system outcomes, found in 17 (85.0%) resources, comprised 19.7% of all indicators (124/631). Of the 124 indicators, 110 (88.7%) related to vaccination coverage, specifically coverage of 1 or more specific vaccines in the target population (84/110), equity of coverage (17/110), or dropout rates (9/110). Indicators for childhood vaccination were most common, with 24 of 67 indicators on childhood vaccination coverage specifically examining coverage of DTP-containing vaccines. Indicators for the proportion of children fully vaccinated according to specific schedules or the proportion of children who have not received any vaccine (i.e., “zero-dose coverage”) occurred in resources published since 2015. Only 17 of the 84 indicators on vaccination coverage pertained to vaccine coverage for adolescents or adults. Four resources included 14 indicators assessing success in introducing new vaccines or sustaining the use of a recently introduced vaccine in national immunization programs. Eight of the 14 indicators (57.1%) were included in the IA2030 M&E framework.
Summary of Included Monitoring and Evaluation Resources and Indicators for Measuring Immunization System Impacts and Outcomes
Performance Indicators for Immunization System Components
The number of resources and indicators measuring the performance of immunization system components is summarized in Table 4. Almost all resources covered metrics for vaccine logistics (19/20 [95.0%]). Indicators for workforce, information systems, and governance were each included in 18 resources (90.0%). Indicators for information systems (92/631 [14.6%]), governance (84/631 [13.3%]), and vaccine logistics (83/631 [13.2%]) were the most frequent.
Summary of Included Monitoring and Evaluation Resources and Indicators Measuring the Performance of Immunization System Components
Among indicators for vaccine logistics and supplies, almost half measured availability of supplies at point of service through frequency and duration of stockouts (40/83 [48.2%]). Indicators assessing the performance of cold chain and supply chain management and functionality were also common (37/77 [44.6%]). Among indicators for service provision, those examining the provision of services through the number of vaccination sessions held or the number of doses given were common across resources (39/63 [61.9%]). Nineteen indicators examined integration (i.e., the co-delivery of immunization with other health services), with all but 1 of them from resources published in 2017 onwards. Nearly half of the 84 indicators for governance, planning, and program management (38/84 [45.2%]) assessed whether specific policies, protocols, or processes were in place. Twelve of 20 indicators assessing program management pertained to the existence or features of national immunization technical advisory groups.
Indicators for information systems and workforce varied to a greater extent than the other domains. Under information systems, most indicators measured data quality (37/92 [40.2%]) and the existence of processes to collect immunization (26/92 [28.3%]) and disease surveillance data (24/92 [26.1%]). Although indicators examined whether specific processes, systems, or resources were available in the country, none examined their functionality, use, or acceptance by workers, and only 1 examined whether disease surveillance and immunization data systems were interoperable. Five indicators (5.4%) measured data use. Most indicators examining the performance of workforce pertained to the availability of health workers for immunization (20/47 [42.6%]) or the frequency and quality of supervision (15/47 [31.8%]). Those on health worker availability largely measured the density of the clinical workforce; 3 assessed the availability of workforce supporting nonclinical functions in the immunization system (specifically data management, human resources, and supply chain management), but none assessed the size or capacity of the public health workforce.
Indicators measuring the performance of regulation and pharmacovigilance and demand generation were the least frequent (19/631 [3.0%] and 28/631 [4.4%], respectively). Indicators for pharmacovigilance and regulation appeared in the fewest number of resources (8/20 [40.0%]). Indicators for safety surveillance largely included those for rates of adverse event reporting and whether a system for safety surveillance exists. Under demand generation, most indicators identified focused on the occurrence of community engagement sessions (16/28 [57.1%]). Indicators for assessing the level of demand for vaccines varied substantially but mostly examined if strategies were being implemented to improve communication and demand for vaccines and if systems were in place to measure vaccine confidence.
DISCUSSION
Our study found a multitude of indicators examining the performance of immunization systems and their underlying components, with 631 distinct indicators across 20 M&E resources. We identified variations in how domains were measured, including those where there was some consistency across resources, like those for coverage, service provision, and vaccine supplies and logistics. Although some differences may be due to the differing purposes of the resources or operational contexts in which they are used, the differences and multiplicity of resources reflect the increasing complexity of immunization programs and the changing policies, priorities, and reporting requirements of immunization actors at the global level. Recently, global focus has shifted from achieving high coverage of individual vaccines through disease-specific initiatives to achieving universal health coverage through system-wide strengthening approaches.5,45 Zero-dose coverage has notably emerged as a metric for childhood immunization performance, featured prominently in the IA2030 framework and Gavi’s 2021–2025 strategy.5,13,46
The differences and multiplicity of identified resources reflect the increasing complexity of immunization programs and global changes in policies, priorities, and reporting requirements.
Global attention is also increasing toward expanding immunization across the life span, but our review found that indicators measuring vaccination coverage in adolescence and adulthood are limited in number and scope. Fourteen of 17 indicators examining vaccination after early childhood were identified from the WHO-UNICEF Joint Reporting Form, and 11 of these pertained to influenza vaccine uptake in high-risk populations. COVID-19 vaccination is the first large-scale program targeting adults globally. Before this, 62% (120/194) of countries reported having at least 1 adult immunization program, but high- and upper-middle-income countries were almost 22 times more likely to have such a program compared with low- and lower-middle income countries.47 Our review did not identify any metrics examining the burden of well-established diseases affecting older adults, like influenza, pneumococcal disease, and herpes zoster. Indicators on both disease incidence and disease surveillance systems are focused on childhood diseases and disease-specific programs, such as polio eradication and measles elimination. Lack of data on the disease burden and the potential public health impact of vaccinating against these diseases in LMICs hinders decision-making related to introducing these programs.47,48 Enhancing surveillance for diseases affecting adults and reporting these data stratified by age and sex will require substantial investment, political will, and advocacy at global and national levels, but it is a necessary precursor to introducing adult vaccination programs in LMICs.
We found that the processes by which immunization data are collected, analyzed, reported, and accessed, and the systems used to do so, are not consistently measured across resources. Most indicators examined whether a system was in place but not its functionality, acceptability, or use. Experience with interpreting data collected and reported at a global level highlights the data quality issues that arise. For example, data on 14 of the 27 indicators proposed to measure progress against the goals of the Global Vaccine Action Plan (which preceded the IA2030 strategy) were difficult to interpret due to poor data quality and variability in reporting across countries.49 One indicator in the plan, “immunization coverage data assessed as high quality by WHO and UNICEF,” was abandoned due to an inability to find a suitable measure.49 Data on denominators based on births and deaths statistics are known to be inaccurate and incomplete in LMICs, particularly at subnational levels.12,50 In 2016, 76 of 96 countries reported at least 1 district with DTP3 vaccine coverage greater than 100%.51 Unreliable data quality means that wide variations are difficult to interpret, as it is unclear if the variations are an artifact of the data or if they are true variations.49 Noting the importance of robust information systems to track coverage and other immunization targets,12,50 newer resources are available such as the Data Quality Self-Assessment tool developed by the WHO Immunization Analysis and Insights Unit.52 Despite improved data quality over the last 2 decades, gains were not universal, with resource-constrained countries and those with lower immunization performance continuing to have limited to poorer quality data.53
Although data on numerous indicators are often collected and reported by countries, we identified few indicators in the resources included in our review that measured data-driven decision-making and program planning at the national level. Availability of data does not necessarily translate into action; mechanisms and accountability frameworks to incorporate data into decision-making are needed. Interventions to improve data use can drive improvements in data quality and increase demand for better data.54 A systematic review of the DHIS2 in 11 countries found that access to data increased a sense of ownership and responsibility for the quality of data, fostering a culture of data use and improvement.55 Although the need for including indicators on data use in M&E resources is clear, identifying user-friendly, valid indicators for data use has proven difficult, with no standard or widely accepted approach to defining or measuring data use available.56 Achieving consensus on what constitutes data use can help to develop performance metrics to measure data use and is an area for future research, particularly given the recent focus on improving health information systems through the implementation of digital technologies. One tool that addresses this, albeit within a specialized system component, is the Effective Vaccine Management assessment tool (excluded from our review for reasons described below), as its indicators provide the basis for recommending actions that can lead to cold chain or other supply chain improvements. Although our review focuses on the national level, data use at subnational and local health facility levels is critical to improving coverage, for example, through tracking and identifying defaulters and under-vaccinated populations, improving vaccine supply management and planning of immunization services, and tailoring strategies to build demand for vaccination.12 With the increasing focus on improving the availability and use of subnational-level data by organizations like Gavi,57 the need for performance metrics measuring data use will likely grow.
Although countries collect and report data on numerous indicators, we identified few indicators for data-driven decision-making and program planning at the national level.
We identified few indicators measuring the performance of regulation and pharmacovigilance systems for vaccines, likely because efforts to implement and strengthen national safety surveillance for vaccines are relatively more recent compared to initiatives targeting other components of the immunization system. The impetus to introduce and enhance safety surveillance systems increased for COVID-19 vaccination.58 The rate of adverse events following immunization (AEFI) reporting, measured as the annual AEFI reports per 100,000 surviving infants, has been proposed as a suitable indicator to monitor the performance of AEFI surveillance systems59 and was 1 of the more common of the 9 indicators for safety surveillance identified in this study. The number of countries meeting this reporting target has grown from 80 (41% of 194 countries) in 2000 to 109 (56%) in 2019 but varied by region.60 However, more granular indicators that more accurately capture the quality, functionality (e.g., ability to assess causality of AEFIs), and effectiveness of these systems are needed. We also did not identify any indicators examining the existence or functionality of no-fault vaccine injury compensation schemes, although the call to implement these, particularly in countries with more advanced safety surveillance systems, is growing.61
Our review found that few M&E resources measuring the performance of immunization systems include indicators that measure the confidence in vaccines or social and behavioral drivers of demand for immunization, despite growing concerns over vaccine hesitancy.62 Data on how countries routinely and systematically track vaccine confidence is currently lacking. Measuring vaccine demand has proved to be a challenge; the indicators identified in this review are limited in their interpretation and utility, as it is unclear how concepts such as “supports vaccination” or “lack of confidence” were defined. The lack of consistency in definitions is echoed in a review examining how outcomes in trials of childhood vaccination communication interventions are measured.63 Composite indicators based on qualitative data collected via the WHO-UNICEF Joint Reporting Form were the primary way to monitor vaccine demand globally64 but were criticized for the lack of clarity in defining what was being measured.49 In another review, 12 of 14 measures of confidence in childhood vaccination identified were developed and validated in high-income countries.65 The recently published resources for behavioral and social drivers of vaccination66 provide a framework to assess reasons for under-vaccination, which may address the gaps and lack of consistency in metrics for this domain and can contribute to formative work on program strengthening.
Indicators examining equitable coverage largely focused on disparities by geographical areas (such as districts), and 5 examined coverage by socioeconomic status. Our review identified some disease burden indicators disaggregated by sex but none for coverage or any that examined or collected data on the role of gender and diversity in making decisions about or implementing immunization programs. Despite known differences in immune responses to certain infections and adverse events following immunization, studies of vaccine effectiveness and safety often do not report results by gender.67,68 Evidence suggests that gender influences health status, access to resources and health services, decision-making autonomy, and the quality of health services.69 Yet, only 5 of 58 countries with COVID-19 vaccine policies in March 2021 referred to gender, and 34 of 180 countries reported sex-disaggregated data on COVID-19 vaccine coverage between April and May 2021.70 The absence of indicators in our study on disability, diversity, and inclusion was also a gap. Achievement of the IA2030 goals of reducing zero-dose children and improving equity of coverage across the life span will require identifying predictors of low coverage and selecting pro-equity strategies to address disparities, particularly in the context of COVID-19-related disruptions to immunization that have disproportionately affected poorer and more vulnerable populations.71 The behavioral and social drivers of vaccination tools66 help to fill this gap, as they collect data on gender and other demographic data to help explain what drives vaccine uptake. Further work to address this gap is needed, as is the need to understand the role of gender, diversity, and inclusion in strengthening decision-making about immunization.
The absence of indicators in our study on disability, diversity, and inclusion was a gap that requires further work to address.
Although 11 of the 17 indicators examining integration of immunization services related to co-delivery of immunization alongside other health services (e.g., antenatal care) or in nontraditional settings (e.g., schools), it is unclear if this is the best way to measure integration of primary care services.72 True integration to achieve universal health coverage requires integration across the system in planning, financial resourcing, training and supervision, and community engagement, and current indicators to measure integration do not reflect this.73 The use of an integration index has been proposed as a way forward72 but runs the risk of measuring concurrent performance across programs without actually measuring the extent to which programs are integrated. Future efforts to define indicators should also consider quantifying the health and efficiency gains that are expected to occur following integration.
The M&E resources in this review did not identify any indicators examining the capacity or competency of the public health workforce or the surge clinical capacity available to draw on in the event of a public health emergency. The need for defining, mapping, and measuring the workforce, including the public health workforce, is recommended in WHO’s Global Strategy on Human Resources for Health 2030.74 A framework of immunization workforce competencies, such as WHO’s roadmap for public health workforce developed by Traicoff et al.,25 can help countries to establish what the minimum skills and competencies needed are and track suitable indicators and numbers of staff trained in those technical competencies. The Joint External Evaluation Tool, which aims to support countries to develop their capacity to prevent public health threats, provides additional metrics and targets for the public health workforce and surge capacity required, particularly in the context of a public health emergency.75 Incorporating these core competencies into public health and clinical training programs can secure surge capacity in the event of future acute emergencies and increase the skills mix of primary health care providers, contributing toward the goal of universal health coverage.
Our study identified 47 distinct indicators across 9/20 (45.0%) M&E resources that measure the impact of immunization programs on disease burden and elimination targets. This small number is likely due to (1) our inclusion criteria and focus on metrics assessing immunization system performance and (2) exclusion of resources that did not measure the operational aspects of immunization systems. Although the indicators identified disease-specific and some summary metrics of disease burden and disease elimination and eradication targets, metrics evaluating wider economic and societal benefits, such as reductions in poverty, improvements in productivity, and financial returns on investment,76 were absent. Including these metrics in evaluations of immunization system performance can help to build the case for financial investment in immunization systems and assist decision-making through comparisons with other initiatives or packages of interventions. However, the societal and financial impacts of immunization have often been estimated through modeling studies,77,78 and it may be challenging for resource-restricted countries to conduct these analyses in the short term.
Strengths and Limitations
To our knowledge, this is the first published review of indicators used to measure system-wide performance of national immunization systems that includes M&E resources published over 2 decades. This review intentionally focuses on M&E resources used globally. Previous global immunization strategies were viewed as top-down strategies, but there is growing recognition that country-specific goals and targets are needed.4 Our review collates the indicators used to measure immunization system performance and provides a starting point for national governments and partner agencies to consider how best to assess performance in their own contexts. Additionally, it draws attention to gaps in indicators used to date, highlighting where development partners and researchers can focus their efforts to develop and validate indicators.
Our review provides a starting point for national governments and partner agencies to assess immunization system performance in their own contexts.
Importantly, our study highlights the large burden of data collection and routine reporting for immunization alone, with 169 indicators identified in the 2019 WHO-UNICEF Joint Reporting Form and 112 indicators for national monitoring in the IA2030 M&E framework. We removed 93 duplicate indicators identified in our review but were still left with 631 indicators. Many indicators measured the same construct but in slightly different ways, which means that data collected with 1 resource may not be appropriate for use with another resource, resulting in duplication of efforts if reporting on those indicators is required by development or financial partners. The workload associated with collecting and reporting data using different M&E resources is significant, especially in resource-constrained settings, where dedicating resources to meet reporting requirements has opportunity costs.12,79,80 Data collection, analysis, and reporting require dedicated financing, resourcing, and accountability mechanisms, potentially diverting resources from other areas of the health system. Countries with constrained systems likely have a greater proportion of their immunization system costs covered by donors and international partners and thus have greater data reporting responsibilities despite having fewer resources to do this. Previous M&E resources have also taken a top-down approach that does not account for variability in country contexts, resulting in reduced country commitment to achieving global immunization goals.4 Further research and coordination on which indicators are most important to measure uniformly across all countries is needed, with clear justification of how these indicators are linked to public health benefits to justify resourcing to collect and report this data. For the remainder, it may be more suitable to identify and focus on key indicators based on the values and challenges of the local context, which aligns with the IA2030 strategy’s call for countries to select indicators based on their needs.13
Although we systematically searched the literature, we identified only 6 peer-reviewed publications. This is unsurprising as the majority of widely used M&E resources are published in the gray literature by global partner agencies. The Medical Subject Headings used in the MEDLINE database for this topic area are not well-defined and provide a substantial volume of false results. A further limitation is that we only included resources that were publicly available, whereas many countries may report on performance directly to their funders. We ensured that our search strategy captured the Medical Subject Headings against which the identified articles were indexed, and 2 authors of this article are content area experts who checked that we included all major or influential M&E resources. We excluded some widely used and well-established resources, particularly those published by the WHO’s Immunization Analysis and Insights group, such as the Effective Vaccine Management assessment tool and vaccine-specific post-campaign assessment resources (e.g., for COVID-19 and influenza vaccines). Some of these were focused on a specific aspect of immunization systems rather than system-wide performance or had a disease-specific or context-specific focus. M&E resources on child health programs, such as MEASURE Evaluation’s A Guide for Monitoring and Evaluating Child Health Programs,81 were not included due to their focus on child health more broadly, of which immunization is a smaller component, and on measuring outcomes of the system rather than the performance of its components. Nevertheless, these are important assessment resources whose use can contribute toward strengthening immunization system components.
We did not conduct a quality assessment of indicators using a standardized quality assessment framework. Not all of the indicators were clearly defined, and few were accompanied by a data dictionary clarifying how to calculate statistics and where to source data. It is possible that some indicators may be interpreted differently depending on the context. Although most resources were developed through iterative and consultative processes, it is unclear if indicators have been validated and are associated with improvements in performance. Thus, the existence of M&E resources and indicators does not necessarily mean they are used or provide insightful information. It is likely that countries are tailoring assessments to suit their contexts and measuring indicators that best match their goals and available resources. Furthermore, performance indicators often have limited use in understanding the drivers of performance. Several M&E resources provide guidance on using other data collection methods, such as interviews and focus groups, to generate information to supplement quantitative indicators that together can inform future actions. Further research is needed to better understand how these evaluation resources are used, whether they drive improvements in performance and the pathways for doing so, and which system-level indicators most closely correlate with improvements in vaccine coverage, equity, and reductions in disease. Measuring implementation strength (i.e., the amount of input or activity to support program implementation),82 a construct that did not appear in the resources included in our review, may be another way to cumulatively interpret data from selected indicators but requires research to determine which group of indicators can do so most effectively. Developers of M&E resources should assess the quality of indicators included to ensure they are measurable, easy to interpret by users, linked to performance improvements, and relevant to decision-making.
Developers of M&E resources should assess the quality of indicators included to ensure they are measurable, easy to interpret, linked to performance improvements, and relevant to decision-making.
We elected to use the WHO health system framework as the basis for the components of immunization systems, which we used to categorize indicators. This framework has rightly been criticized for focusing on the infrastructural and resource-related components of health systems and insufficiently accounting for the interdependencies between components as well as intangible components such as trust and social value.83 In our analysis of indicators, we found that several indicators could have been classified under more than 1 system component, reflecting the interdependencies and connectedness of the components underlying the immunization system. The indicators we identified did not explicitly examine the interactions between components. As discussed earlier, the quality of the indicators was not assessed, and it is unknown if performance of 1 component is linked to or predictive of performance of another. Nevertheless, this framework provides a useful basis to describe the components that describe the health system and its basic functions and has been used to examine the impact of various initiatives on the immunization and health system.22,23
Finally, the inductive method we used to identify measurement areas based on the themes that emerged from our review introduced unavoidable risk of measurement and selection bias. As content area experts, we acknowledge that our prior experiences, assumptions, and beliefs have the potential to influence the research process. We have attempted to minimize this bias by having 2 authors independently code indicators and agree on the definitions set out in Supplement 1.
CONCLUSIONS
This review identified a multitude of indicators to measure immunization system performance. We identified heterogeneity in metrics assessing the performance of some immunization system components; further studies are needed to reach consensus on how to measure performance in these areas. This summary of indicators can inform country-specific approaches to measuring system performance moving forward, particularly as countries seek to evaluate the impact of the COVID-19 pandemic and leverage the investments made during the response. As countries look to identify focus areas for improvement, they can benefit from selecting indicators that are aligned with national goals, values, and priorities and establishing accountability frameworks to monitor performance and take action. Establishing a country-focused set of core indicators and improving the quality of data on these select metrics will position decision-makers to have better access to data that is useful in decision-making, enabling countries to achieve their immunization goals.
Funding
No specific funding was sought for this project. CP is supported by an Australian Government Research Training Program Scholarship.
Author contributions
All authors meet the ICMJE authorship criteria. CP and MS conceived the study idea and jointly developed the study methodology. CP conducted searches, screening, and data extraction, with MS, CM, and RF contributing to identifying suitable monitoring and evaluation tools. CP and NR coded indicators with support from MS. CP analyzed and synthesized the findings. CP drafted the first draft of the article. All authors contributed to interpreting study findings and read the draft, provided comments, revised, and agreed on the final version.
Competing interests
None declared.
Acknowledgments
We thank Dr. Catherine King for assistance in establishing the search strategy.
Notes
Peer Reviewed
First published online: June 7, 2023.
Cite this article as: Patel C, Rendell N, Sargent GM, et al. Measuring national immunization system performance: a systematic assessment of available resources. Glob Health Sci Pract. 2023;11(3):e220055. https://doi.org/10.9745/GHSP-D-22-00555
- Received: December 22, 2022.
- Accepted: May 16, 2023.
- Published: June 21, 2023.
- © Patel et al.
This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly cited. To view a copy of the license, visit https://creativecommons.org/licenses/by/4.0/. When linking to this article, please use the following permanent link: https://doi.org/10.9745/GHSP-D-22-00555
REFERENCES
- 1.↵
- 2.
- 3.↵
- 4.↵
- 5.↵
- 6.↵
- 7.↵
- 8.↵
- 9.↵
- 10.↵
- 11.↵
- 12.↵
- 13.↵
- 14.↵
- 15.↵
- 16.↵
- 17.↵
- 18.↵
- 19.↵
- 20.↵
- 21.↵
- 22.↵
- 23.↵
- 24.↵
- 25.↵
- 26.↵
- 27.↵
- 28.↵
- 29.↵
- 30.
- 31.↵
- 32.↵
- 33.↵
- 34.↵
- 35.↵
- 36.↵
- 37.↵
- 38.↵
- 39.↵
- 40.↵
- 41.
- 42.↵
- 43.
- 44.↵
- 45.↵
- 46.↵
- 47.↵
- 48.↵
- 49.↵
- 50.↵
- 51.↵
- 52.↵
- 53.↵
- 54.↵
- 55.↵
- 56.↵
- 57.↵
- 58.↵
- 59.↵
- 60.↵
- 61.↵
- 62.↵
- 63.↵
- 64.↵
- 65.↵
- 66.↵
- 67.↵
- 68.↵
- 69.↵
- 70.↵
- 71.↵
- 72.↵
- 73.↵
- 74.↵
- 75.↵
- 76.↵
- 77.↵
- 78.↵
- 79.↵
- 80.↵
- 81.↵
- 82.↵
- 83.↵
Statistics from Altmetric.com
Jump to section
Related Articles
Cited By...
- No citing articles found.