Digitalizing Clinical Guidelines: Experiences in the Development of Clinical Decision Support Algorithms for Management of Childhood Illness in Resource-Constrained Settings

Clinical decision support systems can strengthen the quality of IMCI but can vary because of the need for interpretation when translating narrative guidelines into decision logic combined with considerations of context and design choices.


INTRODUCTION
P oor quality of care results in an estimated 5 million excess deaths annually in low-and middle-income countries. 14][5][6] Focused on common conditions contributing to the highest burden of morbidity and mortality, the IMCI strategy may reduce child mortality by 15% when fully implemented. 3,6Since its launch in the 1990s, the IMCI strategy has been rolled out to more than 100 countries. 3,4But adherence by health care providers, faced with the difficult task of providing care with limited training and resources, remains low in many countries. 7,8][17] Differences in approach to and documentation of these decisions have resulted in wide variation in CDSS quality and transparency. 18,19Multisectoral experts have highlighted the particular importance of transparency of the clinical algorithms used in CDSSs to ensure stakeholders are able to understand if (and how) the decision logic reflects or differs from evidence-based recommendations in guidelines and peer-reviewed literature. 20n recognition of these challenges, WHO recently launched the SMART (Standards-based, Machine-readable, Adaptive, Requirements-based, and Testable) Guidelines initiative.The SMART Guidelines provide a standardized approach to the digitalization of WHO recommendations, with the aim of facilitating more rapid and effective uptake of evidence-based practice. 19Elaborating on earlier work by Boxwala, 16 the WHO SMART Guideline framework is organized in different "knowledge layers" (Box 1). 19,21Since 2021, digital adaptation kits (Layer 2) have been published for antenatal care, family planning, and HIV, with forthcoming kits planned for immunization and child health in humanitarian settings. 224][25][26][27][28][29] Having been developed by various groups and for a wide range of contexts, the resulting clinical algorithms (Layer 2) differ despite a shared starting point.
In this technical note, we share lessons learned by the eIMCI working group-a collaboration of IMCI-related CDSS practitioners, scientists, and policy actors-and highlight considerations when developing clinical algorithms (i.e., from Layer 1 to Layer 2).Though not representative of all IMCI-related CDSSs, the group-formed through Geneva Health Forum CDSS sessions-includes representatives from organizations involved in the development of clinical algorithms for 4 major child health CDSSs implemented in collaboration with ministries of health, nongovernmental organizations, academic institutes, and other stakeholders in 15 countries.
Through a series of 10 formal working group meetings, informal discussions, and written exchanges via semistructured questionnaires and work on shared documents between October 2019 and April 2021, we compared the clinical algorithms of these 4 CDSSs with each other and with paper-based IMCI guidance.The comparison was structured around the following axes, defined iteratively over the course of group discussions: (1) context and objectives of the interventions CDSS support, (2) adaptations relative to the WHO IMCI chart booklet, 5 and (3) methods and processes of CDSS development.

CDSS OBJECTIVES AND CONTEXTS OF DEVELOPMENT AND IMPLEMENTATION
The CDSSs we compared are all knowledge-based systems (i.e., rule-based, rather than "nonknowledge-based," which can extract rules using machine learning), given our focus on translation from clinical recommendations to decision logic.They all provide step-by-step guidance for consultations for sick children in facility-based primary care in resource-constrained settings to enable CDSSs provide tailored guidance to support health care providers' adherence to guidelines; however, substantial heterogeneity in acceptability, uptake, and impact has been found.comparability (rather than those developed for only a portion of the consultation such as medication dosing, for well children, or for use at the community level based on integrated community case management). 30[31][32][33][34][35][36][37][38][39][40][41][42] The developing and implementing organizations also acknowledged that they aimed to leverage CDSS potential to: (1) be updated more readily than paper-based guidelines; (2) contribute to health worker development of knowledge and skills through onthe-job training; and (3) enhance the quality and accessibility of data for decision-making and feedback to health care providers.
Although these tools were developed with shared aims, they were developed and adapted for different contexts.The organizations that initiated their development have taken different intervention approaches and reached different scales in field implementation (Table 1).

IDENTIFIED ADAPTATIONS FROM THE IMCI CHART BOOKLET
Differences between the clinical algorithms are to be expected, given their differing implementation contexts, in line with the IMCI strategy of adapting generic global guidelines to local epidemiology and available resources.Yet, above and beyond contextual differences, 3 major categories of adaptation-scope, content, and structure-were identified as common to all the CDSSs when compared to paper-based IMCI guidance.The extent of these adaptations differed according to the individual CDSS.

Extended Scope
The WHO IMCI chart booklet proposes a simple syndromic approach. 5For children aged 2-59 months, this is based on the clinical assessment of a few basic danger signs: assessment of 4 main symptom groups; (1) cough/difficulty breathing, (2) diarrhea, (3) fever (predominantly for malaria and measles), (4) ear problems; and screening for malnutrition, anemia, and HIV.This narrow scope reflected a desire to strike a balance between ensuring low-skilled health care providers were equipped with the guidance to appropriately identify and manage the main causes of morbidity and mortality in primary care while not feeling overburdened with overly complex guidelines.
Relative to IMCI, the scope of most of the clinical algorithms compared was extended to cover a broader range of clinical conditions, such as dermatological or throat problems (Table 2).One tool extended to include over 50 additional diagnoses and a wider age range coverage.
The rationale for extending the scope was to support health care providers to implement evidencebased practice for a wider range of problems, thus enhancing quality of care for children presenting with non-IMCI conditions.Additionally, some organizations noted that broadening scope encouraged adoption by health care providers who found the tools more relevant to their practice.

Content
All tools also included additional or modified content, such as new diagnostic tests or changes to diagnostic criteria, to enhance sensitivity and/or specificity of the algorithm or to align with new evidence (Table 2).
This was particularly notable for children presenting with fever.The IMCI chart booklet focuses fever assessment on malaria and measles.Beyond these 2 conditions, it only advises health care providers to "Look for any bacterial cause of fever BOX 1.The 5 Knowledge Layers of the World Health Organization SMART Guideline Framework   [and]. . .Give appropriate antibiotic treatment for an identified bacterial cause." 5 The footnotes list several symptoms and signs, but no clear diagnostic or management criteria are provided.Participants noted, from experience and literature, that these broad recommendations often led health care providers to overprescribe antibiotics.For this reason, most tools provided decision support for the assessment and management of common or serious bacterial causes of fever, with most also including additional diagnostic tests.
The common rationale for extending or modifying content was to enhance quality of care and support better use of resources, particularly antimicrobials.The degree of modifications varied across the CDSS according to the type of intervention, stakeholder priorities, and developer and stakeholder perceptions of the capacity to support health care providers to appropriately implement more complex algorithms.

Revised Structure
The IMCI chart booklet follows a linear process in which, for each presenting symptom or syndrome, the health worker is advised to "ask" certain questions, then "look, (listen), feel" for signs (and perform measurements/tests where relevant) to classify and identify treatment before moving on to the next presenting symptom/issue (Figure 1A).This modular syndrome-based assessment approach is not aligned with the usual flow of a clinical consultation, where similar tasks are grouped together (Figure 1B).Although the IMCI chart booklet presents syndromes in a set order, health care providers can flip back and forth between charts.
CDSSs tend to enforce a predefined navigation through the clinical algorithm, enforcing mandatory responses to clinical prompts to ensure a systematic and complete (and thus safe) consultation process. 23,25,29,33This is defined by the clinical algorithm and the constraints of the digital solution.Therefore, the pathway structure varies between different CDSSs, with the overall workflow following a structure similar to IMCI guidance (Figure 1A), a classical primary care clinical encounter (Figure 1B), or a specifically tailored workflow (Figure 1C).Regardless of the exact workflow, all organizations emphasized the importance of ensuring that the CDSS should neither disrupt nor delay the consultation.Relative to IMCI guidance, the CDSSs implemented 2 types of modifications aimed at enhancing the consultation structure (Box 2).

COMPARISON OF PROCESSES FOR CDSS DEVELOPMENT
The process of development and refinement of clinical algorithms can take several years, from the identification of needs and objectives of a CDSS through development to implementation and evaluation.Though various steps in the process can be outlined in a linear fashion, there are many feedback loops for iterations of the algorithm over time (Figure 2).We focus here solely on the clinical algorithm development process.Although the details of the process differed between each CDSS, several common steps were identified across organizations.

Sources of Additional Content
All organizations needed to draw on several different sources in the process of algorithm development (Table 3).Alongside national (and/or generic WHO) IMCI guidelines, primary care health care providers are often expected to adhere to various other national or international clinical guidelines, including disease-specific guidelines (such as those for malaria, tuberculosis, or HIV), national formularies, or standard treatment guidelines.Most of the CDSSs integrated several different child health guidelines within their clinical algorithms.
Due to lengthy update cycles, some guidelines may not reflect the latest evidence (and some guidelines may conflict with each other).Furthermore, many narrative clinical guidelines do not explicitly provide the decision logic necessary to develop clinical algorithms.Therefore, clinical algorithm developers also consulted peer-reviewed literature, other guidelines, and national or international expert opinion to ensure that algorithms reflected current evidence and national policymaker perspectives.
The extent to which different sources were drawn varied according to the CDSS purpose and context (Table 3), but all organizations agreed that early and ongoing engagement with stakeholders is needed to determine the relevant sources for the algorithm.
Finally, the algorithms themselves may become a source (i.e., once they have been developed for 1 country, the algorithm may provide a basis or framework for an algorithm in another country [with adaptation to national guidelines]).

Human-Readable Format of the Clinical Algorithm
To enable experts' review and validation of the clinical algorithm and ensure transparency of the Reordered certain assessments to bring those most likely to be associated with severe illness first Created predefined shortcuts (skip logic) when a severe illness is identified to avoid unnecessary tasks and expedite prereferral treatment and referral 2. Modifications to integrate components of the assessment, diagnosis, and management to improve efficiency and user experience: Reordered certain assessments to bring those to the beginning that may influence other diagnoses (e.g., anemia, malnutrition) or to the end that rely on synthesizing information from other components of the consultation (e.g., fever without identified source) Grouped similar tasks according to stages of the consultation (i.e., grouping together medical history items, examination items, diagnostic tests, diagnoses, and management rather than performing each according to presenting syndrome before moving to the next) Integrated different diagnoses and treatments to ensure relevance of proposed final classifications and management recommendations May involve updates based on national/international guidelines (e.g., national pneumonia guidelines may have been updated more recently than IMCI).

All
Other national or international guidelines (e.g., standard treatment guidelines, formularies, disease-specific guidelines for malaria, TB, HIV, etc.) To extend scope (relative to IMCI) by incorporating information from various guidelines relevant to the diagnosis and management of children in primary care (that health workers are expected to implement at a country level) or to reflect most up-to-date relevant guidance not yet incorporated into national guidelines due to long guideline update cycles (e.g., WHO young infant guidelines).

All
Existing or newly conducted evidence syntheses (systematic reviews/metaanalyses) To implement relevant guidance not yet incorporated into national or international guidelines (due to long guideline update cycles), particularly if extending scope.

ALMANACH, MSFeCARE-Ped, ePOCTþ
Epidemiological data (surveillance/ research) To target focus for additional scope/content (on conditions with significant morbidity or mortality).
To incorporate flexibility within the algorithm (e.g., diagnostic and treatment advice differing based on location of facilities in high/low prevalence areas for diseases such as malaria).

MSFeCARE-Ped, ePOCTþ,IeDA
Existing clinical algorithm To use work already conducted on translating narrative guidance into decision logic, including updates that may have been made over time based on implementation experience or novel diagnostic/treatment approaches that may have been evaluated elsewhere.

All Expert opinion (national or international panel)
To advise which source documents (guidelines or other evidence) should be used as a basis for the algorithm.
To advise/review/validate interpretation of narrative guidelines/evidence translation to human-readable algorithm.In the absence of universally agreed standards for clinical algorithm representation at the time of their development, each organization addressed this issue slightly differently.All organizations developed diagrammatic representations, with some demonstrating the entire algorithm and others grouped according to either syndromes, diagnoses, or stages of the consultation.Some organizations used business process model and notation standards; others developed decision tables.Despite differences in format, the common rationale for these representations was to demonstrate the algorithms' clinical workflow, content, and decision logic to support validation by experts and/or end users.

Modifications and Updates
All organizations recognized that modifications are required following digitalization of the clinical algorithm during testing, piloting, and implementation.Given the complex nature of the algorithms, interaction with the end product often uncovers issues not apparent in the human-readable written (or depicted) format.Issues may be identified during verification (ensuring that the digital end product represents the human-readable algorithm) before it is in the hands of end users.Others are only identified from health worker feedback, observation, or analysis of CDSS or research data during piloting or implementation.Lastly, updates to narrative guidelines or new evidence may necessitate a need to update the clinical algorithm.
Participants acknowledged the importance of maintaining an up-to-date human-readable algorithm to ensure transparency of substantive updates to the digital algorithm (i.e., clinical content rather than user interface), which can be validated by relevant experts.

DISCUSSION
This comparison reflects the collective experience of several organizations in clinical algorithm development for CDSSs targeting the diagnosis and management of childhood illness in primary care in resource-constrained settings.In identifying the commonalities and differences in scope, content, structure, and development processes, we aim to highlight important considerations in the development of clinical algorithms for this population and contribute to the global dialogue on improving transparency, trust, and quality of health worker decision support.
Rigorously and transparently developing CDSS clinical algorithms is a complex and lengthy process with many similarities to guideline adaptation. 43ontext was acknowledged as an important critical driver of the content, structure, and development process of clinical algorithms for the CDSSs included in our analysis.Epidemiology, resources, clinical workflow, and the programmatic context (from controlled research settings to long-term health systems strengthening interventions) all influenced the degree to which the clinical algorithm deviated from IMCI guidance.Yet across all contexts, efficiency and fit to the consultation were recognized as critical in ensuring clinical safety and promoting uptake.
All algorithms drew on sources beyond IMCI guidance, including other national and international guidelines, published evidence, and expert opinion.Context was critical in determining the extent of additional sources used-from the availability of up-to-date guidelines to stakeholder priorities-and the extent of the expectation of health care providers to integrate many (sometimes conflicting) guidelines.Extending scope and content provides opportunities to improve quality of care, known to be worse for non-IMCI problems, 44 and uptake by health care providers, who report challenges when not supported by a diagnosis, 39 though further understanding on the usability and impact of more complex content is needed.Incorporating wider evidence was found to be limited by the dearth of literature on prognostic and diagnostic predictors in pediatric primary care in low-and middle-income countries. 45hese issues highlight the critical importance of addressing evidence gaps and of timely updates of international guidelines and guidance on adaptation for resource-constrained settings, 43 whether guidance is in paper or digital form.
Though all organizations acknowledged the importance of clearly documented human-readable algorithms for validation and transparency, different representation approaches were taken.Various methods have been proposed to represent CDSS clinical algorithms, but it is recognized that no single representation can adequately capture the complexity. 46WHO's SMART Guideline digital adaptation kits-which "distill WHO guidelines and operational resources into a standardized format that can be more easily incorporated into digital tracking and decision support systems"-along with the forthcoming handbook for digitizing primary health care, are an important step in enhancing the validity, transparency, and accessibility of CDSSs. 19,21,22ntext was acknowledged as an important critical driver of the content, structure, and development process of clinical algorithms for the CDSSs included in our analysis.
Their description of the process undertaken for the development of a digital antenatal care module reflects many issues applicable to CDSSs for the diagnosis and management of childhood illness. 47,48uthors agreed that when digital adaptation kits become available for child health in primary care, stakeholders should collaborate to support adaptation, implementation, and evaluation while continuing to foster innovation to support future improvements in CDSS quality and impact.

Limitations
This article has several limitations.First, it reflects the work of a predominantly Swiss-based working group, and although the authors have collaborated with ministries of health, nongovernmental organizations, academic institutions, civil society organizations, health care providers, and caregivers, this article does not directly reflect their views.We have since worked to address this by working with the Geneva Digital Health Hub on the formation of a broader CDSS Community of Practice.Although in its infancy, this group already includes a wider membership from many countries with whom we are collaborating to develop a common working approach and objectives.Further, this work does not represent an exhaustive list of IMCI-related CDSSs; however, from our network and literature search, we are only aware of 3 other IMCI-related CDSSs in South Africa, Bangladesh, and Tanzania (1 of which is no longer in use), and no others have been highlighted in a recent systematic review. 12astly, we have not addressed other important considerations in CDSS development and implementation.These include, among others: CDSS evaluation-from performance to usability and acceptability to clinical and cost-effectiveness in controlled settings and at scale 49 ; the algorithm adaptation requirements for different levels of care or differing epidemiology within a country; and wider implementation considerations such as training and mentorship, operational support, IT systems interoperability, and regulation.These issues all influence the content, structure, and development process of clinical algorithms-and their uptake and impact-but were beyond the scope of this article.

CONCLUSION
The results of this comparison reflect the first step by this group of practitioners, scientists, and policy actors in embracing the Principles for Digital Development 50 to collectively share learning and expertise on CDSSs for IMCI in primary care.Further evaluation of the relative effectiveness and cost-effectiveness of different approaches is needed to guide evidence-based practice in this complex field.Building and adhering to standards for CDSS development and implementation through multistakeholder dialogue is critical to ensure digital tools can effectively and equitably contribute to improve health and quality of care for children in resource-constrained settings.

FIGURE 1 .
FIGURE 1. Consultation Workflows: A. Representation of the WHO IMCI Chart Booklet workflow a,b ; B. Generic clinical consultation workflow c ; C. Hybrid workflow combining elements of the symptom-driven workflow (A) With natural consultation process (B) d

All
Abbreviations: ALMANACH, Algorithm for the Management of Childhood Illnesses; ePOCTþ, Electronic Point-of-Care Test Plus; IeDA, Integrated eDiagnosis Approach; IMCI, Integrated Management of Childhood Illness; WHO, World Health Organization.

FIGURE 2 .
FIGURE 2. High-Level Overview of Processes Involved in the Development and Implementation of Clinical Decision Support Systems, With Feedback Loops Between Stages

TABLE 1 .
19,21 1. Narrative: guideline and data recommendations 2. Operational: digital adaptation kits, comprised of semistructured documentation of operational and functional requirements 3. Machine readable: structured, software-neutral specifications, code, terminology, and interoperability standards 4. Executable: reference software, able to execute static algorithms and interoperable digital components and deliver operational and functional requirements 5. Dynamic: executable dynamic algorithms that are trained and optimized with advanced analytics to achieve prioritized outcomes Clinical Decision Support Systems for Pediatric Primary Care in Resource-Constrained Settings Selected for Comparative Assessment improvement; RCT, randomized controlled trial; SDC, Swiss Agency for Development and Cooperation; SPHCDA, State Primary Health Care Development Agency; SwissTPH, Swiss Tropical and Public Health Institute; TIMCI, Tools for Integrated Management of Childhood Illness; UCAD, Université Cheikh Anta Diop de Dakar; UON, University of Nairobi; UNIGE, University of Geneva; WHO, World Health Organization.

TABLE 2 .
Comparison of Scope of Clinical Decision Support Systems to IMCI Guidelines Algorithm for the Management of Childhood Illnesses; ePOCTþ, Electronic Point-of-Care Test Plus; IeDA, Integrated eDiagnosis Approach; IMCI, Integrated Management of Childhood Illness.

TABLE 3 .
Sources or Reference Documents Used in the Adaptation and/or Development of New Content and Content-Validation Committee Source(s)Rationale CDSS IMCI guidelines WHO generic and/or national IMCI guidelines to reflect country specific adaptations.