Skip to main content

Main menu

  • Content
    • Current Issue
    • Advance Access
    • Archive
    • Supplements
    • Special Collections
    • Topic Collections
  • For Authors
    • Instructions for Authors
    • Tips for Writing About Programs in GHSP
      • Local Voices Webinar
      • Connecting Creators and Users of Knowledge
    • Submit Manuscript
    • Publish a Supplement
    • Promote Your Article
    • Resources for Writing Journal Articles
  • About
    • About GHSP
    • Editorial Team
    • Advisory Board
    • FAQs
    • Instructions for Reviewers

User menu

  • My Alerts

Search

  • Advanced search
Global Health: Science and Practice
  • My Alerts

Global Health: Science and Practice

Dedicated to what works in global health programs

Advanced Search

  • Content
    • Current Issue
    • Advance Access
    • Archive
    • Supplements
    • Special Collections
    • Topic Collections
  • For Authors
    • Instructions for Authors
    • Tips for Writing About Programs in GHSP
    • Submit Manuscript
    • Publish a Supplement
    • Promote Your Article
    • Resources for Writing Journal Articles
  • About
    • About GHSP
    • Editorial Team
    • Advisory Board
    • FAQs
    • Instructions for Reviewers
  • Alerts
  • Find GHSP on LinkedIn
  • Visit GHSP on Facebook
  • RSS
PROGRAM CASE STUDY
Open Access

Maturity Assessment of the Health Information System Using Stages of Continuous Improvement Methodology: Results From Serbia

Steve Ollis, Milan Kovačević, Bosiljka Djikanovic, Nikola Radoman, Isidora Smigic and Mamadou Alimou Barry
Global Health: Science and Practice October 2024, 12(5):e2400083; https://doi.org/10.9745/GHSP-D-24-00083
Steve Ollis
aJohn Snow, Inc., Country Health Information Systems and Data Use, Arlington, VA, USA.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Milan Kovačević
aJohn Snow, Inc., Country Health Information Systems and Data Use, Arlington, VA, USA.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Bosiljka Djikanovic
bFaculty of Medicine and Centre-School of Public Health, University of Belgrade, Serbia.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Nikola Radoman
cMinistry of Health Republic of Serbia.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Isidora Smigic
dNational Alliance for Local Economic Development, Belgrade, Serbia.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Mamadou Alimou Barry
aJohn Snow, Inc., Country Health Information Systems and Data Use, Arlington, VA, USA.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
PreviousNext
  • Article
  • Figures & Tables
  • Supplements
  • Info & Metrics
  • Comments
  • PDF
Loading

Key Findings

  • The Health Information System (HIS) Stages of Continuous Improvement (SOCI) toolkit was used to assess the maturity of the Serbian HIS across 5 domains using a 5-point Likert scale.

  • The process of maturity assessment using the HIS SOCI toolkit in Serbia in 2021 served to establish a baseline status, strengthen dialogue among different key stakeholders in the field of digitalization of health care, and identify areas that were not covered by the initially drafted program for digitalization.

Key Implications

  • Policymakers should consider using available methods and tools for qualitative assessment of the maturity level of HIS before developing an eHealth strategy.

  • The use of maturity assessment tools brings additional perspectives and considerations that could supplement and fortify the process of strategy development.

  • Program managers can track the success of HIS development in the future using established and quantified baseline results of HIS maturity assessment.

ABSTRACT

Introduction:

Since the health information system (HIS) in public health care services in Serbia was introduced in 2009, it has gradually expanded. However, it is unclear how well the HIS components have developed and the whole system’s stage of maturity.

Method:

In June–September 2021, a maturity assessment of the Serbian HIS was conducted for the first time using the HIS Stages of Continuous Improvement (SOCI) toolkit. The toolkit measures HIS status across 5 HIS domains: leadership and governance, management and workforce, information and communication technology (ICT), standards and interoperability, and data quality and use. The domains were further divided into 13 components and 39 subcomponents whose maturity stage was assessed on a 5-point Likert scale, indicating the level of development: (1) emerging/ad hoc; (2) repeatable; (3) defined; (4) managed; and (5) optimized. The toolkit was applied in a working group of 32 professionals and experts who were engaged in developing the new national eHealth strategy and action plan.

Results:

The overall maturity score of the Serbian HIS was 1.6, which indicates a low level. The highest baseline score (2) was given to the standards and interoperability domain, and the lowest (1.1) was given to ICT infrastructure. The remaining 3 domains (leadership and governance, Management and Workforce, and Data Quality and Use) were similarly rated (1.7, 1.7, and 1.6, respectively).

Conclusion:

A baseline assessment of the maturity level of Serbian HIS indicates that the majority of components are between the emerging/ad hoc stage and repeatable, which represent isolated, ad hoc efforts, with some basic processes in place and existing and accessible policies. This exercise provided an opportunity to address identified weaknesses in the upcoming national eHealth strategy.

INTRODUCTION

The Republic of Serbia is a middle-income country in Southeastern Europe, commonly known as Western Balkan, with an assessed population size of approximately 6.6 million (without data for Kosovo and Metohija).1 The health care system in Serbia is organized and provided through a well-developed network of public health care services and institutions that function at the primary, secondary, and tertiary health care levels.2 Altogether, there are around 350 different entities (health care institutions) at the national level that are state owned and funded by the Republic Health Insurance Fund. Health information systems (HISs) have been largely implemented and used since 2009, according to the National Program of Work, Development, and Organization of an Integrated Health Information System – E-health, along with an action plan in 2009–2015, which provided a blueprint for the system development in later years.3

Since 2015, the HIS in Serbia has been incrementally developed and upgraded to enable some additional functionalities (e.g., electronic services) (Figure). For example, in 2016, general practitioners could make appointments at the secondary or tertiary level through the HIS. In 2019, eRecept (ePrescription) was introduced to allow paperless communication between primary care physicians and pharmacies to issue prescribed medicines to patients.4 However, despite these additional functionalities, key and strategic issues related to the HIS functioning, such as its governance and further development, remained unaddressed.

FIGURE
  • Download figure
  • Open in new tab
  • Download powerpoint
FIGURE

Timeline of Key Achievements in Developing the Serbia HIS

Abbreviations: HIS, health information system; SOCI, Stages of Continuous Improvement.

The need to achieve a higher level of interoperability of the HIS in Serbia was recognized by the health system’s analysts, professionals, and officials in recent analytical reports in 2021 and the Prime Minister’s public address.5,6 Given the exponential growth of possibilities that information technologies (IT) and communications worldwide can provide, it was clear that the health care system in Serbia needed a strategic approach to further develop its HIS. The strategic approach required developing a national strategy and action plan that would provide a clear pathway for implementing new and innovative IT solutions in the health sector.

The need to achieve a higher level of interoperability of the HIS in Serbia was recognized.

To develop the national eHealth strategy and action plan, in 2021, the Prime Minister’s Office established the eHealth Steering Committee, a national task force co-chaired by both the Ministry of Health and the Prime Minister’s Office. The eHealth Steering Committee included 32 members from the ministries of finance, defense, and health, as well as representatives from various health institutions, health professional associations and chambers, and the private health care sector. The working group used the World Health Organization (WHO)/International Telecommunication Union National eHealth Strategy toolkit as a framework for developing the strategy and action plan.7

The development of the national eHealth strategy and action plan was largely supported by the Country Health Information Systems and Data Use (CHISU) program and other partners, including the United Nations Development Program.8 CHISU is a 5-year (2020–2025) U.S. Agency for International Development-funded initiative focused on strengthening HISs in more than 20 countries.8 CHISU operations in Serbia began in April 2021, with a mandate of strengthening the governance of the HIS and the use of health data for decision-making. During the process of developing the national eHealth strategy and action plan, CHISU was invited to participate and support the work of the eHealth Steering Committee.

Assessment of the Health Information System Maturity Level in Serbia

To strengthen the foundation for developing the eHealth strategy and action plan and review the current maturity status of the Serbian HIS, an assessment toolkit called the HIS Stages of Continuous Improvement (SOCI) toolkit was used, as proposed by CHISU. The HIS SOCI toolkit had already been successfully applied in several low- and middle-income countries that intended to improve their HISs with donor support.9–12 The tool’s measurement scale evaluates processes, people, and functional and operational capabilities that facilitate collecting, managing, and using health information to support the health system.13 The HIS SOCI toolkit was developed by a team of HIS experts from MEASURE Evaluation, the U.S. Centers for Disease Control and Prevention, and the Health Data Collaborative digital health and interoperability technical working group.13 The SOCI toolkit provides a systematic process for assessing HIS maturity across 5 core domains: leadership and governance, management and workforce, information and communication technology (ICT), standards and interoperability, and data quality and use. These domains are divided into 13 components and 39 subcomponents (Table 1). Each subcomponent was assessed using the following 5-point Likert scale: (1) emerging/ad hoc; (2) repeatable; (3) defined; (4) managed; and (5) optimized, with 1 being the lowest and 5 the highest level of maturity. Descriptions of scores for each subcomponent are provided in Supplement 1,13 which was translated into the Serbian language to facilitate ease of use by all stakeholders.

View this table:
  • View inline
  • View popup
TABLE 1.

SOCI Baseline Assessment Scoresa for HIS Core Domains, Components, and Subcomponents, Serbia, 2021

Conducting an HIS maturity assessment can lead to a thorough understanding of the HIS status, and the results can inform improvements in the HIS to a desired status by supporting the development of a clear roadmap to improve processes.

This article describes the results of the initial HIS SOCI assessment in Serbia as a baseline for comparison for future developments and an indication and reminder of areas that should be improved.

METHODS

A SOCI assessment was conducted using a collaborative process in the following 5 consecutive steps.13

Step 1. Establish the Assessment Core Team

The assessment core team was formed by several entities already engaged as part of the eHealth strategy and action plan Steering Committee. These included representatives from the Prime Minister’s Office, the Ministry of Health, the public health sector, the Institute of Public Health of Serbia Dr. Milan Jovanovic Batut (IPH Batut), and the National Alliance for Local Economic Development. These stakeholders from these entities were chosen based on their broad knowledge of the systems, processes, and policies related to the HIS. A complete list of the institutions from which assessment core team members came is given in Step 4.

Step 2. Define the Scope and Assessment Approach

Before defining the scope and the assessment approach, multiple meetings took place to (1) present the concept of the assessment to the key high-level staff or leaders, (2) gain a better understanding of the SOCI toolkit and what needed to be done, and (3) select the best strategy to administer and conduct the SOCI assessment and receive key informant input.

To better understand the SOCI toolkit, the team discussed and addressed issues related to the SOCI terminology and assigning scores, the meaning of each score level, types of key documents to be reviewed to support the scores (evidence), and length of action planning and completing the SOCI assessment.

Given that the development of the eHealth strategy was already underway, the team decided that the best approach would be for the core assessment team to propose initial points per subcomponent, including citing available documents to justify the given points, where applicable, and present this to the broader working group for their review and discussion.

Step 3. Conduct a Landscape Analysis and Desk Review

A thorough desk review and landscape analysis of relevant legislative and strategic documents and reports were performed to provide evidence on the situation in 2021 and justify proposed points per each subcomponent. Examples of reviewed documents included legislation (various laws and sublaws on health evidence and documentation, medical devices, patient rights, eBusiness, protection of personal data, public health, and protecting the population from infectious diseases); strategies that were relevant for HISs; official reports issued by public health institutions; and public procurement plans for Agencija za lekove i medicinska sredstva Srbije (ALIMS, Medicines and Medical Devices Agency of Serbia), Office for IT and eGovernment, and Health Insurance Fund.

At the eHealth Steering Committee’s request, CHISU provided experienced health experts who developed a summary of the current state of analysis, including all reviewed documents. The results were shared with stakeholders in a document summarizing the current state of eHealth legislation and policy in Serbia.

Step 4. Selecting Key Stakeholders

The SOCI assessment was conducted as part of the eHealth strategy development process. As such, the stakeholders for SOCI were composed mostly of the eHealth steering committee members. The steering committee included 32 members who were representatives of the Prime Minister’s Office, the Ministry of Health, the Ministry of Defense, the Ministry of Finance, the Office for IT and eGovernment, the Republic Secretariat for Public Policies, IPH Batut, ALIMS, Medical Chamber of Serbia, Chamber of Commerce of Serbia, Pharmaceutical Chamber of Serbia, Chamber of Nurses and Health Technicians of Serbia, Association of Informatics in Health of Serbia, Association of Private Health Institutions and Private Practices of Serbia, Association of Patients of Serbia, the Council of Foreign Investors, the Nordic Business Alliance, the Digital Serbia Initiative, and the National Alliance for Local Economic Development.

The SOCI assessment was conducted as part of the eHealth strategy development process.

Step 5. Organizing Assessment Workshops

The work on the development of the eHealth strategy and action plan was organized in 3 2-day workshops that were held in 2021. The sessions on the application of HIS SOCI toolkit were included in all 3 workshops.

At the first workshop, held on June 16 and 17, an overview of the SOCI toolkit was presented to the members of the eHealth Steering Committee, and a common understanding of the tool and its purpose was established. Following this workshop, the tool was translated into Serbian to improve its usability by all stakeholders.

At the second workshop, on August 11 and 12, the eHealth steering committee members broke into small groups and discussed the current state of the HIS across the domains and provided appropriate points based on scoring guidance. After this workshop, there was a follow-up with stakeholders to amend the desk review (Step 3) and obtain source documents to justify assigned points.

Between the second and third workshops, the HIS SOCI was emailed to the eHealth Steering Committee members so they could review the toolkit, propose points, and send it back with proposed points and comments to the assessment core team.

During the third workshop, on September 8 and 9, results from the assessment were presented and discussed with a wider group. During the discussion, consensus was achieved based on the evidence collected during the landscape analysis and desk review (Step 3) and arguments presented by stakeholders.

The outcome of the consensus method was a single point assigned to each of the 39 subcomponents on the following 5-point Likert scale: (1) emerging/ad hoc; (2) repeatable; (3) defined; (4) managed; and (5) optimized.

Calculation of Scores

Scores for each of the 5 domains and 13 components within the domains were calculated as an average sum of points per component and per domain, presented to 1 decimal place.

RESULTS

We identified 17 legislative and strategic documents, institutions, and digital platforms that were relevant for the HIS assessment according to the SOCI method. These were used to justify given points.3,4,14–28 Scores from an overall assessment of the maturity level for each of the 5 HIS core domains, components, and consensus-achieved points for the subcomponents are listed in Table 1.

Leadership and Governance

The average score of 6 subcomponents in the leadership and governance domain was 1.7, which is closer to repeatable (2) than ad hoc/emerging status (1). Although Serbia has been making excellent progress in this area by announcing the development of the new eHealth strategy, the situation at the time of the assessment was rather modest because the last strategic document in this field was announced in 2009, with an action plan for the period 2009–2015.3 Some compliance documents and policies, such as the Law on Medical Documentation and Records in the Field of Health care, have not been updated to reflect the availability of digital systems,18 and an overall governing body for HIS development and supervision has not been defined yet. Key points in the areas of strength and major gaps and loopholes are presented in Table 2 for all domains.

View this table:
  • View inline
  • View popup
TABLE 2.

Summary of Main Findings in HIS SOCI Analysis in Serbia, 2021

Management and Workforce

The average score for the HIS management and workforce domain was 1.7. The HIS training and education (including continuous professional development) and human resources policy subcomponents received the lowest points because continuous professional development of IT staff and building and strengthening the knowledge and skills of IT solutions end users had been neither defined nor implemented and regulated. This indicated a clear need and window of opportunity for a systematic approach necessary for this improvement. In contrast, in Serbia’s health sector, there are well-defined and documented competencies, roles, and responsibilities for the workforce. Regarding the availability of the IT workforce with adequate HIS competencies, it was concluded and agreed that a knowledgeable and skillful IT workforce in Serbia exists, and therefore, the HIS competencies subcomponent (knowledge, skills, and abilities) received consensus-based 2 points. However, keeping them in the public health system is challenging because the globally competitive IT labor market offers better working conditions and higher salaries. Some national human resources policies regulate IT personnel positions and deployment, although they seem to underestimate the real needs for this type of staff.15

The subcomponents of HIS financing plan and resource mobilization were also assessed with 2 points because, in general, the use of public funds in Serbia is well defined, according to the Law on Public Procurement16 and the annual Public Procurement Plan of the Ministry of Health.17 These documents establish legal and administrative procedures for competition for public funds. However, regulations related to mobilizing resources and making investments in HIS that come from sustainable and ongoing resources have not yet been fully defined; therefore, the assigned point could not be higher in 2021.

Information and Communication Technology Infrastructure

The HIS ICT infrastructure domain deals with the implementation of the required technology by applying standard operating procedures to enhance the daily business of the health sector and its stakeholders. The average score of the 5 subcomponents in this domain was 1.2. Only the hardware subcomponent through the consensus method was assessed with 2 points. The remaining subcomponents, reliable power/electricity, ICT business infrastructure support, network and Internet connectivity, and business continuity processes and policies, received 1 point (ad hoc/emergency status).

The average score of the 5 subcomponents in the ICT infrastructure domain was 1.2.

Despite these low points, some areas of strength exist, which are prerequisites for improvements (Table 2). The Office of IT and the independent eGovernment entity were established in 2017, and they provide national ICT infrastructure.25 The Office serves a variety of functions for the government and supports data hosting, systems development, and maintenance. In the future, it might have an expanded role in the development and maintenance of an HIS and support. Many eGovernment applications (e.g., the COVID-19 case tracking system) are hosted at the National Data Center. This key resource appears to be well staffed, with a high level of standardized processes in place. However, there was a lack of a unified vision and coordination between different ministries, which has resulted in a proliferation of systems and incomplete sharing of data and information between organizations/institutions.

Although backups of the system hosted at the National Data Center were being regularly conducted, when it came to the fragmented HIS, backups remained up to each health institution and their HIS providers (vendors) and were not done consistently. There is no central process for testing and monitoring the availability of the local systems, including network and system availability. This is especially important because the availability of patient data in the centralized system depends on the source system being online and available at the time of the data request.

Standards and Interoperability

The HIS standards and interoperability domain and its 12 subcomponents are related to health data exchange between different entities and levels using nationally and internationally known and accepted standards. Compared to other domains, this was assessed the highest with an average score of 2.0, which meant that some procedures existed and were in a “repeatable” stage (Table 1).

Standards and interoperability of HIS in Serbia are assured by already existing standards and guidelines, primarily those related to infrastructure, public health, health insurance, and connecting subsystems in institutions, such as Regulation on the Detailed Content of Technological and Functional Requirements for the Establishment of an Integrated Health Information System; Law on Medical Documentation and Records in the Field of Health Care; and Professional Methodological Guidelines for Maintaining Basic Documentation on Resources in Health Care Institutions, Private Practice, and Other Legal Entities Engaged in Health care Activities.14,18,19 These documents’ existence and practical application justify the given points. However, there was also a recognized need to achieve a higher data standardization and interoperability level.

In the Serbian health care system, terminology management is assured by the use of the International Classification of Diseases and Related Health Problems version 10, which is integrated into the HIS.21 It enables Serbia to report selected health indicators to the official authorities in the European Union and the World Health Organization.24 At the national level, there is an e-service called Servis Javnog Zdravlja (public health service), which is hosted at the IPH Batut and allows aggregate data exchange between health care services and institutes of public health (i.e., the network of regional institutes, and IPH Batut).20,24,26 However, there are still many health-related datasets that ought to be collected as required by law18 and that are not reported through this service. There is much work ahead to be done to increase the interoperability of the systems and enable additional functionalities.

A good example of data exchange between primary health care physicians and pharmacies is eRecept.3 In addition, personal data exchange is provided by the Ministry of Health-hosted vertical health care management platform called eZdravlje (eHealth), which allows patients to monitor their health records, see the results of their health care visits, manage appointments, and perform other functions.27 Although this application and its outreach could be improved, it presents a good starting point for further development of the communication between citizens and health care services through digital platforms.

Unique person identity management is regulated by the existence and application of the Law on the Unified Register of Citizens and the Law on the Protection of Personal Data.22,23 However, this subcomponent could also be improved to ensure more data protection in the digital health care system.

Data Quality and Use

The data quality and use domain with its 11 subcomponents indicates how well key stakeholders in the health system are using available data for decision-making and other purposes. The average score for data quality and use domain was 1.4. Four subcomponents, data quality assurance and quality control, data management, data synthesis and communication, and reporting and analytics features, were assessed with 2 points. The remaining 7 subcomponents were assessed with 1 point.

Procedures for data use and their quality are defined by the Law on Medical Documentation and Records in the Field of Health Care and corresponding Professional-Methodological Guidelines for Maintaining Basic Documentation on Resources in Health Care Institutions, Private Practice, and Other Legal Entities Engaged in Health Care activities, which are prepared by IPH Batut.18,19 They are implemented at all levels of health care within the national network of public health care institutions.20 Regional institutes for public health (24 of them) are in charge of data collection and control and send data to the IPH Batut for analyses at the national level.20 However, a functional national body or workgroup for quality management and data usage is needed with an established and standardized process that engages health data stakeholders. Moreover, Serbia has demonstrated the need for additional development of data quality plans that require periodic revisions using defined standards and procedures and methodological professional instructions at all levels. Strengthening the capacity of institutes of public health is crucial to continue to improve data collection and quality work.

DISCUSSION

We present the results of the maturity assessment of the HIS in the Republic of Serbia in 2021. The assessment took a holistic approach and was done in a comprehensive and structured way that used a consultative process with members of a national working group that was in charge of developing the new eHealth strategy and applied the SOCI toolkit to assign consensus-based points to every subcomponent of the HIS across 5 domains.13,28 Several toolkits, such as the Digital Health Assessment Toolkit, Global Digital Health Index, and Maturity Model, have been used to assess the maturity of different HIS aspects, and they have been gradually evolving.29 In general, the results of these assessments serve to inform about the current state of HIS at the national level, clarify areas for further improvements, and serve as a baseline for comparisons with future progress.

The results of these assessments serve to inform about the current state of HIS at the national level, clarify areas for further improvements, and serve as a baseline for comparisons with future progress.

The implications of the HIS assessment in Serbia were beneficial for several reasons. It established a baseline for the maturity status in 2021, which can be used to monitor national progress in achieving desired goals in the future. This process also resulted in recommendations related to adding data quality and use as a program objective (a domain within the SOCI assessment that was missing in a draft plan), which was addressed in the national eHealth strategy.30 The assessment indicated that improvements were necessary for every component. The HIS SOCI toolkit was also used to set projected desirable points (goals) in each HIS subcomponent, which sum up the score of 2.9. Still, due to the limitations in the article length, we presented only the baseline assessment results. Projected goals are listed in Supplement 2.

HIS maturity assessments have been a widely recognized activity, which is especially beneficial for HIS analysis in low- and middle-income countries, such as Ethiopia, Kenya, Ghana, and Uganda.9–12 Improved use of digital technologies for health has been seen as a vehicle that can contribute to achieving universal health coverage, as stated by WHO in 2018.31 In addition, WHO’s Global Strategy on Digital Health 2020–2025 emphasized that an integral part of health priorities should be digital health, and populations should benefit from it “in a way that is ethical, safe, secure, reliable, equitable and sustainable.”32

The comprehensive HIS maturity assessment results clearly indicated where the sector was in terms of 5 key domains. Overall, it showed that the average national HIS maturity level across all 5 domains was 1.6, between emerging/ad hoc and repeatable. This score means there was an awareness of the need to develop different components of HIS, but ad hoc solutions and fragmented approaches dominated. Project-based activities were not sufficiently coordinated among all stakeholders, and there was insufficient institutional framework and support. The assessment also revealed that there was a clear need for standardization of the data collection and data exchange in a machine-readable form, in addition to the need to create an entity/body that would coordinate and oversee the implementation of the program for digitalization in the health system in Serbia and action plan.

The average score for the ICT infrastructure domain was lower (1.2) than the domain HIS standards and interoperability (2.0), which may be surprising considering that the infrastructure is a prerequisite for HIS standards and interoperability. However, we believe that the discrepancy in these scores reflects the level of key informants’ subjective satisfaction with the basic infrastructure, experiencing temporary failures of HIS or lack of IT supplies in a few health facilities. The SOCI tool did not propose quantifying infrastructure; its scores were based on the description, existence of the relevant documents, and mutual consensus, not on quantification, so it is difficult to assess it objectively and compare these results with assessments conducted in other countries worldwide, which we hesitate to do.9–12 The level of satisfaction with the current state of availability of HIS standards and interoperability could be ascribed to recently adopted new system’s functionalities and interoperability, such as a digital platform for data collection Servis Javno zdravlje26 or (eRecept).4

In May 2022, the eHealth strategy and action plan was adopted,33 and all important elements of developing an integrated HIS in Serbia are envisaged and broken down into concrete objectives, measures, and activities. The level of its implementation remains to be seen (i.e., the level of achieving desirable improvements in the upcoming period).

Limitations

We acknowledge that the SOCI tool applied in this research had limitations that were beyond our control. Specifically, the descriptions of the points assigned to subcomponents within the domains were broadly defined, which may have led to variations in individual interpretations depending on the individual perspective, experience, and perceptions of the key informants most involved in the digitalization of the health system. However, final assessments were formed by mutual (dis)agreements, discussions, and eventually achieving consensus, though we acknowledge that this process may have been subject to bias. In some cases, different perspectives among steering committee members led to varied interpretations of the level of maturity development, with those who articulated their views more effectively influencing the group’s consensus. As a result, it is challenging to determine whether all members fully embraced the perspective that reached a consensus or if they agreed to avoid further discussion. An alternative approach to the consensus method could have been the Delphi method, where participants independently assign scores over several rounds without interacting with each other.34 However, we felt that this method might not foster mutual collaboration, exchange of opinions, and capacity-building within the country. For these reasons, we chose to proceed with the consensus method.

The SOCI tool is, by default, meant for systematically evaluating the overall national HIS and its governance, not its fragments, and we acknowledge that end users might have different opinions and perspectives over the level of HIS maturity.

By applying the SOCI tool, we intended to provide a wide landscape of the domains pertinent to digitalization, but they were defined and limited by the proposed SOCI method. Indeed, we recognized that not all important fields were addressed and evaluated by this tool, such as the applicability of artificial intelligence (AI) in health care. Nevertheless, the use of AI was addressed in the eHealth strategy and action plan,33 which indicates a high level of awareness at the national level in Serbia regarding the opportunities of IT and HIS. In the Action Plan, AI was mentioned within activities, such as the regulation of the legal framework for AI application in the health care system; the use of AI in radiological diagnostics, real-time drug interaction checks during the creation of electronic prescriptions, embedding best practice guides and clinical pathways into software solutions, and a range of other highly sophisticated IT solutions for improving clinical practice.33

CONCLUSION

The 2021 SOCI assessment indicated that the level of HIS development in the Republic of Serbia was somewhat low. Still, it provided a good foundation for the activities planned in the health sector. The newly created and adopted eHealth strategy and action plan provide strategic directions and a blueprint for further development of an integrated HIS.33 This integrated HIS would allow a higher level of interoperability, quality, and use of health-related data in circumstances that are characterized by a high level of governmental commitment to defining an appropriate institutional and legislative framework that would be required for further HIS development.

Acknowledgments

The work on the assessment of the health information system (HIS) in Serbia using the Stages of Continuous Improvement methodology was commenced during the process of development of the Program and Action Plan for Digitalization of the Health System in the Republic of Serbia (the Program). The working group members (Steering Committee members) for the development of the Program contributed to the assessment of the HIS in Serbia, as well as the members of the National Alliance for Local Development, who acted as secretary of this working group.

Funding

The process of the maturity assessment of the Health Information System in Serbia was supported by the U.S. Agency for International Development and the Country Health Information Systems and Data Use project. This study was also supported by the Ministry of Science, Technological Development and Innovation Republic of Serbia no. 451-03-66/2024-03/200110.

Disclaimer

The funders had no role in study design, data collection and analysis, the decision to publish, or the preparation of the manuscript.

Author contributions

SO and MK: conceptualized the study report. MK: organized and led data curation, formal analysis, investigation, and project administration. MK and BD: writing-original draft. BD: review and editing of original draft writing. NR, IS and MB: validated study findings. IS: involved in project administration. All authors approved the final version of the manuscript.

Competing interests

None declared.

Notes

Peer Reviewed

First Published Online: October 17, 2024.

Cite this article as: Ollis S, Kovačević M, Djikanovic B, Radoman N, Smigic I, Barry MA. Maturity assessment of the health information system using stages of continuous improvement methodology: results from Serbia. Glob Health Sci Pract. 2024;12(5):e2400083. https://doi.org/10.9745/GHSP-D-24-00083

  • Received: February 1, 2024.
  • Accepted: September 24, 2024.
  • Published: October 29, 2024.
  • © Ollis et al.

This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly cited. To view a copy of the license, visit https://creativecommons.org/licenses/by/4.0/. When linking to this article, please use the following permanent link: https://doi.org/10.9745/GHSP-D-24-00083

REFERENCES

  1. 1.↵
    Estimated population, 2022. Statistical Office Republic of the Republic of Serbia. Accessed September 30, 2024. https://www.stat.gov.rs/en-US/vesti/statisticalrelease/?p=13622&a=18&s=1801?s=1801
  2. 2.↵
    1. Bjegovic-Mikanovic V,
    2. Vasic M,
    3. Vukovic D, et al
    . Serbia: Health System Review. Health Syst Transit. 2019;21(3):1–211. pmid:32851979
    OpenUrlPubMed
  3. 3.↵
    Uredba o Programu rada, razvoja i organizacije integrisanog zdravstvenog informacionog sistema – “e-Zdravlje” [Regulation on the Work Program, Development and Organization of the Integrated Health Information System – “eHealth”]. Official Gazette Republic of Serbia 55/2009. Accessed September 30, 2024. https://pravno-informacioni-sistem.rs/eli/rep/sgrs/vlada/uredba/2009/55/1
  4. 4.↵
    eRecept: A digital prescription service. Ministry of Health Republic of Serbia; 2019.
  5. 5.↵
    Optimization of the network of healthcare services in Serbia. Ministry of Health Republic of Serbia. Accessed September 30, 2024. https://optimizacijazdravstva.rs/lat/
  6. 6.↵
    Serbian Prime Minister’s speech. National Assembly Republic of Serbia. October 2022. Accessed September 30, 2024. https://rsjp.gov.rs/wp-content/uploads/ana-brnabic-ekspoze-1022_cyr.pdf
  7. 7.↵
    World Health Organization (WHO). National eHealth Strategy Toolkit. World Health Organization/International Telecommunication Union; 2012. Accessed September 30, 2024. www.who.int/publications/i/item/national-ehealth-strategy-toolkit
  8. 8.↵
    Serbia. U.S. Agency for International Development Country Health Information System and Data Use. Accessed September 30, 2024. https://chisuprogram.org/where-we-work/serbia
  9. 9.↵
    1. Biru A,
    2. Birhan D,
    3. Melkamu G,
    4. Gebeyehu A,
    5. Omer AM
    . Pathways to improve health information systems in Ethiopia: current maturity status and implications. Health Res Policy Syst. 2022;20(1):78. doi:10.1186/s12961-022-00860-z. pmid:35768819
    OpenUrlCrossRefPubMed
  10. 10.
    1. Nyangena J,
    2. Rajgopal R,
    3. Ombech EA, et al
    . Maturity assessment of Kenya’s health information system interoperability readiness. BMJ Health Care Inform. 2021;28(1):e100241. doi:10.1136/bmjhci-2020-100241. pmid:34210718
    OpenUrlAbstract/FREE Full Text
  11. 11.
    MEASURE Evaluation. Building a Strong and Interoperable Health Information System for Ghana. University of North Carolina at Chapel Hill; 2018. Accessed September 30, 2024. https://www.measureevaluation.org/resources/publications/fs-18-275/at_download/document
  12. 12.↵
    MEASURE Evaluation. Building a Strong and Interoperable Digital Health Information System for Uganda. University of North Carolina at Chapel Hill; 2019. Accessed September 30, 2024. https://www.measureevaluation.org/resources/publications/fs-18-296/at_download/document
  13. 13.↵
    MEASURE Evaluation. Health Information System Stages of Continuous Improvement Toolkit – User Guide. Accessed September 30, 2024. https://www.measureevaluation.org/resources/publications/ms-19-158.html
  14. 14.↵
    Pravilnik o bližoj sadržini tehnoloških i funkcionalnih zahteva za uspostavljanje integrisanog zdravstvenog informacionog sistema [Regulation on the Detailed Content of Technological and Functional Requirements for the Establishment of an Integrated Health Information System]. Official Gazette Republic of Serbia 95/2009. Accessed September 30, 2024. https://pravno-informacioni-sistem.rs/eli/rep/sgrs/ministarstva/pravilnik/2009/95/3/reg
  15. 15.↵
    Uredba o katalogu radnih mesta u javnim službama i drugim organizacijama u javnom sektoru [Regulation on the Catalog of Job Positions in Public Services and Other Organizations in the Public Sector]. Official Gazette Republic of Serbia 81/2017, 6/2018 and 43/2018. Accessed September 30, 2024. https://www.paragraf.rs/propisi/uredba-o-katalogu-radnih-mesta-u-javnim-sluzbama-i-drugim-organizacijama-u-javnom-sektoru.html
  16. 16.↵
    Zakon o javnim nabavkama [Law on Public Procurement]. Official Gazette Republic of Serbia 91/2019. Accessed September 30, 2024. https://www.paragraf.rs/propisi/zakon-o-javnim-nabavkama.html
  17. 17.↵
    Plan javnih nabavki za 2020.godinu. [Public Procurement Plan for 2020]. Ministry of Health of the Republic of Serbia.
  18. 18.↵
    Zakon o medicinskoj dokumentaciji i evidencijama u oblasti zdravstva. [Law on Medical Documentation and Records in the Field of Healthcare]. Official Gazette Republic of Serbia 123/2014, 106/2015, 105/2017, 25/2019 (Article 27, paragraph 7; Article 29, paragraph 33). Accessed September 30, 2024. https://pravno-informacioni-sistem.rs/eli/rep/sgrs/skupstina/zakon/2014/123/2/reg
  19. 19.↵
    Stručno-metodološko uputstvo za vođenje osnovne dokumentacije o resursima u zdravstvenim ustanovama, privatnoj praksi i drugim pravnim licima koja obavljaju poslove zdravstvene delatnosti [Professional-Methodological Guidelines for Maintaining Basic Documentation on Resources in Healthcare Institutions, Private Practice, and Other Legal Entities Engaged in Healthcare Activities]. Institute of Public Health of Serbia “Dr Milan Jovanović Batut”, 2019. Accessed September 30, 2024. https://www.batut.org.rs/download/uputstva/SMUKadrovi-Registar24122019a.pdf
  20. 20.↵
    Uredba o planu mreže zdravstvenih ustanova [Regulation on the Health Institutions Network Plan]. Official Gazette of the Republic of Serbia 5/2020, 11/2020, 52/2020, 88/2020, 62/2021, 69/2021, 74/2021 and 95/2021. Accessed September 30, 2024. https://www.paragraf.rs/propisi/uredba_o_planu_mreze_zdravstvenih_ustanova.html
  21. 21.↵
    MKB-10 (Medjunarodna klasifikacija bolesti i srodnih zdravstvenih problema). [ICD-10 (International Classification of Diseases and Related Health Problems)]. Institute of Public Health of Serbia, “Dr. Milan Jovanović Batut”, 2010. Accessed September 30, 2024. https://www.rfzo.rs/download/dsg/MKB102010Knjiga1%281%29.pdf
  22. 22.↵
    Zakon o jedinstvenom matičnom broju građana [Law on the Unified Register of Citizens]. Official Gazette of the Republic of Serbia 24/2018. Accessed September 30, 2024. https://www.paragraf.rs/propisi/zakon-o-jedinstvenom-maticnom-broju-gradjana.html
  23. 23.↵
    Zakon o zaštiti podataka o ličnosti [Law on the Protection of Personal Data]. Official Gazette of the Republic of Serbia 87/2018. Accessed September 30, 2024. https://www.paragraf.rs/propisi/zakon_o_zastiti_podataka_o_licnosti.html
  24. 24.↵
    Institute of Public Health of Serbia, Dr Milan Jovanović Batut. Accessed September 30, 2024. https://www.batut.org.rs
  25. 25.↵
    The Office for Information Technologies and eGovernment Republic of Serbia. Accessed September 30, 2024. https://www.ite.gov.rs/tekst/en/124/office-for-it-and-egovernment.php
  26. 26.↵
    Servis Javno zdravlje: A digital platform for authorized health-related data collection in machine-readable format. Institute of Public Health of Serbia “Dr Milan Jovanovic Batut”, 2020
  27. 27.↵
    Portal eZdravlje: a digital portal for citizens to track their personal health-related data. Ministry of Health Republic of Serbia. Accessed September 30, 2024. https://www.e-zdravlje.gov.rs
  28. 28.↵
    The Law on Free Access to Information of Public Importance. Official Gazette of the Republic of Serbia 120/2004; 54/2007; 104/2009; 36/2010 and 105/2021. Accessed September 30, 2024. https://www.paragraf.rs/propisi/zakon_o_slobodnom_pristupu_informacijama_od_javnog_znacaja.html
  29. 29.↵
    World Bank. Digital Health Maturity Assessment and Toolkits. Implementation Know-How Brief. World Bank. Accessed September 30, 2024. https://openknowledge.worldbank.org/server/api/core/bitstreams/95128705-06f6-43b8-9236-9e7f8abf301d/content
  30. 30.↵
    New eHealth strategy promotes data use in Serbia. Country Health Information System and Data Use; U.S. Agency for International Development. Accessed September 30, 2024. https://chisuprogram.org/news/blog/new-ehealth-strategy-promotes-data-use-serbia
  31. 31.↵
    Digital health. World Health Organization. Accessed September 30, 2024. https://www.who.int/health-topics/digital-health
  32. 32.↵
    World Health Organization (WHO). Global Strategy on Digital Health 2020–2025. WHO; 2021. Accessed September 30, 2024. https://apps.who.int/iris/handle/10665/344249
  33. 33.↵
    Program digitalizacije u zdravstvenom sistemu Republike Srbije za period od 2022. do 2026. godine [The Digitalization Program in the Health System of the Republic of Serbia for the Period from 2022 to 2026]. Accessed September 30, 2024. https://www.zdravlje.gov.rs/tekst/364590/program-digitalizacije-u-zdravstvenom-sistemu.php
  34. 34.↵
    1. Jason LA,
    2. Glenwick DS
    . Handbook of Methodological Approaches to Community-Based Research: Qualitative, Quantitative, and Mixed Methods. Oxford Academic; 2015.
PreviousNext
Back to top

In this issue

Global Health: Science and Practice: 12 (5)
Global Health: Science and Practice
Vol. 12, No. 5
October 29, 2024
  • Table of Contents
  • About the Cover
  • Index by Author
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word about Global Health: Science and Practice.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Maturity Assessment of the Health Information System Using Stages of Continuous Improvement Methodology: Results From Serbia
(Your Name) has forwarded a page to you from Global Health: Science and Practice
(Your Name) thought you would like to see this page from the Global Health: Science and Practice web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Maturity Assessment of the Health Information System Using Stages of Continuous Improvement Methodology: Results From Serbia
Steve Ollis, Milan Kovačević, Bosiljka Djikanovic, Nikola Radoman, Isidora Smigic, Mamadou Alimou Barry
Global Health: Science and Practice Oct 2024, 12 (5) e2400083; DOI: 10.9745/GHSP-D-24-00083

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Share
Maturity Assessment of the Health Information System Using Stages of Continuous Improvement Methodology: Results From Serbia
Steve Ollis, Milan Kovačević, Bosiljka Djikanovic, Nikola Radoman, Isidora Smigic, Mamadou Alimou Barry
Global Health: Science and Practice Oct 2024, 12 (5) e2400083; DOI: 10.9745/GHSP-D-24-00083
del.icio.us logo Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Statistics from Altmetric.com

Jump to section

  • Article
    • ABSTRACT
    • INTRODUCTION
    • METHODS
    • RESULTS
    • DISCUSSION
    • CONCLUSION
    • Acknowledgments
    • Funding
    • Disclaimer
    • Author contributions
    • Competing interests
    • Notes
    • REFERENCES
  • Figures & Tables
  • Supplements
  • Info & Metrics
  • Comments
  • PDF

Related Articles

  • PubMed
  • Google Scholar

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

  • Creating a Career Development Path for Young Supply Chain Professionals: Three Case Studies in Benin, Kenya, and South Africa
  • Strategic Training Executive Program 2.0: A Leadership and Change Management Program for Health Supply Chains in Low- and Middle-Income Countries
  • Enhancing Antiretroviral Therapy Initiation for Hospitalized and Recently Discharged People Living With HIV in Johannesburg, South Africa
Show more PROGRAM CASE STUDY

Similar Articles

Subjects

  • Cross-Cutting Topics
    • Health Systems
Johns Hopkins Center for Communication Programs

Follow Us On

  • LinkedIn
  • Facebook
  • RSS

Articles

  • Current Issue
  • Advance Access Articles
  • Past Issues
  • Topic Collections
  • Most Read Articles
  • Supplements

More Information

  • Submit a Paper
  • Instructions for Authors
  • Instructions for Reviewers

About

  • About GHSP
  • Advisory Board
  • FAQs
  • Privacy Policy
  • Contact Us

© 2025 Creative Commons Attribution 4.0 International License. ISSN: 2169-575X

Powered by HighWire