Elsevier

Social Science & Medicine

Volume 85, May 2013, Pages 87-92
Social Science & Medicine

Short report
Utilizing community health worker data for program management and evaluation: Systems for data quality assessments and baseline results from Rwanda

https://doi.org/10.1016/j.socscimed.2013.02.033Get rights and content

Abstract

Community health workers (CHWs) have and continue to play a pivotal role in health services delivery in many resource-constrained environments. The data routinely generated through these programs are increasingly relied upon for providing information for program management, evaluation and quality assurance. However, there are few published results on the quality of CHW-generated data, and what information exists suggests quality is low. An ongoing challenge is the lack of routine systems for CHW data quality assessments (DQAs). In this paper, we describe a system developed for CHW DQAs and results of the first formal assessment in southern Kayonza, Rwanda, May–June 2011. We discuss considerations for other programs interested in adopting such systems. While the results identified gaps in the current data quality, the assessment also identified opportunities for strengthening the data to ensure suitable levels of quality for use in management and evaluation.

Highlights

► Community health workers (CHW) are key in health systems, collecting program planning or evaluation data used at all levels. ► Assessing CHW data quality by lot quality assurance sampling is a viable routine method for supervisors or program managers. ► Reliability of data was poor among our sample of health centers in Rwanda in 2011 due to several CHW and program factors. ► Appropriate interventions may address relevant factors at health units identified with poor data quality.

Introduction

Community health workers (CHWs) have been used to decentralize important health services for decades, including to help achieve “health for all” (Berman, Gwatkin, & Burger, 1987; WHO, 1978) and more recently to reduce child and maternal mortality (Bhutta, Darmstadt, Hasan, & Haws, 2005; Edward et al., 2007; Hafeez, Mohamud, Shiekh, Shah, & Jooma, 2011; Haines et al., 2007; Lehmann & Sanders, 2007; Prata et al., 2012; de Sousa, Tiedje, Recht, Bjelic, & Hamer, 2012; United Nations Statistics Division, 2012; WHO & Global Health Workforce Alliance, 2010; WHO/UNICEF, 2004a, b; Young, Wolfheim, Marsh, & Hammamy, 2012). The goals of such programs include increasing access to care by removing barriers of distance and costs, identifying and treating illness earlier, and monitoring uptake of health programs.

In addition to improving access to health care, CHWs are collecting and reporting on a large volume and range of information on a routine basis. Currently, these data are used at all levels to monitor, manage and evaluate CHW programs. Further, the data provide information to guide health system priorities, identify gaps in service delivery, and detect any emerging health issues. Indeed, a well-functioning health information system, including the data collected during community-level service delivery, forms the backbone of an effective health system (WHO, 2007).

However, concerns exist around the quality of CHW data and as a result, the appropriateness of using them to guide program management and evaluation. Studies of facility-based health information systems carried out in sub-Saharan African countries have shown that data (both paper and electronic) can be of sub-optimal quality (Forster et al., 2008; Garrib et al., 2008; Makombe et al., 2008; Maokola et al., 2011; Mate, Bennett, Mphatswe, Barker, & Rollins, 2009; Mavimbe, Braa, & Bjune, 2005; Ndira, Rosenberger, & Wetter, 2008; Rowe et al., 2009). To date, very few studies have assessed the quality of and factors affecting data collected and reported by CHWs. A reliability study in Kisumu, Kenya found varying quality of CHW data depending on activity area; for example, latrine and antenatal care use were more reliable than immunization coverage indicators (Otieno, Kaseje, Ochieng, & Githae, 2012). Authors found accuracy in monthly reports to be <50% in the national Lady Health Workers program in Pakistan, attributed to weak supervision and inappropriate and numerous data collection instruments (Mahmood & Ayub, 2010). In another study in Ghana, misreporting was common, with health center staff shortages during community outreach being associated with under-reporting (Helleringer, Frimpong, Phillips, Avoonor-Williams, & Yeji, 2010). While tools exist to assess the quality of facility-based data (Aqil, Lippeveld, & Hozumi, 2009; The Global Fund, 2008), there is no such standardized, routine methodology for measuring CHW data quality either on its own or as part of a broader community-based health information system.

The Rwanda community health program was initiated in 1995 by the Ministry of Health (MoH) and now has 45,011 CHWs (MoH, 2008; Mugeni, 2012). In 2005, the MoH revised the CHW system to deliver a broader range of preventive and curative services, and is currently training different types of CHWs to carry out these activities. In each village, there is a maternal health CHW who monitors pregnant women and their newborns and at least two multi-disciplinary CHWs who carry out: 1) integrated community case management (iCCM) (a global strategy adapted by Rwanda) (MoH, 2011; Young et al., 2012); 2) malnutrition screening; and 3) other preventive and behavior change activities.

The MoH has established a standardized community health information system that makes CHW data available at the sub-national and national levels. For this paper, we focused on data collected and reported by multi-disciplinary CHWs for their iCCM activities (assessment, classification and treatment or referral of diarrhea, pneumonia, malaria and malnutrition in children aged under five years). During each sick child visit, CHWs complete an encounter form (hereafter “form”) using an algorithm to guide them. Key data from each form are transferred to the iCCM consultation register (hereafter “register”) and the CHWs use the registers to generate the monthly, village-level report (hereafter “report”) (Electronic Supplementary Materials). The village reports are then aggregated – first at the cell level (a cluster of villages); then at the sector level (corresponding to the health center) after which they are sent to the district level, where the data are entered electronically and transmitted to the central level (Fig. 1).

As in other CHW programs, the MoH and partners use CHW-collected data for program management, evaluation and quality assurance of CHW activities. Additionally, some data are included in the national performance-based financing (PBF) system whereby the MoH makes quarterly payments into CHW cooperatives based on the quantity of certain activities reported and the timeliness and completeness of the reports. Because of the reliance on CHW data, maintaining quality data in the forms, registers and reports is of utmost importance. Below, we present details on the system developed for assessing CHW data quality and results from the first implementation. We discuss the strengths and limitations of the data quality assessment (DQA) system to facilitate adoption of such assessments for other CHW programs.

Section snippets

Methods

A DQA system was designed and field tested at the eight health centers (HC1-8) in southern Kayonza – one of three rural districts in which the organization Partners In Health (PIH) has worked since 2005 with the Government of Rwanda (GoR) to support their Health Sector Strategic Plan (GoR MoH, 2005, 2009, 2010). In southern Kayonza, PIH provides technical and financial support to the national community health program based out of Rwinkwavu District Hospital to strengthen CHW training,

Evaluation of report quality

During the field test of the LQAS-based DQA system, we reviewed the reports, registers and forms from 175 out of 251 total villages, representing the work of 463 CHWs. Of the 175 reports, only one was missing (from HC8), and 140 were included in the LQAS analysis.

The number of reports discordant with the registers and the corresponding LQAS classifications are presented in Table 1. With few exceptions, health centers were classified as having “poor” report quality for all three indicators. All

Discussion

The sub-national and national use of CHW data for program management, evaluation, and quality assurance depends on the: 1) establishment of a system to facilitate data collection and feedback between the local and national levels; and 2) confidence that the data are sufficient quality to warrant their use. Despite the development and use of a standardized information system in Rwanda, the results from the DQA field test demonstrate that there are challenges to achieving high quality CHW

Funding

This work was supported by the Doris Duke Charitable Foundation (2009057), and is part of a larger impact evaluation being carried out by Partners In Health and Government of Rwanda partners. BHG was supported by the Harvard Medical School Department of Global Health and Social Medicine Research Core.

Acknowledgments

Data officers Janet Umuhoza, Liliose Mukantaganzwa and Wellars Ndayambaje; community health workers and supervisors in southern Kayonza; Rwinkwavu District Hospital Medical Director Adolphe Karamaga and Community Health Supervisor Denys Ndanurura; and members of the PIH DDCF team Karen Finnegan, Megan Murray, Michael L. Rich and Tom Nicholson.

References (40)

  • GoR MoH

    Health sector strategic plan: July 2009–June 2012 (HSSP II)

    (2009)
  • GoR MoH

    Rwanda health systems strengthening framework and consolidated strategic plan 2009–2012

    (2010)
  • A. Hafeez et al.

    Lady health workers programme in Pakistan: challenges, achievements and the way forward

    Journal of the Pakistan Medical Association

    (2011)
  • B.L. Hedt-Gauthier et al.

    Improving data quality and supervision of antiretroviral therapy sites in Malawi: an application of Lot Quality Assurance Sampling

    BMC Health Services Research

    (2012)
  • Helleringer, S., Frimpong, J. A., Phillips, J. F., Avoonor-Williams, J. K., & Yeji, F. (2010). Abstract: Operational...
  • U. Lehmann et al.

    Community health workers: what do we know about them? The state of the evidence on programmes, activities, costs and impact on health outcomes of using community health workers

    (2007)
  • S.K. Lwanga et al.

    Sample size determination in health studies: a practical manual

    (1991)
  • S. Mahmood et al.

    Accuracy of primary health care statistics reported by community based lady health workers in district Lahore

    Journal of the Pakistan Medical Association

    (2010)
  • S.D. Makombe et al.

    Assessing the quality of data aggregated by antiretroviral treatment clinics in Malawi

    Bulletin of the World Health Organization

    (2008)
  • W. Maokola et al.

    Enhancing the routine health information system in rural southern Tanzania: successes, challenges and lessons learned

    Tropical Medicine & International Health

    (2011)
  • Cited by (19)

    • Successive introduction of four new vaccines in Rwanda: High coverage and rapid scale up of Rwanda's expanded immunization program from 2009 to 2013

      2016, Vaccine
      Citation Excerpt :

      All CHWs must have 6 years of education and payment is based upon performance [15]. CHWs are supervised by a coordinator at their local health post who provides information on immunization campaigns [15,16]. CHWs are additionally responsible for referring patients to health centers and hospitals.

    • Toward universal coverage in Afghanistan: A multi-stakeholder assessment of capacity investments in the community health worker system

      2015, Social Science and Medicine
      Citation Excerpt :

      In some contexts, CHWs have compromised, by selecting a restricted social group or only the tasks that are feasible (or rewarded), and neglecting other responsibilities (Hermann et al., 2009). Complimenting facility information systems with community based vital events registration and disease surveillance is of critical value, but requires considerable time investments (Mitsunaga et al., 2013). Periodic consultations with CHWs are essential for their feedback on constraints to performance to inform adjustments to task expectations.

    • Geographical accessibility to healthcare and malnutrition in Rwanda

      2015, Social Science and Medicine
      Citation Excerpt :

      An encouraging initiative in this direction is the Rwandan government's initiative to improve community health through the mobilization of community health workers (CHW). The government of Rwanda launched the Rwanda community health program in 1995 and currently operates around 45,000 CHWs (Mitsunaga et al., 2013). CHWs increase a community's access to healthcare and improve its health by monitoring maternal health, child nutritional status, and other diseases such as diarrhea, while offering basic curative and preventive care, educating community members in health issues, and referring cases of malnutrition and other diseases to health centers (Mitsunaga et al., 2013).

    View all citing articles on Scopus
    1

    Joint first authors.

    View full text