Getting evidence into practice: what works in developing countries?

Abstract

Purpose. We summarize and comment on the available literature on the effectiveness of interventions designed to change professional behaviour in order to bring evidence into practice in developing countries.

Data sources. We used a strategy adapted from the Effective Practice & Organization Care (EPOC) Cochrane group.

Study selection. Forty-four studies met pre-defined selection criteria. Controlled and uncontrolled trials of interventions were included. Studies measured either professional compliance with agreed standards or patients’ clinical outcomes.

Data extraction. Data were extracted using a pre-defined extraction tool and studies were appraised accordingly.

Results of data synthesis. Data were synthesized and categorized according to different types of intervention. Audit and feedback was found to be effective, at least in the short term, when combined with other approaches. Similarly, educational interventions were more effective when designed to address local educational needs and organizational barriers. We found insufficient evidence to assess the effectiveness of educational outreach, local opinion leaders, use of mass media, and reminders. Educational materials alone are unlikely to influence change. However, the majority of studies had weak designs and failed to exclude possible biases.

Conclusion. Current evidence for the effectiveness of interventions to change health professionals’ behaviour in developing countries is either scanty or flawed due to poorly designed research. Given the recent drive to improve quality of care, this should be a priority area for researchers and international agencies supporting health systems development in developing countries. This review provides an insight into some of the methodological issues that interested researchers may face.

Evaluations of primary care in developing countries have identified deficiencies in the diagnosis, recording, monitoring, treatment, and counselling of patients in child development, immunization, malaria, diarrhoea, and acute respiratory infections [1]. Causes include constraints on human and financial resources, poor health systems, lack of infrastructure, and failure to provide care in accordance with standards of good practice [2]. This increasing realization of deficiencies in the quality of service provision and wide gaps between evidence and practice makes it increasingly important to influence change in health professionals’ clinical practice in developing countries. Use of explicit clinical guidelines and policies to inform and reinforce good practice is often desirable. However, in the majority of such settings, such clinical policies either do not exist or are very poorly complied with [3]. The need to put ‘what works’ into practice is particularly important in resource-poor countries [4]. Ineffective treatments can drain limited resources resulting in further health inequalities. The World Health Organization (WHO) calls for its members to focus on strengthening health systems, and in particular to bring existing evidence into practice [5].

Interventions to bring evidence into practice have not been developed and tested extensively in developing countries. We found only one systematic review (28 studies) examining interventions to influence health professionals’ behaviour in developing countries [6]. This review, limited to only prescribing behaviour, concluded that the majority of studies demonstrated a beneficial effect of these interventions but were of poor design and focused solely on the public sector. We believe there is a need for a wider review of the effectiveness of such interventions. We focus on interventions that are designed to enhance uptake of clinical guidelines, protocols, and policies. Other interventions, which work through organizational changes, financial incentives, or regulatory structures, are beyond the scope of this review.

Search method

We searched (search date 23 September 2003) Medline (from 1966), Embase (from 1980), and Health Management Information Consortium databases using the keywords audit, chart review, feedback, education, information, opinion leaders, outreach, guidelines, algorithm, consensus and combined these with names of developing countries. The full search strategy, adapted from the Effective Practice & Organization Care (EPOC) Cochrane group, is available on request. Articles were not restricted by language. Reference lists of other Cochrane reviews from the EPOC group were manually searched for relevant articles. The International Journal of Quality in Health Care and Health Policy and Planning were also hand-searched for relevant articles published in the last 10 years. Inclusion and exclusion criteria were based on target population (health care professionals), type of intervention (see Table 1), outcomes (compliance with the agreed guidelines and/or clinical outcomes), and research designs [systematic reviews, randomized controlled trials (RCTs) and quasi-experimental designs]. Included articles were appraised and synthesized accordingly.

Results

Out of 3569 citations, 44 articles were subsequently included in this review (Table 2). These findings are summarized in the following sections according to intervention type.

Audit and feedback

Audit and feedback is defined as any summary of performance of health care over a specified period, with or without recommendations for action [7]. We found 15 studies on audit and feedback, only one of which was a RCT [8]. In this study, hospital paediatricians in Thailand participated in clinical audit and peer review. This resulted in increased appropriateness of investigation ordering and reduction in the laboratory utilization cost per patient. The consultants’ investigation ordering showed less improvement compared with residents’, perhaps due to existing optimum performance. Some cross-contamination might have occurred as all participants were working in the same hospital.

Three studies used a controlled non-randomized design. In Niger, an evaluation of self- and peer-assessment in community paediatric services found improvements in the process of care (compliance with fever management standards) but that no structural changes occurred, reflecting local resource issues and management complexities in dealing with such improvements [9]. Direct observation used in this study to measure compliance could have biased observed behaviour. In Ecuador, clinical audit supported by local quality assurance teams and users’ committees in obstetrics and gynaecology departments in four district hospitals resulted in improved compliance with care standards but little improvement in patient satisfaction and service utilization [10]. In Thailand, similar findings were observed in compliance by hospital casualty staff with universal precautions [11]. However, the improvement was not sustained in the post-intervention period.

Of the 11 non-controlled before-and-after studies, seven assessed health care professionals’ compliance with various prescribing and clinical management guidelines. Two found improved compliance in primary care with national hypertension and diabetes management guidelines in Trinidad and Lebanon, respectively [12,13]. A multifaceted approach to feedback demonstrated improved compliance with prescribing guidelines by hospital doctors in Thailand [14]. All four of the other studies measuring compliance of health care professionals against agreed standards also showed improvement [15–18]. Four studies observed only clinical outcomes before and after the clinical audit and feedback. Three of these, all conducted in maternity units in Pakistan, Mozambique, and South Africa, demonstrated improved outcomes [19–21]. The fourth, conducted in a cardio-thoracic unit in India, also showed improved outcomes but confounding factors may have played a part in this improvement [22].

Almost all studies of clinical audit showed it to improve compliance with agreed standards or clinical outcomes. However, the majority had poor study design. Multifaceted approaches were shown to be more effective than a single intervention. Audit combined with local consensus-based guideline development or peer review and targeted training was particularly successful [23,24]. Change was more visible when audit design suited local culture in the health system, where those who need to implement change were involved, and recommendations were feasible given available resources and existing practices [25]. Change strategies were more successful when problems were analysed and solutions were sought with the help of focus groups. These helped in identifying ‘avoidable factors’ in order to inform recommendations for change [26]. Rapid change was observed where government introduced audit and feedback as a quality assurance tool with clear allocation of resources [10]. Almost all studies in the review demonstrated improvement in the short term but often this was not sustained.

Education and training

Education and training interventions are defined by Davis et al. [27] as ‘transferring knowledge to practicing professionals with an aim to change their practice’. We found 19 studies assessing such interventions, four of which were RCTs. Three of the RCTs determined the effect of educational interventions on the prescribing behaviours of primary care doctors: in Zambia, an increased number of appropriate antibiotic prescriptions occurred after a series of educational seminars [28]; in Cuba, prescribing habits improved after a series of refresher courses in the management of acute respiratory infections [29]; and in Indonesia a reduction in the use of injections and inappropriate antibiotics was observed after educational interactional group discussions, which were designed to address cognitive dissonance between physicians’ perceptions and patients’ beliefs [30]. The fourth RCT, in primary care in South Africa, documented improved compliance with sexually transmitted diseases guidelines following training workshops and dissemination of educational packs, treatment packs, partner packs, information leaflets, and condoms [31]. However, simulated patients were used, which may have altered clinical behaviour. Moreover, the educational component was not assessed separately.

We found six non-randomized controlled studies, two of which showed improved doctor–patient communication during consultation following an intensive training course of primary and secondary care doctors in Trinidad and Honduras, respectively [32,33]. However, some selection bias may have arisen due to voluntary participation. Moreover, audiotaping of the encounters may have altered professional behaviour. Similarly, in another study in Guatemala, direct observation may have altered clinical behaviour of primary care health professionals who were shown to have increased compliance with management guidelines for diarrhoeal diseases after an education intervention designed for distant learning [34]. Improved prescribing behaviour was also observed among primary care physicians in Mexico after multiple educational interventions [35]. However, improvement was not sustained long term. In Nigeria, training workshops also resulted in an increased proportion of scheduled follow-up visits in a family planning service [36]. A similar study in Ghana demonstrated partial improvement in clinical outcomes after providing training to the traditional birth attendants [37]. Observation methods relied on subjective information, which may have been biased.

The other nine non-controlled studies also demonstrated moderate improvements in compliance with a variety of management guidelines and protocols in various settings [38–46]. One such study in Mexico was successful in engaging private primary care practitioners and demonstrating improved compliance with case management guidelines of common illnesses [38]. Two studies in Peru and China showed significant improvement following the introduction of an educational package which was informed by an educational needs assessment of the primary and secondary care health professionals, respectively [42,46].

Nearly all studies in this category concluded that educational strategies improve health professionals’ compliance with agreed standards and a sizeable proportion (55%) of these studies used controlled designs. The educational strategies included a seminar, intensive courses, distant learning courses and interactional sessions. No educational design was shown to have an advantage over others but strategies using multiple methods were shown to be more successful. Interventions informed by local educational needs and provided in keeping with the organizational culture were also shown to be successful [42]. Similarly, educational sessions addressing barriers and building on facilitators to change were also effective [30]. However, educational interventions improvements were observed only in the short term and not sustained in the longer term.

Local consensus development

Local consensus processes are defined as decision-making processes that include providers in identifying, prioritizing, and developing solutions. We found only three studies of such processes, all using before-and-after designs to assess the impact of developing local consensus on clinical policies. In Senegal, partial improvement in the prescribing behaviour among primary care physicians was observed subsequent to the development of local guidelines and algorithms; however, a 5-year gap existed between the pre- and post- observations, which may have introduced other confounding factors [47]. In Tanzania, little improvement was found in the appropriateness of hospital-based transfusion requests after the implementation of local protocols, the only improvement being where more focused feedback and training were provided [48]. Likewise, a study in South Africa found that locally developed guidelines failed to change the prescription pattern of doctors in the management of hypertension in community hospitals [49]. However, some cost reduction was observed, probably due to other prescription restrictions introduced simultaneously.

Educational outreach

Educational outreach, often referred to as academic detailing, has been defined as an educational visit by a trained person to a health professional in his/her own setting to influence professional practice [50]. We found two studies by the same research group, one using a before-and-after design in Kenya and the other an RCT in Indonesia, assessing a face-to-face educational intervention for community pharmacists, taking into consideration their educational needs and other local constraints and facilitators [51]. Both demonstrated improved practice.

Local opinion leaders

Local opinion leaders are health professionals nominated by their colleagues as ‘educationally influential’, and are likely to influence their colleagues’ professional behaviour [52]. An RCT in Hong Kong found that education and supervision provided by senior ward nurses gave better compliance with nursing management guidelines than educational interventions alone [53]. However, improvements were based on subjective surveys and direct observations, potentially biasing results.

Educational materials

These include interventions such as printed practice guidelines, manuals, pamphlets, newsletters, and audio-visual materials aiming to influence professional behaviour. An RCT conducted in Sri Lanka to test the value of therapeutic bulletins among primary and secondary care doctors in influencing their prescribing behaviour found a non-significant reduction in antibiotic prescription [54]. However, the study used a small sample size thereby having low power to detect change.

Marketing, reminders, and patient-mediated interventions

Marketing interventions involve the use of personal interviewing, group discussion (focus groups), or a survey of targeted providers to identify barriers to change and subsequent design of interventions [7]. Patient-mediated interventions include any intervention aimed at changing the performance of health care providers for which information is sought from or given directly to patients by others (e.g. direct mailing to patients, patient counselling delivered by others, or clinical information collected from patients and given to providers) [7]. Reminders are defined as any intervention that prompts the health care provider to perform a clinical action [7]. No articles were found in any of these categories.

Multiple strategies

Due to the potential overlap between various interventions, many studies evaluated a complex intervention. However, only three researchers have explicitly acknowledged this. In Indonesia, an RCT to assess interventions to improve the management of diarrhoea compared face-to-face educational outreach, didactic educational interventions designed for primary care professionals, and no intervention [55]. The use of anti-diarrhoeals and antibiotics was reduced. In Uganda, another RCT demonstrated additional benefit of audit and feedback on simple educational materials and interventions in improving compliance with WHO prescription indicators for drug use in common illnesses by primary care practitioners [23]. In Mali, a non-randomized controlled study compared peer review and feedback with educational strategies and found that both resulted in short-term improvement in compliance by community nurses with the integrated management of childhood illnesses algorithm [24]. However, feedback was more effective in more poorly performing areas.

Discussion

The majority of the studies reviewed here have shown improved professional performance subsequent to intervention. However, various Cochrane reviews on influencing professional behaviour (including studies mainly from developed countries) have shown mixed results [50,52,56–58]. Publication bias against studies with weaker designs conducted in the developing world may have prevented many studies appearing in peer-reviewed journals. This is compounded by the fact that many journals until recently were unwilling to publish research with negative findings and it seems likely that in this field, negative results are not even submitted for publication [59].

Only eight studies in this review were RCTs. Of the rest, 10 studies had a control group (which in some studies was not comparable to the intervention group), making it difficult to claim a causal link between intervention and outcome. Most other non-controlled studies had poor designs and often failed to consider the socio-cultural context in which inappropriate clinical behaviour takes place [6]. These studies examined the influence on the professional behaviour of different categories of health professionals (doctors, nurses, health visitors, midwives, etc.). However, the heterogeneity of the studies makes it very difficult to draw any conclusions on the differential impact on these professional categories. Little evidence was found on influencing professional behaviour in the private sector despite its strong presence, often unregulated, in many countries.

Two broad categories of outcome were measured in the studies included in this review: compliance with clinical guidelines/ policies/protocols and patient outcomes. Except in a few RCTs, effect size was generally measured in terms of proportional improvement in compliance with clinical guidelines and patient outcome. A wide range of proportional improvement (10–100%) was observed in the studies measuring improved compliance. Observed behaviour also varied from prescribing, test ordering, referring, and recording clinical information. Due to the wide variation between observed professional behaviour, interventions, and study designs, it was not possible to combine the effect size of these studies. Improvements observed in a variety of patient outcomes could also not be combined for similar reasons. Studies often used direct observation, interviews, audio- or video-recording, and simulated patients to determine change in practice. Such methods are liable to many biases [2].

Changes were often observed only on a short-term basis. The improvement observed subsequently either disappeared or diminished (see Results).

Interventions were often tested in combination. These trials rarely had a comparative design to tease out the effectiveness of different components of the intervention. Many non-controlled studies used changes in local mortality trends to justify their conclusions but failed to adjust for the general mortality trend in the rest of the country. Several controlled studies failed to compare the characteristics of the participants in the controlled and the intervention arm. Many studies lacked power to detect statistically significant differences and often did not state P values and confidence intervals. The cost implications of interventions were not discussed in the vast majority of studies.

Despite being widely practised in developed nations, clinical audit has not attracted much attention in many developing countries, for reasons not yet fully understood. However, in countries with limited resources and extra pressure to provide quality services at minimal cost, interventions such as clinical audit could potentially be of immense value. The combined functions of clinical audit, monitoring practice, and education can have particular value in health services in developing countries, with their constraints on human and financial resources [18].

The evidence found in this review is insufficient to advocate the use of educational outreach and local opinion leaders. However, research from developed countries suggests that these approaches can be promising in the right circumstances [50,52].

This review has identified many research gaps in this field. Use of local opinion leaders, educational bulletins, patient mediated reminders, social marketing, and educational outreach should be examined in the context of developing country health systems. More research is needed into ways of influencing private practitioners and non-medical community/primary health care workers. Given its potential, the impact of educational outreach in influencing private practitioners and other public sector primary care physicians would also be worth exploring. However, the cost effectiveness of such interventions must also be evaluated in order to justify their use.

Conclusion

Strong evidence exists for the effectiveness of audit and feedback, educational strategies, opinion leaders, educational outreach, and local consensus development in combination with other interventions [50,52,56–58,60]. However, most of this is based on studies conducted in developed countries. Since such interventions are dependent on local factors, it is desirable to have strong evidence of their effectiveness in the developing world. Such evidence does not yet exist. Evaluation of tools influencing professional practice, such as audit and feedback, should be a high research priority in developing countries. International institutions with greater experience of health systems where quality assurance is routine should provide technical support to developing countries in conducting such research. This review provides a useful insight to future researchers into some of the methodological issues of conducting similar research in developing countries.

References