Skip to main content

Main menu

  • Content
    • Current Issue
    • Advance Access
    • Archive
    • Supplements
      • The Challenge Initiative Platform
    • Topic Collections
  • For Authors
    • Instructions for Authors
    • Submit Manuscript
    • Publish a Supplement
    • Promote Your Article
    • Resources for Writing Journal Articles
  • About
    • About GHSP
    • Editorial Team
    • Advisory Board
    • FAQs
    • Instructions for Reviewers
  • Webinars
    • Local Voices Webinar
    • Connecting Creators and Users of Knowledge
    • Publishing About Programs in GHSP
  • Other Useful Sites
    • GH eLearning
    • GHJournal Search

User menu

  • My Alerts

Search

  • Advanced search
Global Health: Science and Practice
  • Other Useful Sites
    • GH eLearning
    • GHJournal Search
  • My Alerts

Global Health: Science and Practice

Dedicated to what works in global health programs

Advanced Search

  • Content
    • Current Issue
    • Advance Access
    • Archive
    • Supplements
    • Topic Collections
  • For Authors
    • Instructions for Authors
    • Submit Manuscript
    • Publish a Supplement
    • Promote Your Article
    • Resources for Writing Journal Articles
  • About
    • About GHSP
    • Editorial Team
    • Advisory Board
    • FAQs
    • Instructions for Reviewers
  • Webinars
    • Local Voices Webinar
    • Connecting Creators and Users of Knowledge
    • Publishing About Programs in GHSP
  • Alerts
  • Visit GHSP on Facebook
  • Follow GHSP on Twitter
  • RSS
  • Find GHSP on LinkedIn
TAKING EXCEPTION
Open Access

Response to “A False Dichotomy: RCTs and Their Contributions to Evidence-Based Public Health”

James D Shelton
Global Health: Science and Practice March 2015, 3(1):141-143; https://doi.org/10.9745/GHSP-D-15-00045
James D Shelton
aEditor-in-Chief, Global Health, Science and Practice.
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: jshelton@ghspjournal.org
PreviousNext
  • Article
  • Info & Metrics
  • Comments
  • PDF
Loading

While randomized controlled trials (RCTs) can and do make valuable contributions, they also have severe limitations, including in answering the basic question of “Does it work?” and, even more so, in steering how to proceed with complex public health programming at scale. They deserve no exalted position in the pantheon of methodologies for evidence-based public health.

See related articles by Hatt and Shelton.

I appreciate the thoughtful response by Hatt et al.1 to my editorial on evidence-based public health2 and am happy we agree on several points:

  1. There is some definite value to randomized trials.

  2. Because public health operates in complex program environments, it is generally necessary to lay out a valid “theory of change” or causal pathway to understand and evaluate the intended and actual program effects.

  3. Mixed-method approaches are essential to understand what is going on in such complex arenas.

“Does it work” is always affected by context. The paradigm Hatt et al. put forward asserts that one strength of randomized trials is to answer definitively, “Does it work?” But for the kinds of complex programs public health must muster, there is generally no absolute answer to that question or to its companion question, “How well does it work?” Rather, the answer much depends on how and in what situation “it” is done. That is often true even for fairly consistent biologic phenomena, such as the wide variation in polio vaccine efficacy that I cited in my editorial. Perhaps the most recent example is the use of antenatal corticosteroids to prevent newborn mortality, which is effective in developed countries with sophisticated health resources but was found to actually increase mortality in certain resource-constrained environments.3 Yes, randomized trials and other research methods can answer, “Can it work?” and are often fairly generalizable for discrete biologic questions. But for programming at scale, that takes us only part of the way.

How it might be made to work practicably at scale is the key question for public health. A good example is the paper by Curry et al. in this issue of GHSP, describing the many programmatic elements implemented to provide contraception very successfully and at fairly large scale in crisis-affected situations.4 Had the program failed to assure an effective supply chain, provide competency-based training, ensure good supervision, and mobilize communities, would the results have been so successful? Highly doubtful. And the richness of their evidence is enhanced by their description of the effect of variation in country context; for example, policy differences on the availability of contraceptive implants and the impact of poor compensation of health workers in certain places made big differences.

Variation in country context can make big differences in the effects of interventions.

Ironically, some of the randomized studies Hatt et al. cite neither completely answer the question of “Does it work?” nor provide enough understanding through mixed methods of what is going on.

  • The study on pay for performance in Rwanda found an increase in such outcomes as institutional deliveries and children's preventive health visits but no increase in completion of 4 prenatal care visits or of full immunization schedules.5 And other than some “anecdotal evidence,” we are left wondering about the crucial question of how the incentive system actually may have influenced the behavior of providers and clients.

  • The deworming study in Kenya found a decrease in worms and an increase in school participation and attendance.6 But there was no discernible impact on anemia (a commonly hypothesized mechanism to affect school attendance) through which deworming might improve school attendance and there was no impact on actual school achievement. Also the study was carried out in a situation with high worm infestation. Might the results be different in areas with less infestation? And we have no direct information from families themselves on what may have influenced school attendance.

Community deworming is actually a prime example of the very kind of variability that undermines the generalizability of randomized trials. Notably, a 2012 Cochrane review of numerous studies on community deworming concluded7:

For haemaglobin and cognition, community deworming seems to have little or no effect, and the evidence in relation to school attendance and school performance is generally poor, with no obvious or consistent effect. Our interpretation of these data is that it is probably misleading to justify contemporary deworming programmes [emphasis added] based on consistent benefit on nutrition, haemoglobin, school attendance or school performance as there is simply insufficient reliable information to know whether this is so.

Most studies failed to show impact, inescapably because situations varied. So actually, the issue of whether an intervention “is effective on a larger scale” has no single answer. And the recent decision in India to extend mass deworming to large populations has been justifiably roundly criticized.8

While I really do appreciate randomized studies, perhaps my biggest concern is the “hierarchy” whereby some colleagues place controlled trials at the top of a pyramid as manifestly the best evidence. For understanding public health programming, I see that as quite misguided. Randomized studies help us to understand some things, but they are only one piece of the picture in “triangulating” evidence for programming. And evidence from real-world programming is especially key.

RCTs are only one piece of the picture in triangulating evidence for public health programming.

Building on the core randomized component and adding other methodologies to the conduct of those randomized trials makes them much more useful. The large Mexico PROGRESA conditional cash transfer program focusing on health, education, and nutrition9 cited by Hatt et al. is to some extent an example of such a mixed-method approach and was conducted at large scale. My fear, however, is that the desire to control the research environment, and the resulting narrow focus and often artificiality of trials, limits understanding of the potential programmatic practicability. It also may limit the ability to use additional methodologies to help answer whether the intervention might work practicably at scale.

Thus, to answer the questions well of what, how, and why an intervention may have worked, we need lots of methodologies. Ultimately, some of the “best evidence” or gold standard comes from programs already operating successfully at scale, as illustrated by the paper by Curry.4 For public health programming, there is no absolute methodological hierarchy. We need to respect and use all legitimate methodologies.

Notes

Cite this article as: Shelton JD. Response to “a false dichotomy: rcts and their contributions to evidence-based public health”. Glob Health Sci Pact. 2015;3(1):141-143. http://dx.doi.org/10.9745/GHSP-D-15-00045.

  • © Shelton.

This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are properly cited. To view a copy of the license, visit http://creativecommons.org/licenses/by/3.0/. When linking to this article, please use the following permanent link: http://dx.doi.org/10.9745/GHSP-D-15-00045.

References

  1. ↵
    1. Hatt LE,
    2. Chatterji M,
    3. Miles LS,
    4. Comfort AB,
    5. Bellows BW,
    6. Okello FO
    . A false dichotomy: RCTs and their contributions to evidence-based public health. Glob Health Sci Pract. 2015;3(1): 138-140. doi:10.9745/GHSP-D-14-00245
    OpenUrlFREE Full Text
  2. ↵
    1. Shelton JD
    . Evidence-based public health: not only whether it works, but how it can be made to work practicably at scale. Glob Health Sci Pract. 2014;2(3): 253-258. doi:10.9745/GHSP-D-14-00066. pmid:25276583
    OpenUrlFREE Full Text
  3. ↵
    1. Vogel JD,
    2. Souza JP,
    3. Gulmezoglu AM,
    4. Mori R,
    5. Lumbiganon P,
    6. Qureshi Z,
    7. et al
    . Use of antenatal corticosteroids and tocolytic drugs in preterm births in 29 countries: an analysis of the WHO Multicountry Survey on Maternal and Newborn Health. Lancet. 2014;384(9957): 1869-1877. doi:10.1016/S0140-6736(14)60580-8. pmid:25128271
    OpenUrlCrossRefPubMed
  4. ↵
    1. Curry DW,
    2. Rattan J,
    3. Nzau JJ,
    4. Giri K
    . Delivering high-quality family planning services in crisis-affected settings I: program implementation. Glob Health Sci Pract. 2015;3(1): 14-24. doi:10.9745/GHSP-D-14-00164
    OpenUrlAbstract/FREE Full Text
  5. ↵
    1. Basinga P,
    2. Gertler PJ,
    3. Binagwaho A,
    4. Soucat ALB,
    5. Sturdy J,
    6. Vermeersch CMJ
    . Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: an impact evaluation. Lancet. 2011;377(9775): 1421-1428. doi:10.1016/S0140-6736(11)60177-3. pmid:21515164
    OpenUrlCrossRefPubMed
  6. ↵
    1. Miguel E,
    2. Kremer M
    . Worms: identifying impacts on education and health in the presence of treatment externalities. Econometrica. 2004;72(1): 159-217. doi:10.1111/j.1468-0262.2004.00481.x
    OpenUrlCrossRef
  7. ↵
    1. Taylor-Robinson DC,
    2. Maayan N,
    3. Soares-Weiser K,
    4. Donegan S,
    5. Garner P
    . Deworming drugs for soil-transmitted intestinal worms in children: effects on nutritional indicators, haemoglobin and school performance. Cochrane Database Syst Rev. 2012;11:CD000371. doi:10.1002/14651858.CD000371.pub5. pmid:23152203
    OpenUrlCrossRefPubMed
  8. ↵
    1. Madur G
    . Deworming programme in India sparks controversy. BMJ. 2015;350:h720. doi:10.1136/bmj.h720. pmid:25672781
    OpenUrlFREE Full Text
  9. ↵
    1. Gertler P
    . Do conditional cash transfers improve child health? Evidence from PROGRESA’s control randomized experiment. Am Econ Rev. 2004;94(2): 336-341. doi:10.1257/0002828041302109
    OpenUrlCrossRef
PreviousNext
Back to top

In this issue

Global Health: Science and Practice: 3 (1)
Global Health: Science and Practice
Vol. 3, No. 1
March 01, 2015
  • Table of Contents
  • About the Cover
  • Index by Author
  • Complete Issue (PDF)
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word about Global Health: Science and Practice.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Response to “A False Dichotomy: RCTs and Their Contributions to Evidence-Based Public Health”
(Your Name) has forwarded a page to you from Global Health: Science and Practice
(Your Name) thought you would like to see this page from the Global Health: Science and Practice web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Response to “A False Dichotomy: RCTs and Their Contributions to Evidence-Based Public Health”
James D Shelton
Global Health: Science and Practice Mar 2015, 3 (1) 141-143; DOI: 10.9745/GHSP-D-15-00045

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Share
Response to “A False Dichotomy: RCTs and Their Contributions to Evidence-Based Public Health”
James D Shelton
Global Health: Science and Practice Mar 2015, 3 (1) 141-143; DOI: 10.9745/GHSP-D-15-00045
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Statistics from Altmetric.com

Jump to section

  • Article
    • Notes
    • References
  • Info & Metrics
  • Comments
  • PDF

Related Articles

  • No related articles found.
  • PubMed
  • Google Scholar

Cited By...

  • No citing articles found.
  • Google Scholar

More in this TOC Section

  • Response to Austad: Offering a Range of Methods, Including Fertility Awareness Methods, Facilitates Method Choice
  • Fertility Awareness Methods Are Not Modern Contraceptives: Defining Contraception to Reflect Our Priorities
  • A False Dichotomy: RCTs and Their Contributions to Evidence-Based Public Health
Show more TAKING EXCEPTION

Similar Articles

US AIDJohns Hopkins Center for Communication ProgramsUniversity of Alberta

Follow Us On

  • Twitter
  • Facebook
  • LinkedIn
  • RSS

Articles

  • Current Issue
  • Advance Access Articles
  • Past Issues
  • Topic Collections
  • Most Read Articles
  • Supplements

More Information

  • Submit a Paper
  • Instructions for Authors
  • Instructions for Reviewers
  • GH Journals Database

About

  • About GHSP
  • Advisory Board
  • FAQs
  • Privacy Policy
  • Contact Us

© 2023 Creative Commons Attribution 4.0 International License. ISSN: 2169-575X

Powered by HighWire