You are here

Article Text

Article menu

Download PDFPDF

XML
HIV/AIDS
Research
Do digital innovations for HIV and sexually transmitted infections work? Results from a systematic review (1996-2017)
  1. Jana Daher1,
  2. Rohit Vijh1,
  3. Blake Linthwaite1,
  4. Sailly Dave1,
  5. John Kim2,
  6. Keertan Dheda3,
  7. Trevor Peter4,
  8. Nitika Pant Pai1,5
  1. 1 Division of Clinical Epidemiology, Research Institute of the McGill University Health Centre, Montreal, Canada
  2. 2 National HIV/AIDS Labs, National Labs, Winnipeg, Manitoba, Canada
  3. 3 Department of Pulmonology, UCT Lung Institute, University of Cape Town, Cape Town, South Africa
  4. 4 Clinton Health Access Initiative (CHAI), Boston, USA
  5. 5 Department of Medicine, McGill University, Montreal, Quebec, Canada
  1. Correspondence to Dr Nitika Pant Pai; Nitika.Pai{at}mcgill.ca

Abstract

Objective Digital innovations with internet/mobile phones offer a potential cost-saving solution for overburdened health systems with high service delivery costs to improve efficiency of HIV/STI (sexually transmitted infections) control initiatives. However, their overall evidence has not yet been appraised. We evaluated the feasibility and impact of all digital innovations for all HIV/STIs.

Design Systematic review.

Setting/participants All settings/all participants.

Intervention We classified digital innovations into (1) mobile health-based (mHealth: SMS (short message service)/phone calls), (2) internet-based mobile and/or electronic health (mHealth/eHealth: social media, avatar-guided computer programs, websites, mobile applications, streamed soap opera videos) and (3) combined innovations (included both SMS/phone calls and internet-based mHealth/eHealth).

Primary and secondary outcome measures Feasibility, acceptability, impact.

Methods We searched databases MEDLINE via PubMed, Embase, Cochrane CENTRAL and Web of Science, abstracted data, explored heterogeneity, performed a random effects subgroup analysis.

Results We reviewed 99 studies, 63 (64%) were from America/Europe, 36 (36%) from Africa/Asia; 79% (79/99) were clinical trials; 84% (83/99) evaluated impact. Of innovations, mHealth based: 70% (69/99); internet based: 21% (21/99); combined: 9% (9/99).

All digital innovations were highly accepted (26/31; 84%), and feasible (20/31; 65%). Regarding impacted measures, mHealth-based innovations (SMS) significantly improved antiretroviral therapy (ART) adherence (pooled OR=2.15(95%CI: 1.18 to 3.91)) and clinic attendance rates (pooled OR=1.76(95%CI: 1.28, 2.42)); internet-based innovations improved clinic attendance (6/6), ART adherence (4/4), self-care (1/1), while reducing risk (5/5); combined innovations increased clinic attendance, ART adherence, partner notifications and self-care. Confounding (68%) and selection bias (66%) were observed in observational studies and attrition bias in 31% of clinical trials.

Conclusion Digital innovations were acceptable, feasible and generated impact. A trend towards the use of internet-based and combined (internet and mobile) innovations was noted. Large scale-up studies of high quality, with new integrated impact metrics, and cost-effectiveness are needed. Findings will appeal to all stakeholders in the HIV/STI global initiatives space.

  • mhealth/ehealth
  • innovations
  • hiv
  • sexually transmitted infections
  • systematic reviews

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/

https://doi.org/10.1136/bmjopen-2017-017604

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strength and limitations of this study

    • An updated and comprehensive systematic review/meta-analysis of all innovations in HIV/STI.

    • Evaluation of study quality with biases, subgroup analyses and sensitivity analyses.

    • Evaluation of metrics and measures for objective and subjective data.

    • Limited data were reported from Sub-Saharan Africa and Southeast Asia (29%, 29/99).

    • Limited evidence (18/99, 18%) was available for STIs (other than HIV).

    • Limited data on cost-effectiveness from high burden settings.

    • A lack of integrated online impact metrics to evaluate internet-based eHealth innovations.

    Introduction

    HIV/STIs remain a public health concern worldwide—a million new HIV/STIs are acquired every day, with cumulative disease burden exceeding 500 million infections.1–5 Regarding HIV, countries are working hard to achieve the new UNAIDS 90-90-90 treatment targets6; however, structural and societal barriers, such as stigma, low socioeconomic status and geographical isolation, impede access to quality care for marginalised populations who are disproportionately impacted by the HIV/AIDS epidemic.7 8 Likewise, a lack of timely testing and poor retention in care impairs the efforts to control HIV/STIs.7 9 10 To improve early testing, linkage and retention in care, healthcare systems globally are seeking solutions to improve population engagement, awareness and education, and efficient care for their hard-to-reach populations. It is imperative to plug gaps in healthcare service delivery.11 12 Digital innovations such as electronic health (eHealth), mobile health (mHealth) and combined innovations offer promising solutions to improve health service delivery. eHealth encompasses non-internet and internet-enabled mHealth as well as other internet-based health interventions. These innovations, together with expanded mobile and internet networks, global connectivity and affordability, present opportunities to change the future landscape of healthcare service delivery.

    The World Bank estimates that globally 96% of the world’s population and 70% of the world’s poorest have access to a mobile phone.13 Of seven billion, two billion (30%) individuals own a smartphone; approximately 50% of mobile phone users access the internet through their phones.14 15 Technological access has created a portal for social media and other internet-based health interventions.16 A rapid diffusion of mobile phones and internet technologies are prime drivers of this disruptive phenomenon in health, aptly titled, the creative destruction of medicine.17 In recent years, visionary foundations (Grameen, Bill and Melinda Gates Foundation, UNAIDS, Vodafone) have, with funding, created opportunities for innovative thinking in health. To date, 95 countries have evaluated some digital health innovations.11 This is most evident in under-resourced settings where low-cost and sustainable solutions are needed to solve complex global health challenges.18

    Digital innovations were first used in non-communicable diseases and later became popular in infectious disease.19 In the field of HIV/STIs, a Lancet study demonstrated the effectiveness of mHealth-based short message service (SMS) innovations on adherence to antiretroviral therapy (ART).20 As novel digital innovations and strategies continue to be developed and tested, many smaller reviews and systematic reviews were published. However, a vast majority of these reviews only evaluated a single innovation (eg, SMS, social media), one or two outcomes and restricted exploration in select subgroups (people living with HIV (PLHIV), pregnant women, adolescents, men who have sex with men (MSM)).21–27 These reviews failed to provide a comprehensive summary of all innovations for programme planning and research. Due to a rapid expansion of digital innovations, and an increased popularity of combined innovations (2013), a need for a comprehensive up-to-date synthesis on all innovations for HIV/STIs was felt.

    Our primary objective was to generate a high-quality overview/systematic review that summarizes all digital innovations across all populations and outcomes in HIV/STIs. Our secondary objective was to inform researchers, policy makers and funders with evidence for their decisions on implementation and scale-up.11

    Methods

    PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) and Cochrane guidelines were followed.28

    Data sources and searches

    We searched MEDLINE via PubMed, Embase, Cochrane CENTRAL and Web of Science for a 21-year period from February 1996 up to March 2017, with no language restrictions.

    Search strategy

    Keywords used were HIV, AIDS, STI, mhealth, mobile health, ehealth, telemedicine, mobile applications and social media. For a full search strategy, refer to online supplementary appendix file 1. (#1 (’HIV' [MeSH] OR ’acquired immunodeficiency syndrome' [tiab]), OR #2 (sexually transmitted infections [mh] OR sexually transmitted disease* [tiab]), AND #3 (’mHealth' [tiab] OR ’mobile health' [tiab] OR short messag* [tiab] OR ’eHealth' [MeSH] OR ’telemedicine' [MeSH] OR social medi* [tiab] OR ’mobile applications' [tiab]).

    Supplementary file 1

    Study selection

    Two reviewers independently screened and evaluated citations for eligibility (JD and RV) and two others (BL and SD) independently assessed quality. A senior reviewer was consulted (NPP) for discordance.

    Eligibility criteria

    Any clinical trials or observational study designs that evaluated any digital (mHealth/eHealth) technology with any reported outcomes (refer to figure 1) were included.

    Figure 1
    Figure 1

    PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.

    Data abstraction

    Two reviewers (RV, JD) independently abstracted all the data. A prepiloted data abstraction form was used to abstract the following items: study design, study population, sample size, digital innovation type, HIV/STIs, outcome measures (eg, impact, acceptability and feasibility) and metrics (eg, attendance rate, completion rate, satisfaction) (refer to online supplementary appendix file 2). We referred to a previously published framework to define and further classify the following metrics for impact, acceptability and feasibility.29

    Supplementary file 2

    Subgroup pooled analyses

    We classified study designs and then classified digital innovations into three groups30:

    1. mHealth (SMS and phone calls only, that is, non-internet based);

    2. internet-enabled mHealth and other internet-based eHealth (mobile application, website, online campaign, streamed soap opera videos, avatar-guided computer programs);

    3. combined innovations (innovations that combined both mHealth (SMS/phone calls) with internet enabled mHealth/eHealth).

    Only one subgroup reported similar outcomes which could be pooled, SMS and phone calls, for the following outcomes: (1) clinic attendance with SMS and (2) ART adherence via Medication Event Monitoring System (MEMS) caps, with SMS. We pooled these outcomes using a random effects subgroup analysis. Given the diversity in the sample populations between studies, we used the random effect meta-analysis model with the DerSimonian and Laird estimator (moments method) of the between-study variance to calculate the pooled effect. We generated forest plots for visual representation of heterogeneity and pooled OR with 95% CI. We performed all statistical analyses using Stata/IC, V.13 (Stata).31

    Narrative analysis

    We narratively described all other data using as follows:

    Digital innovations were classified into the following groups based on the strength of evidence: high/strong evidence (metrics at 75%–100%), moderate evidence (51%–74%) and low/weak evidence (50% or less).

    Acceptability

    Acceptability was defined as the receptivity in using digital innovations.

    Feasibility

    Feasibility was defined as the perceived convenience in using digital innovations. It was reported with various metrics: completion, retention, response and referral rates.

    Impact

    Impact was defined as a statistically significant improvement in measured outcomes compared with a comparator group (ie, control group or baseline observations). The metrics used to evaluate impact were (1) attendance rate, (2) ART adherence, (3) risk reduction, (4) self-care and (5) partner notification. Impact measures were evaluated on two criteria: effect size and precision. Effect size was assessed when data on a comparator group were made available. Precision of the effect estimate was assessed whenever reported, as it reflects the variance or spread of results.

    Quality assessment

    We assessed study quality for both clinical trials and observational studies. We used the Cochrane Risk of Bias Tool for trials, and the Newcastle-Ottawa quality assessment scale for observational studies.

    Results

    Of 4252 citations identified through our extensive search, 792 were selected for full-text screening, and 99 citations met our inclusion criteria and were included in this review for evidence synthesis (refer figure 1).

    Study characteristics

    By geographical location, 37% (37/99) of studies were conducted in North America, 26% (26/99) in Sub-Saharan Africa, 24% (24/99) in Europe, 7% (7/99) in Oceania, 3% (3/99) in South-East Asia and 2% (2/99) in South America.

    By study design, the majority were trials: 38% (38/99) were RCTs, 16% (16/99) uncontrolled trials and 1% (1/99) non-randomised controlled trials. Others included quasi-experimental studies, of which many used historical controls (24%, 24/99) and observational studies (ie, cross-sectional and feasibility studies) (20%, 20/99).

    HIV was the most frequently reported infection (74%, 73/99 studies), followed by chlamydia/gonorrhoea (CT/GC) (10%, 10/99). Combinations of HIV with STIs (eg, syphilis) (8%, 8/99), human papillomavirus (HPV) (4%, 4/99) and hepatitis A/B/C (HBV) (4%, 4/99) were also reported.

    In terms of study populations, PLHIV were prominent across studies (42%, 42/99) followed by other high-risk groups (ie, MSM/bisexual men, drug users, pregnant women/mother–infant pairs, African-Americans, sex workers and visible minorities) (28%, 28/99), general clinic attendees (16%, 16/99), CT/HBV-infected individuals (4%, 4/99) and residents of a specific area (9%, 9/99).

    Innovations

    Digital innovations were documented across the spectrum.

    mHealth innovations (SMS/phone calls only) were evaluated in 70% (69/99) of studies.20 32–99Seventy-two per cent (50/69) were SMS-based and 28% (19/69) used phone calls or a combination of both (refer to figure 2 and see online supplementary appendix file 3).

    Supplementary file 3

    Figure 2
    Figure 2

    All innovations by outcome type (font size enlarged). ART, antiretroviral therapy; SMS, short message service

    Internet-enabled mHealth and other internet-based eHealth innovations were evaluated in 21% (21/99) of studies.100–120 These innovations consisted of many different forms: social media and online campaigns (9/21), avatar-guided computer programs (2/21), mobile applications (5/21), combination of social media and websites (2/21), websites (1/21), telemedicine services (1/21) and streamed soap opera videos (1/21) (refer to figure 2 and see online supplementary appendix file 3).

    Combined innovations were evaluated in 9% (9/99) of studies.121–129 Innovations consisted of SMS+websites/ interactive websites (4/9), SMS+mobile application (3/9) and SMS+social media (including online campaigns) (2/9) (refer to figure 2 and see online Supplementary appendix file 3).

    Measures and metrics

    A vast majority (84%, 83/99) of studies focused on impact measure and metrics, while about 12% (12/99) focused only on feasibility and the remaining 4% (4/99) on acceptability. Within impact measures, metrics such as clinic attendance rates were reported in 45% (37/83) of studies, followed by ART adherence at 35% (29/83), HIV/STIs risk reduction behaviours at 13% (11/83), turnaround time from testing to treatment at 2% (2/83), partner notification at 2% (2/83) and self-care at 2% (2/83).

    Analyses

    Subgroup pooled analyses

    It was possible to perform subgroup analyses on outcomes that were consistently documented: clinic attendance in 14 quasi-experimental studies that used SMS reminders and MEMS-based ART adherence in four randomised controlled trials (RCTs) evaluating SMS. The pooled estimate for the impact of SMS reminders on attendance rates was 1.76 (95% CI: 1.28 to 2.42) (refer to figure 3A). The pooled estimate for the impact of SMS on ART adherence tracked via MEMS caps was also significant, OR=2.15 (95% CI: 1.18 to 3.91) (refer to figure 3B).32 47 48

    Figure 3
    Figure 3

    Subgroup analyses.

    Narrative analysis

    Impact

    Non-internet-based mHealth (SMS/PC only)

    Of 69 studies, positive results were reported for the following outcomes: clinic attendance (63%, 19/30 studies, of which 84% reached statistical significance), ART adherence (63%, 15/24 studies, of which 93% reached statistical significance), turnaround time from testing to treatment (67%, 2/3 studies). However, SMS reported a limited effect on risk reduction behaviours (3/7, 43%).

    Internet-based mHealth/eHealth

    Studies evaluating internet-based eHealth innovations (21/99) reported results that were largely in favour of the following innovations: social media-based interventions for clinic attendance; avatar-guided and mobile applications for ART adherence; social media, avatar and soap opera videos for risk reduction behaviours; mobile app for self-care.

    Social media contributed to higher testing uptake rates in all studies (6/6, 100%). A social media-based campaign increased HIV testing by 252% (n=1500; 19% from baseline 5.4%, p<0.01) and Syphilis testing by 248% (18.8% from baseline 5·4%, p<0.01), while another campaign increased HIV testing by 52% compared with control (n=625; 63.7% vs 42% in controls, OR=2.9 (95% CI: 1.8 to 4.7)).100 115 Four campaigns guaranteed rapid in-home HIV testing for all those who requested it online.100 101 108 111 116

    Avatar-guided programs and mobile applications improved ART adherence in all studies (4/4). Statistically significant outcomes were reported in 2/4 programs (50%). These were (1) a personalised avatar-guided computer program improved adherence (n=240; p=0.046); (2) a mobile application with immunosuppression graphs and medication reminders lowered viral load (n=28; p=0.023) and improved adherence (p=0.03) as well.102 104 In the other two studies, an avatar-guided program improved viral suppression and a mobile application allowed for 100% adherence, but these were underpowered to detect a significant effect (n=76 and n=28, respectively).107 110

    Social media, avatar and soap opera videos were successful at reducing risky sexual behaviour in all the reported studies (5/5). However, significant results were reported in only three of five studies: (1) social media-based interventions decreased unprotected sex acts by 65% (n=31; 3.11 vs baseline 8.96, p=0.042); (2) soap opera videos on HIV prevention reduced condomless sex by 78% (n=117; 78% reduction from baseline, p<0.001)103 106; (3) an avatar-guided computer program also lowered the odds of HIV transmission (n=240; OR=0.46, p=0.012).102 103 106 Even in two underpowered studies, social media-based interventions led to 40% and 67% higher condom uptake (n=70 and n=50, respectively).105 117

    A mobile application increased self-care in the sole study in this category (1/1). A significantly higher self-care performance among chronic HBV-infected individuals was reported compared with controls (n=53; p=0.001).112

    Combined innovations

    Studies evaluating combined innovations (9/99) showed success of social media+SMS in increasing clinic attendance and partner notification; interactive websites+SMS in improving ART adherence; and mobile app+SMS in increasing self-care. Among the five impact studies, 80% reported a favourable outcome. An online campaign with SMS services increased CT, GC and HIV tests uptake by 41%, 91% and 190%, respectively123; an interactive website with SMS reminders improved ART adherence in drug users (n=20; p=0.02)121; a social media-based partner notification with SMS increased notified contacts by 144% (63.5% in 2011 from baseline 26% in 2010)126; and a mobile app with SMS significantly improved self-care performance in HIV-infected individuals compared with baseline (n=19; p=0.002).129

    Acceptability and feasibility

    Overall, across studies that assessed acceptability/feasibility, digital innovations were found to be highly acceptable and feasible (75%–100%)%) in 26/31 and 20/31 studies, respectively. mHealth innovations (SMS/PC only) were highly acceptable and feasible in 81% (13/16) and 75% (12/16) of studies, respectively.

    Internet-based mHealth/eHealth innovations were highly acceptable and feasible in 92% (11/12) and 45% (5/11) of studies, respectively. All included innovations (ie, avatar, mobile app, social media and streamed videos) were highly acceptable.102–104 106 107 While avatar-guided program was rated high on feasibility, social media-based strategies were found to be less feasible101–103

    Combined innovations were highly acceptable and feasible in 67% (2/3) and 75% (3/4) of studies, respectively.121 124 The innovations that were rated high involved a combination of SMS and interactive websites.

    Quality

    Studies were individually evaluated on quality criteria, and biases were noted. Across trials, losses to follow-up were reported in 31% of RCTs and 55% of quasi-trials. Additionally, biases (ie, misclassification, recall bias) were of concern in 58% of the RCTs and 64% of quasi-randomised trials (refer to online supplementary appendix file 4 and 5).

    In observational studies, confounding (68%) and selection bias (66%) were observed (refer to  online supplementary appendix file 6). Studies with small sample sizes, low power or insufficient follow-up time (eg, 3 weeks or less) sometimes provided contradictory results when objective and subjective metrics evaluated the same outcome.

    Supplementary file 6

    Discussion

    Summary of findings

    Overall, digital innovations reported positive effects on key metrics. We noted a strong positive effect of digital innovations on clinic attendance rates (70%; 26/37), ART adherence (69%; 20/29), risk reduction behaviours (67%; 8/12) and self-care (100%; 2/2). SMS/phone calls were not able to reduce risky sexual behaviours; however, social media-based interventions, particularly interactive social media, were effective in reducing risky sexual behaviours. Acceptability was found to be high for all innovations. Feasibility estimates also remained high for all innovations, except for social media-based interventions, possibly due to a perceived lack of privacy and confidentiality. Combined innovations may thus offer promise in plugging this feasibility gap, with internet-based innovations compensating for limitations in SMS-only strategies and vice versa.

    While mHealth (SMS/phone calls only) innovations were highly effective in improving clinic attendance, ART adherence and turnaround time from testing to treatment, they did not report on other outcomes. It should be noted that SMS and phone calls alone failed to reduce risky sexual behaviours, which was not surprising as it is challenging to reduce risky behaviours with a prescriptive SMS alone. Population engagement is essential for risk reduction through qualitative research.

    While internet-based mHealth/eHealth innovations (social media, avatar-guided computer programs, mobile apps and soap opera videos) demonstrated positive evidence on impact metrics, not all studies reached statistical significance. Those that failed to report a statistically significant improvement in ART adherence had small sample sizes and were underpowered to detect these outcomes (n=76 vs n=240), and had less frequent sessions over a shorter evaluation period (2 sessions over 6 months vs 4 sessions over 9 months).102 107 For mobile applications, studies which reported significant effects recruited participants with varying level of adherence,104 110 compared with studies which had high adherence at baseline (≥95%) and did not show significance (due to smaller changes in effect). For social media-based campaigns, the two studies that did not reach statistical significance in reducing risky sexual behaviours lacked an interactive component and simply displayed educational material, while the study that showed significant effect engaged the participants by allowing them to contact professional cognitive behavioural therapists via live chat sessions.103 105 117

    In terms of quality, confounding and selection bias were noted in observational and quasi-experimental studies, and loss to follow-up in some trials. Nevertheless, the overall validity of the findings from this review was not threatened by biases, as a large proportion of our data were derived from trials. While clinical trials were generally high quality, observational studies were medium to low quality.

    Consistent reporting of metrics was lacking, which prevented a comprehensive meta-analysis. Objectives, end points, metrics and measures are equally important in feasibility studies and must be well designed to generate high-quality evidence.

    Our review is an exhaustive assessment of the role of digital innovations in improving prevention and care for HIV/STIs. Our findings resonate with many smaller systematic reviews, which have separately evaluated individual components of digital innovation, such as SMS-based mHealth.22 23 130–137 Other systematic reviews evaluating social media-based interventions reported similar findings to ours, in improved testing uptake or improvements in sexual health.25–27 138 139

    Our review makes a valuable addition to the growing body of evidence by highlighting the success of other interactive and engaging innovations such as avatar-guided computer programs, mobile apps, streamed soap opera videos and combined innovations. These integrated innovations and programs are gaining in popularity because of their power to engage rural and urban audiences at many levels.

    Designing combined innovations that are complementarity of various media, methods, platforms and messaging may deliver best results. This complementarity may also encourage participant engagement to improve prevention and care metrics and measures sustainably over time. Engagement is challenging when only one innovation (eg, mHealth SMS/phone calls only) is the sole focus, where boredom is likely.

    Caveats and implications for future research

    There are some caveats to considering design and evaluation of innovations. Future research needs to be focused on tailoring innovations to the context and population, and program objectives. Innovations aiming to reduce risky sexual behaviours could be interactive and tailored to the setting and population, with a deep understanding of patients’ needs and preferences.137 140 141 Any communication with patients could be customised for timing to avoid fatigue with its uptake. For example, patients may be more responsive to weekly versus daily SMS ART reminders.32 142

    Study quality is essential to generating meaningful results. Large and representative samples of the underlying population and sound statistical techniques during data analysis or sampling methodology can minimise selection bias. Exploring reasons for differential losses to follow-up could inform future studies. Wherever possible, a control group should be included to differentiate the Hawthorne effect from the effect of the intervention.143 Trials and impact designs can prevent or reduce confounding. Following checklists, like the one by the WHO mHealth Technical Evidence Review Group on mHealth innovations, is suggested and encouraged.144

    Objective measures (eg, HIV/STIs diagnosis, viral load) are desired in reporting of quantitative outcomes, over subjective self-reported data (eg, condom use, self-reported adherence). This could potentially reduce some biases (misclassification biases or desirability/recall biases) that are observed with subjective reporting.

    Qualitative data are rich and complement the understanding of all the contextual and population needs, and capture the dynamics of sustainability and change. They need to be integrated with quantitative data to provide a holistic picture of uptake of any digital innovation.

    Quality of digital data will merit from an improvement. Across studies, a lack of integrated online impact metrics in evaluating the success of innovations was evident. With continuously evolving digital media, inventing new ways to evaluate acceptability and feasibility becomes necessary. For example, some studies tracked online metrics via Google analytics.74 100 101 121–124 Synergy with industry powered metrics could be a new wave to measure success of digital innovations.

    To scale up proven innovations, a multistakeholder engagement is necessary. For that, data and metrics that appeal to all sections of stakeholders will be needed. In addition to improving the quality of randomised controlled trials and quasi-experimental impact studies, qualitative studies, cost-effectiveness studies and usability studies are also needed.

    Implications for policy and practice

    In consonance with other systematic reviews, evidence at scale and over time was scarce.138 This limits the projection of the long-term sustainability and cost-effectiveness of digital innovations. More evidence on scale-up, cost-savings and cost-effectiveness from Sub-Saharan Africa and Asia is needed. Future investments that incentivise both the development and evaluation of combined innovations by government and industry alike, and focus on sustainability of digital innovations with public and private partnerships, are urgently needed.

    Conclusion

    To control HIV/STIs globally, we need novel and disruptive innovations that will uniquely impact health outcomes across the spectrum of access, engagement, treatment and retention so as to impact health service delivery. On one hand, mHealth (SMS/phone calls only) and internet-based mHealth/eHealth were found acceptable, feasible and offered complementarity in improving prevention and care of HIV/STIs. On the other hand, when combined, they provided customised and contextualised solutions for hard-to-reach populations.

    Innovations need to be proven for impact and cost-effectiveness, using a combination of clinical trials, quasi-randomised studies, observational studies and qualitative research studies. Integrating these innovations across various levels of healthcare with clear evaluation, monitoring and documentation of metrics will facilitate their integration within existing health service delivery models so as to efficiently impact health outcomes over time.

    Findings from this comprehensive review will be informative to all stakeholders—innovators, researchers, healthcare practitioners, policy makers and funders—worldwide seeking evidence on integrating and funding innovations, to make (impact) the entire spectrum of HIV/STI care.

    Supplementary file 4

    Supplementary file 5

    Acknowledgments

    The authors would like to acknowledge Ms. Megan Smallwood for her assistance in editing the manuscript.

    References

    1. 1.↵
      The World Health Organization. Sexually transmitted infections (STIs): fact sheet N°110: WHO media centre, 2015. Updated Dec 2015. http://www.who.int/mediacentre/factsheets/fs110/en/. (accessed 27 Jan 2016).
    2. 2.↵
      The World Health Organization. Hepatitis B: fact sheet N°204: WHO media centre. 2015 http://www.who.int/mediacentre/factsheets/fs204/en/ (accessed 27 Jan 2016).
    3. 3.↵
      The World health Organization. Hepatitis C: fact sheet N°164, 2015. http://www.who.int/mediacentre/factsheets/fs164/en/ (accessed 27 Jan 2016).
    4. 4.↵
      The Joint United Nations Programme on HIV/AIDS. Fact Sheet 2015, 2015. http://www.unaids.org/en/resources/campaigns/HowAIDSchangedeverything/factsheet (accessed 17 Dec 2016).
    5. 5.↵
      2. Newman L ,
      3. Rowley J ,
      4. Vander Hoorn S , et al
      . Global estimates of the prevalence and incidence of four curable sexually transmitted infections in 2012 based on systematic review and global reporting. PLoS One 2015;10:e0143304.doi:10.1371/journal.pone.0143304
    6. 6.↵
      The Joint United Nations Programme on HIV/AIDS. The gap report. Geneva: UNAIDS, 2014.
    7. 7.↵
      2. Kagee A ,
      3. Remien RH ,
      4. Berkman A , et al
      . Structural barriers to ART adherence in Southern Africa: Challenges and potential ways forward. Glob Public Health 2011;6:83–97.doi:10.1080/17441691003796387
      OpenUrlCrossRefPubMedWeb of Science
    8. 8.↵
      2. Pai NP
      . Multiplexed point-of-care assays for HIV and co-infections for resource constrained settings: a perspective. Future Microbiol 2015;10:1393–6.doi:10.2217/fmb.15.77
      OpenUrl
    9. 9.↵
      2. Pai NP ,
      3. Vadnais C ,
      4. Denkinger C , et al
      . Point-of-care testing for infectious diseases: diversity, complexity, and barriers in low- and middle-income countries. PLoS Med 2012;9:e1001306.doi:10.1371/journal.pmed.1001306
      OpenUrlCrossRefPubMed
    10. 10.↵
      2. Rosen S ,
      3. Fox MP
      . Retention in HIV care between testing and treatment in sub-Saharan Africa: a systematic review. PLoS Med 2011;8:e1001056.doi:10.1371/journal.pmed.1001056
      OpenUrlCrossRefPubMed
    11. 11.↵
      2. Ryu S
      . Book review: mHealth: new horizons for health through mobile technologies: based on the findings of the second global survey on eHealth (global observatory for eHealth series, volume 3). Healthc Inform Res 2012;18:231–3.doi:10.4258/hir.2012.18.3.231
      OpenUrlCrossRef
    12. 12.↵
      2. Free C ,
      3. Phillips G ,
      4. Watson L , et al
      . The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLoS Med 2013;10:e1001363.doi:10.1371/journal.pmed.1001363
      OpenUrlCrossRefPubMed
    13. 13.↵
      The World Bank. Maximizing mobile: information and communication technologies for development. Washington DC: International Bank for Reconstruction and Development The World Bank, 2012.
    14. 14.↵
      eMarketer. Smartphone user penetration as percentage of total global population from 2011 to 2018: Statistica- The Statistics Portal, 2014. http://www.statista.com/statistics/203734/global-smartphone-penetration-per-capita-since-2005/ (accessed 19 Feb 2016).
    15. 15.↵
      eMarketer. Mobile phone internet user penetration worldwide from 2014 to 2019: Statistica- The Statistics Potal, 2014. http://www.statista.com/statistics/284202/mobile-phone-internet-user-penetration-worldwide/ (accessed 19 Feb 2016).
    16. 16.↵
      2. Oulasvirta A ,
      3. Rattenbury T ,
      4. Ma L , et al
      . Habits make smartphone use more pervasive. Pers Ubiquitous Comput 2012;16:105–14.doi:10.1007/s00779-011-0412-2
      OpenUrl
    17. 17.↵
      2. Topol EJ
      . The creative destruction of medicine: how the digital revolution will create better health care. New York: Basic Books, 2012.
    18. 18.↵
      2. Akter S ,
      3. Ray P
      . mHealth - an ultimate platform to serve the unserved. Yearb Med Inform 2010:94–100 (Epub ahead of print 13 Oct 2010).
    19. 19.↵
      2. Ali EE ,
      3. Chew L ,
      4. Yap KY-L
      . Evolution and current status of mhealth research: a systematic review. BMJ Innov 2016;2:33–40.doi:10.1136/bmjinnov-2015-000096
      OpenUrlAbstract/FREE Full Text
    20. 20.↵
      2. Lester RT ,
      3. Ritvo P ,
      4. Mills EJ , et al
      . Effects of a mobile phone short message service on antiretroviral treatment adherence in Kenya (WelTel Kenya1): a randomised trial. Lancet 2010;376:1838–45.doi:10.1016/S0140-6736(10)61997-6
      OpenUrlCrossRefPubMedWeb of Science
    21. 21.↵
      2. Ambia J ,
      3. Mandala J
      . A systematic review of interventions to improve prevention of mother-to-child HIV transmission service delivery and promote retention. J Int AIDS Soc 2016;19:20309.doi:10.7448/IAS.19.1.20309
      OpenUrlCrossRefPubMed
    22. 22.↵
      2. Conserve DF ,
      3. Jennings L ,
      4. Aguiar C , et al
      . Systematic review of mobile health behavioural interventions to improve uptake of HIV testing for vulnerable and key populations. J Telemed Telecare 2017;23:347–59.doi:10.1177/1357633X16639186
      OpenUrl
    23. 23.↵
      2. L’Engle KL ,
      3. Mangone ER ,
      4. Parcesepe AM , et al
      . Mobile phone interventions for adolescent sexual and reproductive health: a systematic review. Pediatrics 2016;138:e20160884.doi:10.1542/peds.2016-0884
      OpenUrlAbstract/FREE Full Text
    24. 24.↵
      2. Lima IC ,
      3. Galvão MT ,
      4. Alexandre HO , et al
      . Information and communication technologies for adherence to antiretroviral treatment in adults with HIV/AIDS. Int J Med Inform 2016;92:54–61.doi:10.1016/j.ijmedinf.2016.04.013
      OpenUrl
    25. 25.↵
      2. Dunne A ,
      3. McIntosh J ,
      4. Mallory D
      . Adolescents, sexually Transmitted infections, and education using social media: a review of the literature. J Nurse Pract 2014;10:409–16.doi:10.1016/j.nurpra.2014.03.020
      OpenUrl
    26. 26.↵
      2. Jones K ,
      3. Eathington P ,
      4. Baldwin K , et al
      . The impact of health education transmitted via social media or text messaging on adolescent and young adult risky sexual behavior: a systematic review of the literature. Sex Transm Dis 2014;41:413–9.doi:10.1097/OLQ.0000000000000146
      OpenUrlCrossRefPubMed
    27. 27.↵
      2. Schnall R ,
      3. Travers J ,
      4. Rojas M , et al
      . eHealth interventions for HIV prevention in high-risk men who have sex with men: a systematic review. J Med Internet Res 2014;16:228–37.doi:10.2196/jmir.3393
      OpenUrl
    28. 28.↵
      2. Moher D ,
      3. Liberati A ,
      4. Tetzlaff J , et al
      . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009;151:264–9.doi:10.7326/0003-4819-151-4-200908180-00135
      OpenUrlCrossRefPubMedWeb of Science
    29. 29.↵
      2. Chiavegatti T ,
      3. Vijh R ,
      4. Pant Pai N
      . Metrics beyond accuracy for HIV point-of-care tests: a systematic review of evidence. Can J Infect Dis Med Microbiol 2014;25(Suppl A):90.
      OpenUrl
    30. 30.↵
      2. Pai M ,
      3. Filion K
      . Classification of study designs (Version 8) Montreal. McGill University, 2014. http://www.teachepi.org/documents/courses/Classification%20Design.pdf.
    31. 31.↵
      2. Harris R ,
      3. Bradburn M ,
      4. Deeks J , et al
      . Metan: fixed- and random-effects meta-analysis. Stata Journal 2008;8:3–28.
      OpenUrlWeb of Science
    32. 32.↵
      2. Pop-Eleches C ,
      3. Thirumurthy H ,
      4. Habyarimana JP , et al
      . Mobile phone technologies improve adherence to antiretroviral treatment in a resource-limited setting: a randomized controlled trial of text message reminders. AIDS 2011;25:825–34.doi:10.1097/QAD.0b013e32834380c1
      OpenUrlCrossRefPubMedWeb of Science
    33. 33.↵
      2. Dowshen N ,
      3. Kuhns LM ,
      4. Johnson A , et al
      . Improving adherence to antiretroviral therapy for youth living with HIV/AIDS: a pilot study using personalized, interactive, daily text message reminders. J Med Internet Res 2012;14:e51.doi:10.2196/jmir.2015
      OpenUrlCrossRefPubMed
    34. 34.↵
      2. Menon-Johansson AS ,
      3. McNaught F ,
      4. Mandalia S , et al
      . Texting decreases the time to treatment for genital Chlamydia trachomatis infection. Sex Transm Infect 2006;82:49–51.doi:10.1136/sti.2004.014381
      OpenUrlAbstract/FREE Full Text
    35. 35.↵
      2. Finocchario-Kessler S ,
      3. Gautney BJ ,
      4. Khamadi S , et al
      . If you text them, they will come: using the HIV infant tracking system to improve early infant diagnosis quality and retention in Kenya. AIDS 2014;28(Suppl 3):S313–21.doi:10.1097/QAD.0000000000000332
      OpenUrlCrossRefPubMed
    36. 36.↵
      2. Lim EJ ,
      3. Haar J ,
      4. Morgan J
      . Can text messaging results reduce time to treatment of Chlamydia trachomatis? Sex Transm Infect 2008;84:563–4.doi:10.1136/sti.2008.032441
      OpenUrlAbstract/FREE Full Text
    37. 37.↵
      2. Bailey S ,
      3. Scalley B ,
      4. Gilles M
      . Text 2 treat - using SMS to recall clients for treatment. Int J STD AIDS 2014;25:1038–40.doi:10.1177/0956462414530108
      OpenUrlCrossRefPubMed
    38. 38.↵
      2. Smillie K ,
      3. Van Borek N ,
      4. Abaki J , et al
      . A qualitative study investigating the use of a mobile phone short message service designed to improve HIV adherence and retention in care in Canada (WelTel BC1). J Assoc Nurses AIDS Care 2014;25:614–25.doi:10.1016/j.jana.2014.01.008
      OpenUrlCrossRefPubMed
    39. 39.↵
      2. Reback CJ ,
      3. Fletcher JB ,
      4. Shoptaw S , et al
      . Exposure to theory-driven text messages is associated with HIV risk reduction among methamphetamine-using men who have sex with men. AIDS Behav 2015;19(Suppl 2):130–41.doi:10.1007/s10461-014-0985-7
      OpenUrl
    40. 40.↵
      2. Evans C ,
      3. Turner K ,
      4. Blake H , et al
      . Text messaging to encourage uptake of HIV testing amongst African communities: findings from a theory based feasibility study. HIV Medicine 2015;16(Suppl 2):54.
      OpenUrl
    41. 41.↵
      2. Dowshen N ,
      3. Kuhns LM ,
      4. Gray C , et al
      . Feasibility of interactive text message response (ITR) as a novel, real-time measure of adherence to antiretroviral therapy for HIV+ youth. AIDS Behav 2013;17:2237–43.doi:10.1007/s10461-013-0464-6
      OpenUrlCrossRefPubMed
    42. 42.↵
      2. Cornelius JB ,
      3. Dmochowski J ,
      4. Boyer C , et al
      . Text-messaging-enhanced HIV intervention for African American adolescents: a feasibility study. J Assoc Nurses AIDS Care 2013;24:256–67.doi:10.1016/j.jana.2012.06.005
      OpenUrlCrossRefPubMed
    43. 43.↵
      2. Ammassari A ,
      3. Trotta MP ,
      4. Mussini C , et al
      . Mobile phone intervention based on short message system (SMS) improve adherence behavior in HIV-infected non-adherent to antiretroviral therapy. The HI-Vision Study. Infection 2010;38(Suppl 52):52.
      OpenUrl
    44. 44.↵
      2. Ammassari A ,
      3. Trotta MP ,
      4. Shalev N , et al
      . Timed short messaging service improves adherence and virological outcomes in HIV-1-infected patients with suboptimal adherence to antiretroviral therapy. J Acquir Immune Defic Syndr 2011;58:e113–e115.doi:10.1097/QAI.0b013e3182359d2a
      OpenUrlCrossRefPubMed
    45. 45.↵
      2. Segaren N ,
      3. Lewis T ,
      4. Desinor O , et al
      . The distribution and use of cell phones to mothers of HIV-positive infants identified by the Haiti National Early Infant Diagnosis of HIV program (EID): a model for increasing adherence? XIX International AIDS Conference  Washington DC, 2012.
    46. 46.↵
      2. Sheppard Z ,
      3. Walsh J ,
      4. Weston R
      . The TRxCare™ adherence support system: a pilot study of its acceptability to patients on virologically successful HAART. Birmingham: 18th Annual Conference of British HIV Association, 2012.
    47. 47.↵
      2. da Costa TM ,
      3. Barbosa BJ ,
      4. Gomes e Costa DA , et al
      . Results of a randomized controlled trial to assess the effects of a mobile SMS-based intervention on treatment adherence in HIV/AIDS-infected Brazilian women and impressions and satisfaction with respect to incoming messages. Int J Med Inform 2012;81:257–69.doi:10.1016/j.ijmedinf.2011.10.002
      OpenUrlCrossRefPubMed
    48. 48.↵
      2. Sabin LL ,
      3. Bachman DeSilva M ,
      4. Gill CJ , et al
      . Improving adherence to antiretroviral therapy with triggered real-time text message reminders: the China adherence through technology study. J Acquir Immune Defic Syndr 2015;69:551–9.doi:10.1097/QAI.0000000000000651
      OpenUrlCrossRefPubMed
    49. 49.↵
      2. Brook G ,
      3. Farmer T ,
      4. Murphy S , et al
      . Using SMS Text reminders to reduce ‘did not attend’ (DNA) rates in sexual health and HIV appointment clinics. Sex Transm Infect 2013;89(Suppl 1):A1–A428.doi:10.1136/sextrans-2013-051184.0116
      OpenUrlAbstract/FREE Full Text
    50. 50.↵
      2. Kliner M ,
      3. Knight A ,
      4. Mamvura C , et al
      . Using no-cost mobile phone reminders to improve attendance for HIV test results: a pilot study in rural Swaziland. Infect Dis Poverty 2013;2:12.doi:10.1186/2049-9957-2-12
      OpenUrlCrossRefPubMed
    51. 51.↵
      2. Farmer T ,
      3. Brook G ,
      4. McSorley J , et al
      . Using short message service text reminders to reduce ’did not attend' rates in sexual health and HIV appointment clinics. Int J STD AIDS 2014;25:289–93.doi:10.1177/0956462413502325
      OpenUrlCrossRefPubMed
    52. 52.↵
      2. Brook G ,
      3. Burton J ,
      4. McSorley J , et al
      . The Effectiveness of SMS Texts For Reminding Patients at High Risk of STIs and HIV to Return For Testing. Sex Transm Infect 2013;89(Suppl 1):A1–A428.doi:10.1136/sextrans-2013-051184.0430
      OpenUrlAbstract/FREE Full Text
    53. 53.↵
      2. Guy R ,
      3. Wand H ,
      4. Knight V , et al
      . SMS reminders improve re-screening in women and heterosexual men with chlamydia infection at Sydney Sexual Health Centre: a before-and-after study. Sex Transm Infect 2013;89:11–15.doi:10.1136/sextrans-2011-050370
      OpenUrlAbstract/FREE Full Text
    54. 54.↵
      2. Burton J ,
      3. Brook G ,
      4. McSorley J , et al
      . The utility of short message service (SMS) texts to remind patients at higher risk of STIs and HIV to reattend for testing: a controlled before and after study. Sex Transm Infect 2014;90:11–3.doi:10.1136/sextrans-2013-051228
      OpenUrlAbstract/FREE Full Text
    55. 55.↵
      2. Desai M ,
      3. Burns F ,
      4. Mercey D , et al
      . Active recall of men who have sex with men (MSM) for an HIV/STI testing: a feasible and effective strategy? HIV Medicine 2014;15(Suppl 3):17–159.
      OpenUrlCrossRef
    56. 56.↵
      2. Kharbanda EO ,
      3. Stockwell MS ,
      4. Fox HW , et al
      . Text message reminders to promote human papillomavirus vaccination. Vaccine 2011;29:2537–41.doi:10.1016/j.vaccine.2011.01.065
      OpenUrlCrossRefPubMedWeb of Science
    57. 57.↵
      2. Vilella A ,
      3. Bayas JM ,
      4. Diaz MT , et al
      . The role of mobile phones in improving vaccination rates in travelers. Prev Med 2004;38:503–9.doi:10.1016/j.ypmed.2003.12.005
      OpenUrlCrossRefPubMedWeb of Science
    58. 58.↵
      2. Downing SG ,
      3. Cashman C ,
      4. McNamee H , et al
      . Increasing chlamydia test of re-infection rates using SMS reminders and incentives. Sex Transm Infect 2013;89:16–19.doi:10.1136/sextrans-2011-050454
      OpenUrlAbstract/FREE Full Text
    59. 59.↵
      2. Odeny TA ,
      3. Bailey RC ,
      4. Bukusi EA , et al
      . Text messaging to improve attendance at post-operative clinic visits after adult male circumcision for HIV prevention: a randomized controlled trial. PLoS One 2012;7:e43832.doi:10.1371/journal.pone.0043832
    60. 60.↵
      2. de Tolly K ,
      3. Skinner D ,
      4. Nembaware V , et al
      . Investigation into the use of short message services to expand uptake of human immunodeficiency virus testing, and whether content and dosage have impact. Telemed J E Health 2012;18:18–23.doi:10.1089/tmj.2011.0058
      OpenUrlCrossRefPubMed
    61. 61.↵
      2. Bigna JJ ,
      3. Noubiap JJ ,
      4. Kouanfack C , et al
      . Effect of mobile phone reminders on follow-up medical care of children exposed to or infected with HIV in Cameroon (MORE CARE): a multicentre, single-blind, factorial, randomised controlled trial. Lancet Infect Dis 2014;14:600–8.doi:10.1016/S1473-3099(14)70741-8
      OpenUrlCrossRefPubMed
    62. 62.↵
      2. Rand CM ,
      3. Brill H ,
      4. Albertin C , et al
      . Effectiveness of centralized text message reminders on human papillomavirus immunization coverage for publicly insured adolescents. J Adolesc Health 2015;56:S17–S20.doi:10.1016/j.jadohealth.2014.10.273
      OpenUrlPubMed
    63. 63.↵
      2. Matheson EC ,
      3. Derouin A ,
      4. Gagliano M , et al
      . Increasing HPV vaccination series completion rates via text message reminders. J Pediatr Health Care 2014;28:e35–9.doi:10.1016/j.pedhc.2013.09.001
      OpenUrlCrossRefPubMed
    64. 64.↵
      2. Schwartz SR ,
      3. Clouse K ,
      4. Yende N , et al
      . Acceptability and feasibility of a mobile phone-based case management intervention to retain mothers and infants from an option B+ program in postpartum HIV care. Matern Child Health J 2015;19:2029–37.doi:10.1007/s10995-015-1715-0
      OpenUrlCrossRefPubMed
    65. 65.↵
      2. Belzer ME ,
      3. Naar-King S ,
      4. Olson J , et al
      . The use of cell phone support for non-adherent HIV-infected youth and young adults: an initial randomized and controlled intervention trial. AIDS Behav 2014;18:686–96.doi:10.1007/s10461-013-0661-3
      OpenUrlCrossRefPubMed
    66. 66.↵
      2. Maduka O ,
      3. Tobin-West CI
      . Adherence counseling and reminder text messages improve uptake of antiretroviral therapy in a tertiary hospital in Nigeria. Niger J Clin Pract 2013;16:302–8.doi:10.4103/1119-3077.113451
      OpenUrlCrossRefPubMed
    67. 67.↵
      2. Kebaya L ,
      3. Nduati R ,
      4. Wamalwa D , et al
      . Efficacy of mobile phone use on adherence to nevirapine prophylaxis and retention In care among the Hiv-exposed infants In pmtct: a randomised controlled trial. Arch Dis Child 2014;99(Suppl 2):A1–620.doi:10.1136/archdischild-2014-307384.911
      OpenUrlAbstract/FREE Full Text
    68. 68.↵
      2. Hardy H ,
      3. Kumar V ,
      4. Doros G , et al
      . Randomized controlled trial of a personalized cellular phone reminder system to enhance adherence to antiretroviral therapy. AIDS Patient Care STDS 2011;25:153–61.doi:10.1089/apc.2010.0006
      OpenUrlPubMedWeb of Science
    69. 69.↵
      2. DiClemente RJ ,
      3. Wingood GM ,
      4. Sales JM , et al
      . Efficacy of a telephone-delivered sexually transmitted infection/human immunodeficiency virus prevention maintenance intervention for adolescents: a randomized clinical trial. JAMA Pediatr 2014;168:938–46.doi:10.1001/jamapediatrics.2014.1436
      OpenUrl
    70. 70.↵
      2. Cornelius JB ,
      3. Cato M ,
      4. Lawrence JS , et al
      . Development and pretesting multimedia HIV-prevention text messages for mobile cell phone delivery. J Assoc Nurses AIDS Care 2011;22:407–13.doi:10.1016/j.jana.2010.11.007
      OpenUrlCrossRefPubMedWeb of Science
    71. 71.↵
      2. Dean AL ,
      3. Makin JD ,
      4. Kydd AS , et al
      . A pilot study using interactive SMS support groups to prevent mother-to-child HIV transmission in South Africa. J Telemed Telecare 2012;18:399–403.doi:10.1258/jtt.2012.120118
      OpenUrlCrossRefPubMed
    72. 72.↵
      2. Juzang I ,
      3. Fortune T ,
      4. Black S , et al
      . A pilot programme using mobile phones for HIV prevention. J Telemed Telecare 2011;17:150–3.doi:10.1258/jtt.2010.091107
      OpenUrlCrossRefPubMedWeb of Science
    73. 73.↵
      2. Kalichman SC ,
      3. Kalichman MO ,
      4. Cherry C , et al
      . Brief behavioral self-regulation counseling for HIV treatment adherence delivered by cell phone: an initial test of concept trial. AIDS Patient Care STDS 2011;25:303–10.doi:10.1089/apc.2010.0367
      OpenUrlPubMed
    74. 74.↵
      2. Mbuagbaw L ,
      3. Thabane L ,
      4. Ongolo-Zogo P , et al
      . The Cameroon Mobile Phone SMS (CAMPS) trial: a randomized trial of text messaging versus usual care for adherence to antiretroviral therapy. PLoS One 2012;7:e46909.doi:10.1371/journal.pone.0046909
    75. 75.↵
      2. Moore DJ ,
      3. Poquette A ,
      4. Casaletto KB , et al
      . Individualized texting for adherence building (iTAB): improving antiretroviral dose timing among HIV-infected persons with co-occurring bipolar disorder. AIDS Behav 2015;19:459–71.doi:10.1007/s10461-014-0971-0
      OpenUrlCrossRefPubMed
    76. 76.↵
      2. Norton BL ,
      3. Person AK ,
      4. Castillo C , et al
      . Barriers to using text message appointment reminders in an HIV clinic. Telemed J E Health 2014;20:86–9.doi:10.1089/tmj.2012.0275
      OpenUrl
    77. 77.↵
      2. Odeny TA ,
      3. Bailey RC ,
      4. Bukusi EA , et al
      . Effect of text messaging to deter early resumption of sexual activity after male circumcision for HIV prevention: a randomized controlled trial. J Acquir Immune Defic Syndr 2014;65:e50–7.doi:10.1097/QAI.0b013e3182a0a050
      OpenUrlCrossRefPubMed
    78. 78.↵
      2. Bauermeister J ,
      3. Giguere R ,
      4. Leu CS , et al
      . Interactive voice response system: data considerations and lessons learned during a rectal microbicide placebo adherence trial for young men who have sex with men. J Med Internet Res 2017;19:S57–8.doi:10.2196/jmir.7682
      OpenUrl
    79. 79.↵
      2. Belzer ME ,
      3. Kolmodin MacDonell K ,
      4. Clark LF , et al
      . Acceptability and Feasibility of a Cell Phone Support Intervention for Youth Living with HIV with Nonadherence to Antiretroviral Therapy. AIDS Patient Care STDS 2015;29:338–45.doi:10.1089/apc.2014.0282
      OpenUrl
    80. 80.↵
      2. Shet A ,
      3. De Costa A ,
      4. Kumarasamy N , et al
      . Effect of mobile telephone reminders on treatment outcome in HIV: evidence from a randomised controlled trial in India. BMJ 2014;349:g5978.doi:10.1136/bmj.g5978
      OpenUrlPubMed
    81. 81.↵
      2. Reid MJ ,
      3. Dhar SI ,
      4. Cary M , et al
      . Opinions and attitudes of participants in a randomized controlled trial examining the efficacy of SMS reminders to enhance antiretroviral adherence: a cross-sectional survey. J Acquir Immune Defic Syndr 2014;65:e86–8.doi:10.1097/QAI.0b013e3182a9c72b
      OpenUrl
    82. 82.↵
      2. Roth AM ,
      3. Hensel DJ ,
      4. Fortenberry JD , et al
      . Feasibility and acceptability of cell phone diaries to measure HIV risk behavior among female sex workers. AIDS Behav 2014;18:2314–24.doi:10.1007/s10461-014-0718-y
      OpenUrl
    83. 83.↵
      2. Barnabas RV ,
      3. van Rooyen H ,
      4. Tumwesigye E , et al
      . Uptake of antiretroviral therapy and male circumcision after community-based HIV testing and strategies for linkage to care versus standard clinic referral: a multisite, open-label, randomised controlled trial in South Africa and Uganda. Lancet HIV 2016;3:e212–20.doi:10.1016/S2352-3018(16)00020-5
      OpenUrlCrossRefPubMed
    84. 84.↵
      2. Bassett IV ,
      3. Coleman SM ,
      4. Giddy J , et al
      . Sizanani: a randomized trial of health system navigators to improve linkage to HIV and TB care in South Africa. J Acquir Immune Defic Syndr 2016;73:154–60.doi:10.1097/QAI.0000000000001025
      OpenUrl
    85. 85.↵
      2. Cantudo-Cuenca MR ,
      3. Manzano-Garcia M ,
      4. Jimenez-Galan R , et al
      . Effect of mobile phone short messages on antiretroviral treatment adherence among HIV infected patients. Int J Clin Pharm 2016;38:517.
      OpenUrl
    86. 86.↵
      2. Haberer JE ,
      3. Musiimenta A ,
      4. Atukunda EC , et al
      . Short message service (SMS) reminders and real-time adherence monitoring improve antiretroviral therapy adherence in rural Uganda. AIDS 2016;30:1295–300.doi:10.1097/QAD.0000000000001021
      OpenUrlCrossRefPubMed
    87. 87.↵
      2. Jeffries C ,
      3. Ross P ,
      4. Matoff-Stepp S , et al
      . Ucare4life: mobile texting to improve HIV care continuum outcomes for minority youth. Topics in Antiviral Medicine 2016;24:427.
      OpenUrl
    88. 88.↵
      2. Joseph Davey D ,
      3. Nhavoto JA ,
      4. Augusto O , et al
      . SMSaúde: Evaluating Mobile Phone Text Reminders to Improve Retention in HIV Care for Patients on Antiretroviral Therapy in Mozambique. J Acquir Immune Defic Syndr 2016;73:e23–30.doi:10.1097/QAI.0000000000001115
      OpenUrl
    89. 89.↵
      2. Kampman C ,
      3. Koedijk F ,
      4. Driessen-Hulshof H , et al
      . Retesting young STI clinic visitors with urogenital Chlamydia trachomatis infection in the Netherlands; response to a text message reminder and reinfection rates: a prospective study with historical controls. Sex Transm Infect 2016;92:124–9.doi:10.1136/sextrans-2015-052115
      OpenUrlAbstract/FREE Full Text
    90. 90.↵
      2. Kassaye SG ,
      3. Ong’ech J ,
      4. Sirengo M , et al
      . Cluster-randomized controlled study of SMS text messages for prevention of mother-to-child transmission of HIV in rural kenya. AIDS Res Treat 2016;2016:1289328–8.doi:10.1155/2016/1289328
      OpenUrlCrossRefPubMed
    91. 91.↵
      2. McIver R ,
      3. Dyda A ,
      4. McNulty AM , et al
      . Text message reminders do not improve hepatitis B vaccination rates in an Australian sexual health setting. J Am Med Inform Assoc 2016;23:e88–e92.doi:10.1093/jamia/ocv145
      OpenUrlCrossRefPubMed
    92. 92.↵
      2. Njuguna N ,
      3. Ngure K ,
      4. Mugo N , et al
      . The effect of human immunodeficiency Virus prevention and reproductive health text messages on human immunodeficiency virus testing among young women in rural Kenya: a pilot study. Sex Transm Dis 2016;43:353–9.doi:10.1097/OLQ.0000000000000450
      OpenUrl
    93. 93.↵
      2. Nsagha DS ,
      3. Lange I ,
      4. Fon PN , et al
      . A randomized controlled trial on the usefulness of mobile text phone messages to improve the quality of care of HIV and AIDS patients in Cameroon. Open AIDS J 2016;10:93–103.doi:10.2174/1874613601610010093
      OpenUrl
    94. 94.↵
      2. Nyatsanza F ,
      3. McSorley J ,
      4. Murphy S , et al
      . ’It’s all in the message': the utility of personalised short message service (SMS) texts to remind patients at higher risk of STIs and HIV to reattend for testing-a repeat before and after study. Sex Transm Infect 2016;92:393–5.doi:10.1136/sextrans-2015-052216
      OpenUrlAbstract/FREE Full Text
    95. 95.↵
      2. Rana AI ,
      3. van den Berg JJ ,
      4. Lamy E , et al
      . Using a mobile health intervention to support HIV treatment adherence and retention among patients at risk for disengaging with care. AIDS Patient Care STDS 2016;30:178–84.doi:10.1089/apc.2016.0025
      OpenUrlPubMed
    96. 96.↵
      2. Rand CM ,
      3. Vincelli P ,
      4. Goldstein NP , et al
      . Effects of Phone and Text Message Reminders on Completion of the Human Papillomavirus Vaccine Series. J Adolesc Health 2017;60:113–9.doi:10.1016/j.jadohealth.2016.09.011
      OpenUrlPubMed
    97. 97.↵
      2. Coleman J ,
      3. Bohlin KC ,
      4. Thorson A , et al
      . Effectiveness of an SMS-based maternal mHealth intervention to improve clinical outcomes of HIV-positive pregnant women. AIDS Care 2017;29:1–8.doi:10.1080/09540121.2017.1280126
      OpenUrlPubMed
    98. 98.↵
      2. Garofalo R ,
      3. Kuhns LM ,
      4. Hotton A , et al
      . A randomized controlled trial of personalized text message reminders to promote medication adherence among HIV-positive adolescents and young adults. AIDS Behav 2016;20:1049–59.doi:10.1007/s10461-015-1192-x
      OpenUrlPubMed
    99. 99.↵
      2. Georgette N ,
      3. Siedner MJ ,
      4. Zanoni B , et al
      . The acceptability and perceived usefulness of a weekly clinical SMS program to promote HIV antiretroviral medication adherence in KwaZulu-Natal, South Africa. AIDS Behav 2016;20:2629–38.doi:10.1007/s10461-016-1287-z
      OpenUrlPubMed
    100. 100.↵
      2. Dowshen N ,
      3. Lee S ,
      4. Matty Lehman B , et al
      . IknowUshould2: feasibility of a youth-driven social media campaign to promote STI and HIV testing among adolescents in Philadelphia. AIDS Behav 2015;19(Suppl 2):106–11.doi:10.1007/s10461-014-0991-9
      OpenUrlPubMed
    101. 101.↵
      2. Jones M ,
      3. Aderogba K ,
      4. Drunis L
      . Providing rapid HIV testing in their homes for men who have sex with men, recruited via social media. HIV Medicine 2015;16(Suppl 2):50–1.
      OpenUrl
    102. 102.↵
      2. Kurth AE ,
      3. Spielberg F ,
      4. Cleland CM , et al
      . Computerized counseling reduces HIV-1 viral load and sexual transmission risk: findings from a randomized controlled trial. J Acquir Immune Defic Syndr 2014;65:611–20.doi:10.1097/QAI.0000000000000100
      OpenUrl
    103. 103.↵
      2. Lelutiu-Weinberger C ,
      3. Pachankis JE ,
      4. Gamarel KE , et al
      . Feasibility, acceptability, and preliminary efficacy of a live-chat social media intervention to reduce HIV risk among young men who have sex with men. AIDS Behav 2015;19:1214–27.doi:10.1007/s10461-014-0911-z
      OpenUrlPubMed
    104. 104.↵
      2. Perera AI ,
      3. Thomas MG ,
      4. Moore JO , et al
      . Effect of a smartphone application incorporating personalized health-related imagery on adherence to antiretroviral therapy: a randomized clinical trial. AIDS Patient Care STDS 2014;28:579–86.doi:10.1089/apc.2014.0156
      OpenUrlCrossRefPubMed
    105. 105.↵
      2. Jones K ,
      3. Baldwin KA ,
      4. Lewis PR
      . The potential influence of a social media intervention on risky sexual behavior and Chlamydia incidence. J Community Health Nurs 2012;29:106–20.doi:10.1080/07370016.2012.670579
      OpenUrlCrossRefPubMedWeb of Science
    106. 106.↵
      2. Jones R ,
      3. Hoover DR ,
      4. Lacroix LJ
      . A randomized controlled trial of soap opera videos streamed to smartphones to reduce risk of sexually transmitted human immunodeficiency virus (HIV) in young urban African American women. Nurs Outlook 2013;61:205–15.doi:10.1016/j.outlook.2013.03.006
      OpenUrlCrossRefPubMed
    107. 107.↵
      2. Naar-King S ,
      3. Outlaw AY ,
      4. Sarr M , et al
      . Motivational enhancement system for adherence (MESA): pilot randomized trial of a brief computer-delivered prevention intervention for youth initiating antiretroviral treatment. J Pediatr Psychol 2013;38:638–48.doi:10.1093/jpepsy/jss132
      OpenUrlCrossRefPubMedWeb of Science
    108. 108.↵
      2. Elliot E ,
      3. Rossi M ,
      4. McCormack S , et al
      . Identifying undiagnosed HIV in men who have sex with men (MSM) by offering HIV home sampling via online gay social media: a service evaluation. Sex Transm Infect 2016;92:470–3.doi:10.1136/sextrans-2015-052090
      OpenUrlAbstract/FREE Full Text
    109. 109.↵
      2. Garett R ,
      3. Menacho L ,
      4. Young SD
      . Ethical issues in using social media to deliver an HIV prevention intervention: results from the HOPE Peru Study. Prev Sci 2017;18:225–32.doi:10.1007/s11121-016-0739-z
      OpenUrl
    110. 110.↵
      2. Himelhoch S ,
      3. Kreyenbuhl J ,
      4. Palmer-Bacon J , et al
      . Pilot feasibility study of Heart2HAART: a smartphone application to assist with adherence among substance users living with HIV. AIDS Care 2017;29:898–904.doi:10.1080/09540121.2016.1259454
      OpenUrl
    111. 111.↵
      2. Huang E ,
      3. Marlin RW ,
      4. Young SD , et al
      . Using grindr, a smartphone social-networking application, to increase HIV self-testing among black and latino men Wwho have sex with men in Los Angeles, 2014. AIDS Educ Prev 2016;28:341–50.doi:10.1521/aeap.2016.28.4.341
      OpenUrl
    112. 112.↵
      2. Jeon JH
      . Evaluation of a smartphone application for self-care performance of patients with chronic hepatitis B: a randomized controlled trial. Appl Nurs Res 2016;32:182–9.doi:10.1016/j.apnr.2016.07.011
      OpenUrl
    113. 113.↵
      2. Polilli E ,
      3. Sozio F ,
      4. Di Stefano P , et al
      . Web-based HIV testing in Abruzzo, Italy: analysis of 15-Month activity Results. AIDS Patient Care STDS 2016;30:471–5.doi:10.1089/apc.2016.0082
      OpenUrlCrossRef
    114. 114.↵
      2. Przybyla SM ,
      3. Eliseo-Arras RK ,
      4. Krawiec G , et al
      . Feasibility and acceptability of a smartphone app for daily reports of substance use and antiretroviral therapy adherence among HIV-infected adults. AIDS Res Treat 2016;2016:1–12.doi:10.1155/2016/9510172
      OpenUrlPubMed
    115. 115.↵
      2. Rhodes SD ,
      3. McCoy TP ,
      4. Tanner AE , et al
      . Using social media to increase HIV testing among gay and bisexual men, other men who have sex with men, and transgender persons: outcomes from a randomizedcommunity trial. Clin Infect Dis 2016;62:1450–3.doi:10.1093/cid/ciw127
      OpenUrlCrossRefPubMed
    116. 116.↵
      2. Rosengren AL ,
      3. Huang E ,
      4. Daniels J , et al
      . Feasibility of using GrindrTM to distribute HIV self-test kits to men who have sex with men in Los Angeles, California. Sex Health 2016;13:389.doi:10.1071/SH15236
      OpenUrlCrossRef
    117. 117.↵
      2. Solorio R ,
      3. Norton-Shelpuk P ,
      4. Forehand M , et al
      . Tu Amigo Pepe: Evaluation of a Multi-media Marketing Campaign that Targets Young Latino Immigrant MSM with HIV Testing Messages. AIDS Behav 2016;20:1973–88.doi:10.1007/s10461-015-1277-6
      OpenUrl
    118. 118.↵
      2. Talal AH ,
      3. Andrews P ,
      4. McLeod A , et al
      . Telemedicine-based Hepatitis C Virus (HCV) management for individuals on opioid agonist treatment (OAT). Hepatology 2016;63(Suppl 1):475A.
      OpenUrl
    119. 119.↵
      2. Henwood R ,
      3. Patten G ,
      4. Barnett W , et al
      . Acceptability and use of a virtual support group for HIV-positive youth in Khayelitsha, Cape Town using the MXit social networking platform. AIDS Care 2016;28:898–903.doi:10.1080/09540121.2016.1173638
      OpenUrlCrossRefPubMed
    120. 120.↵
      2. Brayboy LM ,
      3. Sepolen A ,
      4. Mezoian T , et al
      . Girl Talk: a smartphone application to Tteach sexual health education to adolescent girls. J Pediatr Adolesc Gynecol 2017;30:23–8.doi:10.1016/j.jpag.2016.06.011
      OpenUrl
    121. 121.↵
      2. Horvath KJ ,
      3. Oakes JM ,
      4. Rosser BR , et al
      . Feasibility, acceptability and preliminary efficacy of an online peer-to-peer social support ART adherence intervention. AIDS Behav 2013;17:2031–44.doi:10.1007/s10461-013-0469-1
      OpenUrlCrossRefPubMed
    122. 122.↵
      2. Götz HM ,
      3. van Rooijen MS ,
      4. Vriens P , et al
      . Initial evaluation of use of an online partner notification tool for STI, called ’suggest a test': a cross sectional pilot study. Sex Transm Infect 2014;90:195–200.doi:10.1136/sextrans-2013-051254
      OpenUrlAbstract/FREE Full Text
    123. 123.↵
      2. Friedman AL ,
      3. Brookmeyer KA ,
      4. Kachur RE , et al
      . An assessment of the GYT: Get Yourself Tested campaign: an integrated approach to sexually transmitted disease prevention communication. Sex Transm Dis 2014;41:151–7.doi:10.1097/OLQ.0000000000000100
      OpenUrlCrossRefPubMed
    124. 124.↵
      2. Hightow-Weidman LB ,
      3. Muessig KE ,
      4. Pike EC , et al
      . healthMpowerment.org: building community through a mobile-optimized, online health promotion intervention. Health Educ Behav 2015;42:493–9.doi:10.1177/1090198114562043
      OpenUrlCrossRefPubMed
    125. 125.↵
      2. Dokkum NF ,
      3. Koekenbier RH ,
      4. van den Broek IV , et al
      . Keeping participants on board: increasing uptake by automated respondent reminders in an Internet-based chlamydia screening in the Netherlands. MC Public Health 2012;12:176.doi:10.1186/1471-2458-12-176
      OpenUrl
    126. 126.↵
      2. Hightow-Weidman L ,
      3. Beagle S ,
      4. Pike E , et al
      . "No one’s at home and they won’t pick up the phone": using the Internet and text messaging to enhance partner services in North Carolina. Sex Transm Dis 2014;41:143-8.doi:10.1097/OLQ.0000000000000087
      OpenUrl
    127. 127.↵
      2. Aronson ID ,
      3. Cleland CM ,
      4. Perlman DC , et al
      . Mobile screening to identify and follow-up with high risk, hiv negative youth. J Mob Technol Med 2016;5:9–18.doi:10.7309/jmtm.5.1.3
      OpenUrlPubMed
    128. 128.↵
      2. Hirsch-Moverman Y ,
      3. Daftary A ,
      4. Yuengling KA , et al
      . Using mHealth for HIV/TB Treatment Support in Lesotho: Enhancing Patient-Provider Communication in the START Study. J Acquir Immune Defic Syndr 2017;74(Suppl 1):S37–S43.doi:10.1097/QAI.0000000000001202
      OpenUrlPubMed
    129. 129.↵
      2. John ME ,
      3. Samson-Akpan PE ,
      4. Etowa JB , et al
      . Enhancing self-care, adjustment and engagement through mobile phones in youth with HIV. Int Nurs Rev 2016;63:555–61.doi:10.1111/inr.12313
      OpenUrl
    130. 130.↵
      2. Guy R ,
      3. Hocking J ,
      4. Wand H , et al
      . How effective are short message service reminders at increasing clinic attendance? A meta-analysis and systematic review. Health Serv Res 2012;47:614–32.doi:10.1111/j.1475-6773.2011.01342.x
      OpenUrlCrossRefPubMedWeb of Science
    131. 131.↵
      2. Free C ,
      3. Phillips G ,
      4. Galli L , et al
      . The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS Med 2013;10:e1001362.doi:10.1371/journal.pmed.1001362
      OpenUrlCrossRefPubMed
    132. 132.↵
      2. Ershad Sarabi R ,
      3. Sadoughi F ,
      4. Jamshidi Orak R , et al
      . The Effectiveness of Mobile Phone Text Messaging in Improving Medication Adherence for Patients with Chronic Diseases: a Systematic Review. Iran Red Crescent Med J 2016;18:e25183.doi:10.5812/ircmj.25183
      OpenUrlPubMed
    133. 133.↵
      2. Finitsis DJ ,
      3. Pellowski JA ,
      4. Huedo-Medina TB , et al
      . Visual analogue scale (VAS) measurement of antiretroviral adherence in people living with HIV (PLWH): a meta-analysis. J Behav Med 2016;39:1043–55.doi:10.1007/s10865-016-9770-6
      OpenUrlCrossRef
    134. 134.↵
      2. Kanters S ,
      3. Park JJ ,
      4. Chan K , et al
      . Interventions to improve adherence to antiretroviral therapy: a systematic review and network meta-analysis. Lancet HIV 2017;4:e31–e40.doi:10.1016/S2352-3018(16)30206-5
      OpenUrlCrossRefPubMed
    135. 135.↵
      2. Mayer JE ,
      3. Fontelo P
      . Meta-analysis on the effect of text message reminders for HIV-related compliance. AIDS Care 2017;29:1–9.doi:10.1080/09540121.2016.1214674
      OpenUrlPubMed
    136. 136.↵
      2. Thakkar J ,
      3. Kurup R ,
      4. Laba TL , et al
      . Mobile telephone text messaging for medication adherence in chronic disease: A meta-analysis. JAMA Intern Med 2016;176:340–9.doi:10.1001/jamainternmed.2015.7667
      OpenUrlCrossRefPubMed
    137. 137.↵
      2. Mbuagbaw L ,
      3. Mursleen S ,
      4. Lytvyn L , et al
      . Mobile phone text messaging interventions for HIV and other chronic diseases: an overview of systematic reviews and framework for evidence transfer. BMC Health Serv Res 2015;15:33.doi:10.1186/s12913-014-0654-6
      OpenUrlCrossRefPubMed
    138. 138.↵
      2. Muessig KE ,
      3. Nekkanti M ,
      4. Bauermeister J , et al
      . A systematic review of recent smartphone, Internet and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep 2015;12:173–90.doi:10.1007/s11904-014-0239-3
      OpenUrlCrossRefPubMed
    139. 139.↵
      2. Gabarron E ,
      3. Wynn R
      . Use of social media for sexual health promotion: a scoping review. Glob Health Action 2016;9:32193.doi:10.3402/gha.v9.32193
      OpenUrlPubMed
    140. 140.↵
      2. Chávez NR ,
      3. Shearer LS ,
      4. Rosenthal SL
      . Use of digital media technology for primary prevention of STIs/HIV in youth. J Pediatr Adolesc Gynecol 2014;27:244–57.doi:10.1016/j.jpag.2013.07.008
      OpenUrlCrossRefPubMed
    141. 141.↵
      2. Barello S ,
      3. Triberti S ,
      4. Graffigna G , et al
      . eHealth for Patient Engagement: A Systematic Review. Front Psychol 2013;2015:6.
      OpenUrl
    142. 142.↵
      2. Finitsis DJ ,
      3. Pellowski JA ,
      4. Johnson BT
      . Text message intervention designs to promote adherence to antiretroviral therapy (ART): a meta-analysis of randomized controlled trials. PLoS One 2014;9:e88166.doi:10.1371/journal.pone.0088166
    143. 143.↵
      2. Fletcher WR ,
      3. Fletcher WS
      . Chapter 8: Treatment. Clinical Epidemiology: The Essentials. 141. 4th ed. Baltimore: Lippincott Williams, 2005.
    144. 144.↵
      2. Agarwal S ,
      3. LeFevre AE ,
      4. Lee J , et al
      . Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016;352:i1174.doi:10.1136/bmj.i1174
      OpenUrlFREE Full Text

    Footnotes

    • Contributors NPP, JD: concept, design. NPP: data critiquing, write-up, critique, and overall responsibility of the data. JD: data synthesis, write-up, critiquing. RV, BL and SD: data synthesis, write"up and critique. JK, TP and KD: write"up and critique.

    • Funding Grand Challenges Canada Transition to Scale; Grant number 0710"05.FRSQ Salary Award Chercheur"Boursier Junior 2.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement No additional data are available. This is a systematic review/syntheses of existing studies, therefore all data are reported in the tables.