http://orcid.org/0000-0001-7608-9038
Figures
Abstract
Background
With declines in development assistance for health and growing interest in country ownership, donors are increasingly faced with the task of transitioning health programs to local actors towards a path to sustainability. Yet there is little available guidance on how to measure and evaluate the success of a transition and its subsequent effects. This study assesses the transition of the Avahan HIV/AIDS prevention program in India to investigate how preparations for transition affected continuation of program activities post-transition.
Methods
Two rounds of two surveys were conducted and supplemented by data from government and Avahan Computerized Management Information Systems (CMIS). Exploratory factor analysis was used to develop two measures: 1) transition readiness pre-transition, and 2) institutionalization (i.e. integration of initial program systems into organizational procedures and behaviors) post-transition. A fixed effects model was built to examine changes in key program delivery outcomes over time. An ordinary least square regression was used to assess the relationship between transition readiness and sustainability of service outcomes both directly, and indirectly through institutionalization.
Results
Transition readiness data revealed 3 factors (capacity, alignment and communication), on a 15-item scale with adequate internal consistency (alpha 0.73). Institutionalization was modeled as a unidimensional construct, and a 12-item scale demonstrated moderate internal consistency (alpha 0.60). Coverage of key populations and condom distribution were sustained compared to pre-transition levels (p<0.01). Transition readiness, but not institutionalization, predicted sustained outcomes post-transition. Transition readiness did not necessarily lead to institutionalization of key program elements one year after transition.
Conclusion
Greater preparedness prior to transition is important to achieve better service delivery outcomes post-transition. This paper illustrates a methodology to measure transition readiness pre-transition to identify less ready organizations or program components in advance, improving the likelihood of service sustainability. Further research is needed around the conceptualization and development of measures of institutionalization and its effects on long-term program sustainability.
Citation: Ozawa S, Singh S, Singh K, Chhabra V, Bennett S (2016) The Avahan Transition: Effects of Transition Readiness on Program Institutionalization and Sustained Outcomes. PLoS ONE 11(7): e0158659. https://doi.org/10.1371/journal.pone.0158659
Editor: Shira M. Goldenberg, Simon Fraser University, CANADA
Received: June 18, 2015; Accepted: June 20, 2016; Published: July 19, 2016
Copyright: © 2016 Ozawa et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper.
Funding: This study was funded by the Bill and Melinda Gates Foundation (OPPGH5190, http://www.gatesfoundation.org). Foundation staff advised the study team, but did not substantively affect the study design, instruments, interpretation of data, or decision to publish. The authors confirm that Amaltas Consulting Pvt. Ltd. provided support in the form of salaries for authors SS, KS, and VC but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the 'author contributions' section.
Competing interests: Some members of the research team (SS, KS, and VC) are employed by Amaltas Consulting Pvt Ltd, a private for-profit research and consulting firm. This work was conducted by Amaltas under a contract with Johns Hopkins University (supported by the Bill and Melinda Gates Foundation). The authors have no further competing interests or financial disclosures to make in this respect. This does not alter the authors' adherence to PLOS ONE policies on sharing data and materials.
Introduction
One of the greatest challenges of donor-led programs is to sustain activities and outcomes beyond the project funding period. Too often projects end abruptly when donor funding discontinues. This is particularly the case when the donor has supported service delivery through systems that are parallel to and separate from government’s own health systems. Sometimes donors invest in planning for a transition process whereby key elements of the program are handed over to local partners including government, but this kind of planned transition process is relatively rare [1]. Quantitative studies measuring the success of transitions are uncommon.
The challenges of transition are shared across many donor-led programs, and is a particularly pertinent question at this time given tempered growth in development assistance for health [2, 3] as well as for HIV/AIDS [4, 5], and plans to promote country ownership and transition within the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) [6–8]. This challenge is also relevant to ongoing graduation of countries from donor programs such as the Global Fund to Fight AIDS, Tuberculosis and Malaria [9] and Gavi, the Vaccine Alliance [10–12].
Transitions, if done well, provide an opportunity for donors to steward successful programs into the capable hands of local actors and integrate them into domestic country health systems. A successful transition involves formally handing over a program to one or more local partners and ensuring that the key outcomes of the program are sustained over time. An effective transition is often characterized by alignment and smooth transfer of key services, good communication about transition preparations and plans, and sufficient capacity among local program owners [1]. Preparations pre-transition are frequently described to be essential in this process [13, 14].
Transition may also involve a process of change, as donors and local actors may have different structures, management practices and organizational cultures. The extent of continuity may be captured by institutionalization, which considers the extent to which the initial program systems are integrated into organizational procedures and behaviors post-transition [15]. Institutionalization is seen to be important in program sustainability, considering the extent to which innovation and learning not only gets adopted and continued, but also gets reflected in institutional standards and norms that govern multiple organizations within the broader health system [15]. To date, there has been a dearth of empirical work to measure and assess the degrees of institutionalization post-transition [16].
Increasing attention has been given in recent years to examine whether various outcomes of health programs are sustained [17, 18]. A review of empirical literature identified 125 studies on sustainability which examined the performance of health programs after initial implementation [19]. Many studies reported that partial sustainability was more common than continuation of the entire program or intervention. However, the wide array of studied outcomes and durations made it difficult for reviewers to generalize the overall extent to which new programs and practices are sustained. The review also found the majority of studies to be retrospective, with few studies employing rigorous methods of evaluation. The large majority of studies took place in high income countries.
This study uses data from an assessment of the transition of the Avahan India HIV/AIDS prevention initiative to investigate how activities undertaken to prepare Avahan for transition affected continuation of program activities post-transition. Prior to transition, Avahan was fully supported by the Bill and Melinda Gates Foundation, whose staff managed grants or contracts with a range of international and national Non-Government Organizations (NGOs) [20]. These organizations in turn contracted with smaller NGOs and Community Based Organizations (CBO) to provide HIV/AIDS prevention services for key populations known as Targeted Interventions (TIs) [21, 22]. Key populations in this study included female sex workers and/or high-risk men having sex with men. Transition experiences from the community perspective were examined separately through qualitative studies [23].
Service delivery of Avahan was transitioned to the Government of India in three increasingly larger rounds of transition in 2009, 2011 and 2012. This study examines the transitions that took place in tandem in southern Indian states. Post-transition, the government took on responsibilities to fund and manage TIs, where they must adhere to national guidelines set by the National AIDS Control Organization (NACO), while being accountable to State AIDS Control Societies (SACS). TIs were transitioned to four different SACS where application of NACO norms varied by state in practice, making them a heterogeneous group. While the majority of TIs transitioned in original forms, some TIs had split because they were too large for NACO guidelines, or separated by key populations. Other TIs merged because they were too small for NACO norms, or discontinued. Other papers provide a fuller overview of the Avahan transition process, and the strengths and weaknesses of this process [1, 24–27].
This study examines the transition by focusing on three core elements: 1) how well prepared programs were prior to transition, 2) whether key elements of the program were institutionalized post-transition, and 3) whether outcomes were sustained through the transition. In particular, we examined relationships between transition readiness, institutionalization and outcomes to provide insights in the effectiveness of transition planning. We hypothesized that transition preparedness may explain what happens post-transition in terms of (i) institutionalization of key program elements one year after transition and (ii) sustained program delivery. This paper is unique in developing measures of transition preparedness and institutionalization, using data from both before and after transition to analyze the relationship between these variables, and examining linkages with sustainability of service outcomes.
Materials and Methods
Surveys were carried out in five southern states of India where Avahan was transitioned to local partners: Andhra Pradesh (now Andhra Pradesh and Telangana), Karnataka, Maharashtra, and Tamil Nadu. We conducted two rounds each of the transition readiness and institutionalization surveys. The transition readiness survey was conducted at the time of transition, whereas the institutionalization survey was carried out 12 to 18 months post-transition. All of the TIs from the 2011 round of transition were surveyed, whereas we used a list of transitioning TIs stratified by state to randomly select TIs from the larger 2012 round of transition. Data from 2011 and 2012 rounds of transition were combined together in the analysis to achieve sufficient sample size.
The transition readiness survey examined a range of indicators that sought to assess how well prepared the TIs were for transition. Through a literature review and conceptual framework of a transition logic model which was developed to evaluate the Avahan transition [1], we identified three elements of transition readiness: 1) capacity, 2) alignment, and 3) communication. Capacity indicators captured key operations of the TI, such as linkages made with government health facilities, formation of key population community groups and functioning of crisis response committees [28]. Alignment indicators measured levels of preparations made by the TI towards meeting NACO norms in areas such as team structure, budgeting and reporting. Communication captured whether staff were informed about the transition, transition plans incorporated staff inputs and project coordinators received training for the transition. Indicators were considered important where there were government norms to assess the extent to which TIs are aligned, as well as questions to understand how well prepared the TIs were for the transition in terms of capacity and communication [24, 29]. In addition to interview questions, the survey also included a review of data in the Avahan Computerized Management Information System (CMIS) from which we abstracted relevant indicators. For each indicator gathered from interviews and documents, we defined 3 levels (0 = low, 1 = medium and 2 = high) of transition readiness based on how well prepared TIs were for transition. For example, we classified the extent to which TIs met NACO norms, established linkages with government health services or had informed staff about the transition. The transition readiness survey captured 21 indicators, including 18 which were asked to TI managers and 3 which were abstracted from CMIS data.
The institutionalization survey examined whether transitioned TIs had institutionalized characteristics of the original program post-transition. Specifically, we asked whether transitioned TIs were continuing to regularly practice certain characteristics of the Avahan program. Core Avahan characteristics were identified from an earlier Delphi study, of which 13 of the 17 features were considered to be relevant post-transition [30]. The institutionalization survey included 18 interview questions which asked how frequently TIs carry out identified Avahan practices (i.e. regularly, sometimes, never). We then asked about Avahan practices in 20 statements using a five-point Likert scale, including whether the transition experience went smoothly overall. Additional 20 questions asked whether these practices changed due to the transition, which institution (i.e. government, TI or others) brought about the changes and whether these changes were for the better, worse, or made no difference. Interview questions were supplemented by a review of the government’s CMIS for relevant indicators.
Program outcome data were gathered from our surveys, the Avahan CMIS database and TI proposal documents. To observe changes pre- and post-transition, we selected indicators which were recorded in both the Avahan CMIS (for the pre-transition period) and the government’s CMIS (for the post-transition period). Available outcome indicators were limited based on the data that TIs gathered and reported. As a primary outcome indicator, we used the average percentage of key populations contacted by peer educators per month. Specifically, Avahan CMIS and government CMIS had collected the number of female sex workers and high-risk men having sex with men who were newly or repeatedly contacted by peer educators using counseling materials. These contacts were divided by the number of key populations served by each TI. As a secondary outcome indicator, we examined the average number of condoms distributed per key population per month. Sizes of key populations for each TI were obtained from TI proposal documents or the Avahan CMIS database [31]. Pre-transition data were abstracted from the Avahan CMIS database for 18 TIs to supplement the government CMIS data received from TIs. Panel analysis was used to plot observations. Heteroskedasticity was assessed using the Breusch-Pagan / Cook-Weisberg test [32, 33]. For each outcome a fixed effects model was developed with robust standard errors.
Measures of transition readiness and institutionalization were separately constructed through exploratory factor analysis. Based on the literature, we hypothesized the transition readiness measure to be multi-dimensional capturing elements of capacity, alignment and communication, and developed 3 sub-scales for this measure. However, there was no prior literature to inform the items to construct the scales. The literature also did not provide evidence to suggest the dimensionality of the institutionalization measure. Items for both scales were factor analyzed using iterated principal factor extraction with Varimax orthogonal rotation. The number of factors was determined using a scree test with eigenvalue greater than one. Items were analyzed and retained based on factor loadings, item-to-total correlation and inclusion of diverse elements of the concepts. Construct validity was examined by correlation analyses against other theoretically related constructs. Internal consistency was assessed by Cronbach’s alpha.
Using the two developed measures, ordinary least squares regression was subsequently carried out to examine the relationship between transition readiness and program outcomes directly, and indirectly through institutionalization (Fig 1). Relationships between transition readiness subscales and outcomes were also separately examined. For both outcome indicators, we examined the relationship of average outcomes 12 months pre-transition and 6 months post-transition. Univariate analyses were followed by multivariate analyses controlling for rounds of transition, whether the TI had split at the time of transition, NGO or CBO status of the TI, target key population (female sex workers, high-risk men having sex with men, or both), and state.
This study was ethically reviewed by Johns Hopkins School of Public Health Institutional Review Board (IRB No. 3157) and exempted as the study collected data from key informants but not data about individuals. In India, it was reviewed and approved by YRG Care Institutional Review Board (IRB No. 1423). Participants provided written informed consent to participate in the study.
Results
Transition readiness
Transition readiness data were collected from 80 TIs including all 27 TIs that transitioned in 2011 (100%) and 53 of the 155 TIs that transitioned in 2012 (34%). Our sample was representative of different states (34% Andhra Pradesh/Telangana, 29% Karnataka, 24% Maharashtra and 14% Tamil Nadu), type of organizations (64% NGO, 36% CBO) and TI support for key populations (84% served female sex workers, 57% served men having sex with men) (Table 1).
Overall, many TIs scored high across transition readiness indicators, demonstrating various adjustments made to meet NACO norms, build capacity and communicate prior to the handover. Most TIs had appropriately communicated and aligned the reporting systems (average score of 1.88 out of 2), staff structure (1.95 out of 2) and budgets (1.88 out of 2) prior to transition. Where transition readiness scores were lower, we found that some TIs delayed alignment on procurement of medicines due to buffer stocks (0.96 out of 2), or had removed condom outlets in hotspots because of vandalism and privacy concerns (1.21 out of 2). Scores were also lower for visits to Integrated Counselling and Testing Centers (ICTC) post-referral (1.10 out of 2), where government CMIS records suggested that 80% of referrals were actually seen at ICTC centers compared to 100% requested by NACO (Table 2).
Based on an item analysis of 21 transition readiness items, those with negative factor loadings and item-to-total correlations below 0.2 were removed. This resulted in 15 items in the transition readiness scale explaining 81% of the variance. Items were summed unweighted, with a mean score of 24.34 (standard deviation 3.09) and a range of 16–29. Indexed to a 0–100 scale, the mean was 81.11 (standard deviation 10.30). The scale demonstrated acceptable internal consistency with a Cronbach’s alpha of 0.73. Some evidence toward construct validity was found where the transition readiness scale was correlated with TIs reporting smooth transition experience at r = 0.29 (p = 0.03).
Exploratory factor analysis revealed 3 factors which appeared to reflect the three elements of transition readiness from the literature: capacity (eigenvalue 3.92), alignment (eigenvalue 2.57), and communication (eigenvalue 2.08), although not all items were aligned with the factor with which we had originally associated them. Three subscales were then developed for sub-analyses corresponding to the 3 identified factors. Factor 1 (capacity) was associated with 4 items, factor 2 (alignment) with 4 items, and factor 3 (communication) with 7 items (Table 3).
Institutionalization
Institutionalization data were collected from 70 TIs, including 28 TIs that transitioned in 2011 and 42 TIs that transitioned on time in 2012. Although we sought to conduct the transition readiness assessment and the institutionalization survey in the same sample of TIs, several TIs discontinued (1 TI in 2011 round), split up (2 TIs in 2011 round), merged (1 TI in 2012 round) or delayed the time of transition (10 TIs in 2012 round), resulting in differences in sample sizes. However, distributions of sampled TI characteristics were largely not affected (Table 1).
Frequency of practice of Avahan characteristics were examined on a scale from 0 to 2 (0 = never, 1 = sometimes or 2 = regularly). Most TIs reported actively using data for program planning (average score of 1.99 out of 2), regularly using the pictorial micro-planning tool (1.96 out of 2), and practicing rigorous performance monitoring of outreach workers (1.99 out of 2) [see Table 4]. However, some Avahan characteristics were less regularly maintained post-transition, such as providing flexibility on budgets (0.41 out of 2) and allowing exceptions to operating norms (0.89 out of 2). Another Avahan characteristic to have on-time, adequate and uninterrupted flow of funds to the grassroots level was often not continued, where 7 TIs (10%) reported regularly and 31 TIs (44%) sometimes facing challenges with cash flow that affected their operations during the 12 months post-transition.
Factor analysis revealed one dominant factor with an eigenvalue of 2.35. Among the 16 original items, 4 were removed due to negative factor loadings and low item-to-total correlation below 0.25 (Table 4). Dropped items included those with high average values with little variation, where items were less useful to differentiate TIs in a scale. The institutionalization scale was then developed using 12 items, with an average score of 18.04 and standard deviation of 2.59. On a scale from 0–100, this corresponds to a mean of 75.17 and standard deviation of 10.79. The scale explained 52% of the variance and demonstrated moderate internal consistency with a Cronbach’s alpha of 0.60. In assessing construct validity, the institutionalization scale was appropriately but weakly negatively correlated with TIs reporting that the overall program has changed significantly as compared to pre-transition r = -0.17 (p = 0.15).
Sustained Outcomes
TIs with outcomes data both pre- and post-transition were limited even after combining multiple data sources (government CMIS data obtained from TIs, Avahan CMIS database and TI proposal documents). Data were available for 55 TIs (79% of the institutionalization sample) for the primary outcome indicator: average percentage of key populations contacted by peer educators per month. Data were obtained for 64 TIs (91% of institutionalization sample) for the secondary outcome indicator: average number of condoms distributed per key populations per month.
Both outcomes were relatively stable between 12 months pre-transition and 6 months post-transition. While there was a minor drop in the average percentage of key populations contacted by peer educators one month post-transition, there were no trends observed comparing pre- and post-transition in the levels of either indicator based on the fixed effects model (Table 5). For both outcomes, the data was found to be heteroskedastic, where variance of the data reduced post-transition (p<0.001).
Effect of transition readiness and institutionalization on sustained outcomes
Based on a regression analysis, we observed a statistically significant relationship between transition readiness and sustained outcomes post-transition, for both key population coverage (2.47, p<0.01) and condom distribution (2.17, p = 0.03). The transition readiness measure was also predictive of outcomes comparing 6 months pre- and post-transition, for key population coverage (0.03, p<0.01) and condom distribution (0.04, p = 0.02). Examining the subcomponents of the transition readiness measure, this relationship was strongest across the dimensions of alignment and communication (Table 6). None of the variables we controlled for were statistically significant, including rounds of transition, NGO or CBO status of the TI, target key population or state.
However, this relationship between transition readiness and service delivery outcomes did not hold through institutionalization. Specifically, the transition readiness measure was weakly associated with institutionalization (0.01, p = 0.91) and significant associations were not observed between institutionalization and service outcomes (3.06, p = 0.08 for key population coverage; 0.25, p = 0.85 for condom distribution). While transition readiness was predictive of sustained outcomes, there were no indirect effects observed through institutionalization.
Discussion
We find that transition readiness can explain sustained program delivery in an HIV/AIDS prevention program in India handed over from a donor to the local government. Specifically, TIs with greater preparedness prior to the transition were more likely to achieve better service delivery outcomes post-transition, and this was also true for dimensions of the transition readiness scale, including alignment and communication. Although the four distinct SACS implemented NACO norms with some variation, we found that the relationship between transition readiness and sustained outcomes did not significantly differ by state.
Yet we also found that transition readiness did not necessarily lead to institutionalization of key program elements one year after transition, nor was institutionalization predictive of sustained program delivery. These results may be explained partly by difficulties in measuring the construct of institutionalization. While we developed our institutionalization measure based on the core characteristics of the Avahan program identified through a Delphi study [30], the measure may not have sufficiently captured the construct of institutionalization as indicated by the scale’s moderate internal consistency and modest proportion of explained variance. In addition, our study design did not capture the extent of Avahan practices prior to transition, thus making it difficult to assess whether there was truly a decline in the prevalence of these practices.
Furthermore, there appears to be some conceptual misalignment between measures of alignment with government norms in the transition readiness scale vis-à-vis the continuation of specific Avahan program characteristics post-transition captured through institutionalization. For example, ensuring on-time, adequate and uninterrupted flow of commodities was viewed to be an Avahan characteristic, yet TIs had to change their procurement with government supply systems to be transition ready, which likely led to interruptions in commodities supplied. Conversely, TIs that were better prepared for transition, as embodied by well-informed and skilled staff and high institutional capacity, may have been more inclined to innovate and change as compared to TIs with less capacity. Indeed, there appeared to have been substantial change in the TIs surveyed since transition, and much of this change was positively perceived by TI managers. As programs evolve over time, the linkages between transition readiness, institutionalization and sustained impact become more complex and would benefit from dynamic systems modeling [34].
While there was some evidence to suggest that the three elements of transition readiness—capacity, alignment and communication—based on a literature review and conceptual framework [1] was supported by the data, we found it to be far from conclusive. For instance, the loadings of the three factors did not align entirely with our initial conception of the three sub-scales. In addition, raising the item-to-total correlation cutoff level in the analysis removed variables and collapsed the subscales into one factor. This provides good fodder for future research to look into the sub-components of the transition readiness measure.
Our main findings support the largely grey literature in this area describing transitions and sustainability [18, 19, 35–37]. It also adds to the limited measurement literature around transitions and institutionalization [38–41]. While Goodman et al provides an eight-factor model on institutionalization of health promotion programs, we found it difficult to apply this in our study in the context of an evolving and transitioning project with a short follow-up time period [40]. The mostly descriptive accounts of how communication and transition planning affect the overall success of transition of HIV/AIDS programs [13, 14, 42] are supported through our analysis.
Some additional study limitations are important to note. First, the study was limited by the small sample size of TIs that took part in the program transition. While we surveyed 100% of TIs that transitioned in round 1 and 34% from round 2, the sample size of 80 TIs limit the power in our statistical analyses. Moreover, some TIs were split or discontinued, making it difficult to track their status post-transition. The quality of administrative data gathered through CMIS was also a limiting factor, where some data points were missing and restricted possible service delivery outcomes for analysis. Where possible, we supplemented the government CMIS data by extracting data from other sources such as the Avahan CMIS database and TI protocol documents. Finally, our analysis examined institutionalization one-year post transition, which may not have provided sufficient time for program characteristics to be reflected in institutional standards and norms, or capture the types of feed-back loops in the diffusion of service innovations [16]. Despite these limitations, this paper adds value by evaluating the transition of a large-scale donor-led program to local counterparts both prior to and after the transition.
This study contributes to the literature by developing approaches to measure transition readiness and institutionalization in the context of a health program transitioning from a donor to local partners. While our measure particularly of institutionalization appears imperfect, this study suggests that it may be practical and useful to assess transition readiness quantitatively prior to transition. While such measures should be tailored to fit the nature of the program being transitioned, it appears that they may be a good indicator of service sustainability post-transition, as well as helping to identify needs for further transition preparation.
Conclusion
This study offers important lessons for future transitions of donor programs. Specifically, we illustrate a methodology to measure transition readiness prior to transition, which could be predictive of service delivery outcomes post-transition. Such a measure could identify less ready organizations or program components in advance, improving the likelihood of service sustainability. Further research is needed around the conceptualization and development of measures of institutionalization and its effects on long-term program sustainability.
Acknowledgments
The Bill and Melinda Gates Foundation staff advised the study team but did not substantively affect the study design, instruments or interpretation of data. We particularly appreciate the advice and guidance of Hari Menon, Alkesh Wadhwani, Ganesh Ramakrishnan and Gina Dallabetta of the Foundation who provided advice and guidance. Ganesh Ramakrishnan and Gina Dallabetta also reviewed and commented on an earlier draft of this paper. The study team would like to thank respondents in all study sites for the time and insights they contributed to this project, without which this study would not be possible.
Author Contributions
Conceived and designed the experiments: SO SS SB. Performed the experiments: SS KS VC. Analyzed the data: SO SB. Wrote the paper: SO SS KS VC SB.
References
- 1. Bennett S, Singh S, Ozawa S, Tran N, Kang JS. Sustainability of donor programs: evaluating and informing the transition of a large HIV prevention program in India to local ownership. Glob Health Action. 2011;4. Epub 2011/12/21. pmid:22184502; PubMed Central PMCID: PMC3241943.
- 2. Dieleman JL, Graves CM, Templin T, Johnson E, Baral R, Leach-Kemon K, et al. Global health development assistance remained steady in 2013 but did not align with recipients' disease burden. Health Aff (Millwood). 2014;33(5):878–86. Epub 2014/04/10. pmid:24714869.
- 3.
Valentine A, Wexler A, Oum S, Kates J. Donor funding for health in low- and middle-income countries 2002–2013. Washington, D.C.: The Henry J. Kaiser Family Foundation, 2015.
- 4.
Smith K. Kaiser/UNAIDS study finds dip in donor government commitments for AIDS in 2013 July 17, 2014. Available from: http://kff.org/global-health-policy/press-release/kaiserunaids-study-finds-dip-in-donor-government-commitments-for-aids-in-2013/.
- 5.
Funders Concerned About AIDS. Global philanthropic support to address HIV/AIDS in 2012. Washington, D.C.: Funders Concerned About AIDS, 2013.
- 6. Palen J, El-Sadr W, Phoya A, Imtiaz R, Einterz R, Quain E, et al. PEPFAR, health system strengthening, and promoting sustainability and country ownership. J Acquir Immune Defic Syndr. 2012;60 Suppl 3:S113–9. Epub 2012/09/14. 00126334-201208153-00010 [pii]. 22797732. pmid:22797732
- 7. Goosby E, Von Zinkernagel D, Holmes C, Haroz D, Walsh T. Raising the bar: PEPFAR and new paradigms for global health. J Acquir Immune Defic Syndr. 2012;60 Suppl 3:S158–62. Epub 2012/07/20. 00126334-201208153-00016 [pii]. pmid:22797738.
- 8.
Institute of Medicine of the National Academies. Preparing for the Future of HIV/AIDS in Africa: a Shared Responsibility. Washington, D.C.: The National Academies Press, 2010.
- 9. Atun R, Kazatchkine M. Promoting country ownership and stewardship of health programs: The global fund experience. J Acquir Immune Defic Syndr. 2009;52 Suppl 1:S67–8. Epub 2009/11/05. pmid:19858945.
- 10. Saxenian H, Hecht R, Kaddar M, Schmitt S, Ryckman T, Cornejo S. Overcoming challenges to sustainable immunization financing: early experiences from GAVI graduating countries. Health Policy Plan. 2014. Epub 2014/02/11. pmid:24510369.
- 11. Le Gargasson JB, Breugelmans JG, Mibulumukini B, Da Silva A, Colombini A. Sustainability of National Immunization Programme (NIP) performance and financing following Global Alliance for Vaccines and Immunization (GAVI) support to the Democratic Republic of the Congo (DRC). Vaccine. 2013;31(15):1886–91. Epub 2013/03/07. pmid:23462529.
- 12. Shen AK, Farrell MM, Vandenbroucke MF, Fox E, Pablos-Mendez A. Applying lessons learned from the USAID family planning graduation experience to the GAVI graduation process. Health Policy Plan. 2014. Epub 2014/06/30. pmid:24974106.
- 13. Katz I, Glandon D, Wong W, Kargbo B, Ombam R, Singh S, et al. Lessons learned from stakeholder-driven sustainability analysis of six national HIV programmes. Health Policy Plan. 2013. Epub 2013/04/25. czt024 [pii] pmid:23612848.
- 14. Torpey K, Mwenda L, Thompson C, Wamuwi E, van Damme W. From project aid to sustainable HIV services: a case study from Zambia. J Int AIDS Soc. 2010;13:19. Epub 2010/06/10. 1758-2652-13-19 [pii]. pmid:20529255; PubMed Central PMCID: PMC2890667.
- 15. Pluye P, Potvin L, Denis JL. Making public health programs last: conceptualizing sustainability. Evaluation and Program Planning. 2004;27(2):121–33.
- 16. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. The Milbank quarterly. 2004;82(4):581–629. Epub 2004/12/15. pmid:15595944; PubMed Central PMCID: PMC2690184.
- 17. Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health programs. Am J Public Health. 2011;101(11):2059–67. Epub 2011/09/24. AJPH.2011.300193 [pii]. pmid:21940916.
- 18. Scheirer MA. Is sustainability possible? A review and commentary on empirical studies of program sustainability. American Journal of Evaluation. 2005;26(3):320–47.
- 19. Stirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7:17. Epub 2012/03/16. 1748-5908-7-17 [pii]. pmid:22417162; PubMed Central PMCID: PMC3317864.
- 20.
The Bill and Melinda Gates Foundation—Avahan—the India AIDS Initiative. The business of HIV/AIDS prevention at scale. New Delhi, India: The Bill and Melinda Gates Foundation, 2008.
- 21. Chandrasekaran P, Dallabetta G, Loo V, Mills S, Saidel T, Adhikary R, et al. Evaluation design for large-scale HIV prevention programmes: the case of Avahan, the India AIDS initiative. AIDS. 2008;22 Suppl 5:S1–15. Epub 2009/01/07. pmid:19098469.
- 22. Pickles M, Boily MC, Vickerman P, Lowndes CM, Moses S, Blanchard JF, et al. Assessment of the population-level effectiveness of the Avahan HIV-prevention programme in South India: a preplanned, causal-pathway-based modelling analysis. The Lancet Global health. 2013;1(5):e289–99. Epub 2014/08/12. pmid:25104493.
- 23. Rodriguez DC, Tripathi V, Bohren M, Paul A, Singh K, Chhabra V, et al. "From me to HIV": a case study of the community experience of donor transition of health programs. BMC infectious diseases. 2015;15:349. pmid:26286724; PubMed Central PMCID: PMC4545328.
- 24. Bennett S, Ozawa S, Rodriguez D, Paul A, Singh K, Singh S. Monitoring and evaluating transition and sustainability of donor-funded programs: Reflections on the Avahan experience. Eval Program Plann. 2015;52:148–58. pmid:26099560.
- 25. Bennett S, Rodriguez D, Ozawa S, Singh K, Bohren M, Chhabra V, et al. Management practices to support donor transition: lessons from Avahan, the India AIDS Initiative. BMC Health Serv Res. 2015;15:232. pmid:26071052; PubMed Central PMCID: PMC4465612.
- 26. Bennett S, Singh S, Rodriguez D, Ozawa S, Singh K, Chhabra V, et al. Transitioning a Large Scale HIV/AIDS Prevention Program to Local Stakeholders: Findings from the Avahan Transition Evaluation. PLoS One. 2015;10(9):e0136177. pmid:26327591; PubMed Central PMCID: PMC4556643.
- 27. Sgaier SK, Ramakrishnan A, Dhingra N, Wadhwani A, Alexander A, Bennett S, et al. How the Avahan HIV prevention program transitioned from the Gates Foundation to the government of India. Health Aff (Millwood). 2013;32(7):1265–73. Epub 2013/07/10. pmid:23836743.
- 28. Wheeler T, Kiran U, Dallabetta G, Jayaram M, Chandrasekaran P, Tangri A, et al. Learning about scale, measurement and community mobilisation: reflections on the implementation of the Avahan HIV/AIDS initiative in India. J Epidemiol Community Health. 2012;66 Suppl 2:ii16–25. Epub 2012/09/14. pmid:22945907.
- 29.
National AIDS Control Organization. National AIDS Control Programme Phase III: 2006–2011. Delhi: National AIDS Control Organization, Ministry of Health and Family Welfare, Government of India, 2006.
- 30. Mabuchi S, Singh S, Bishnu R, Bennett S. Management characteristics of successful public health programs: "Avahan" HIV prevention program in India. Int J Health Plann Manage. 2012. Epub 2012/12/12. pmid:23224946.
- 31.
Avahan Computerized Management Information System (CMIS) Data [Internet]. FHI 360. 2014. Available from: http://dx.doi.org/10.7910/DVN/24631.
- 32. Breusch TS, Pagan AR. The Lagrange multiplier test and its applications to model specification in econometrics. Review of Economic Studies. 1980;47(1):239–53.
- 33. Cook D, Weisberg S. Diagnostics for heteroscedasticity in regression. Biometrika. 1983;70(1):1–10.
- 34. Paina L, Peters DH. Understanding pathways for scaling up health services through the lens of complex adaptive systems. Health Policy Plan. 2012;27(5):365–73. Epub 2011/08/09. pmid:21821667.
- 35. Hirschhorn LR, Talbot JR, Irwin AC, May MA, Dhavan N, Shady R, et al. From scaling up to sustainability in HIV: potential lessons for moving forward. Global Health. 2013;9:57. Epub 2013/11/10. pmid:24199749; PubMed Central PMCID: PMC3826849.
- 36. Sarriot EG, Winch PJ, Ryan LJ, Bowie J, Kouletio M, Swedberg E, et al. A methodological approach and framework for sustainability assessment in NGO-implemented primary health care programs. Int J Health Plann Manage. 2004;19(1):23–41. Epub 2004/04/06. pmid:15061288.
- 37. Gruen RL, Elliott JH, Nolan ML, Lawton PD, Parkhill A, McLaren CJ, et al. Sustainability science: an integrated approach for health-programme planning. Lancet. 2008;372(9649):1579–89. Epub 2008/11/06. S0140-6736(08)61659-1 [pii]. pmid:18984192.
- 38. Slaghuis SS, Strating MM, Bal RA, Nieboer AP. A framework and a measurement instrument for sustainability of work practices in long-term care. BMC Health Serv Res. 2011;11:314. Epub 2011/11/18. pmid:22087884; PubMed Central PMCID: PMC3234291.
- 39. Sarriot E, Ricca J, Ryan L, Basnet J, Arscott-Mills S. Measuring sustainability as a programming tool for health sector investments: report from a pilot sustainability assessment in five Nepalese health districts. Int J Health Plann Manage. 2009;24(4):326–50. Epub 2009/12/01. pmid:19946943.
- 40. Goodman RM, McLeroy KR, Steckler AB, Hoyle RH. Development of level of institutionalization scales for health promotion programs. Health education quarterly. 1993;20(2):161–78. Epub 1993/01/01. pmid:8491630.
- 41. Ridde V, Pluye P, Queuille L. Assessing program sustainability in public health organizations: a tool-kit application in Haiti. Revue d'epidemiologie et de sante publique. 2006;54(5):421–31. Epub 2006/12/07. pmid:17149163.
- 42. Swidler A, Watkins SC. 'Teach a Man to Fish': The Doctrine of Sustainability and Its Effects on Three Strata of Malawian Society. World development. 2009;37(7):1182–96. Epub 2010/02/18. pmid:20161458; PubMed Central PMCID: PMC2791326.