Responsive feedback: Towards a new paradigm to enhance intervention effectiveness

The current dominant models of intervention design are fraught with challenges that make it difficult for programs to be responsive to the complexity and unpredictability of implementation challenges. Partly, the dominant model is influenced by medical or pharmaceutical trial models which use randomized controlled trials as the gold standard, assume high degree of fidelity and certainty to intervention and seemingly invariant conditions when the stimulus or drug is administered. The reality of social and behavioral interventions, on the other hand, are far too different posing tremendous challenges. For example, current tools and models for understanding deviations from what was originally expected and what is observed during implementation do not allow programs to respond with the flexibility and agility required in rapidly changing field situations. Limited flexibility in programmatic response is often seen when limited or no feedback is provided to decision-makers, including implementers, during the implementation process. This lack of agility may be a function of the types of data, study designs and management skills needed to respond to changing implementation needs. It is equally possible that an organization’s culture may not be open to receiving and acting on feedback.

A culture of continuous learning may be needed for a host of reasons, including changing consumer expectations, unexpected implementation constraints or faulty assumptions made at the project design stage. Our conventional approach to assessing the effectiveness of interventions by collecting data at the end of the intervention, provides limited opportunity to be responsive to day-to-day developments in the field. The long interval between the end of the intervention and completion of the evaluation means that the evidence generated is of limited utility in course correction and in providing timely feedback to implementers. Moreover, the lack of a systematic process for linking ongoing implementation learnings to modifications in project design - the feedback loop - often precludes adaptive implementation.

Recent thinking has posed a bold challenge to this orthodoxy aided by developments in theory, methods and practice, calling for an approach that promotes interaction between project designers, implementers, researchers and decision-makers, to encourage adaptation through learning. This newer approach, often using such terms as “responsive feedback” or “feedback loops,” calls for timely assessments that provide actionable feedback to implementers to course correct and achieve intended outcomes. Three developments have contributed to the increasing momentum of this approach. One, the advent of information and communication technologies (ICTs) has increased opportunities to collect, analyze and disseminate evidence/data more rapidly. Two, multi-disciplinary and multi-sector thinking has allowed for greater sharing, adoption and adaptation of lessons from different sectors and is increasingly infusing thinking in the development sector. Three, there is a shift among implementing organizations from considering monitoring and evaluation as an accounting or “auditing” function (Colquhoun et al., 2017) to a learning function that drives continuous improvement.

In this letter, we are calling for a wider adoption of this new approach to enhance the effectiveness of interventions, ultimately leading to improvements in the lives of people for whom the interventions are intended. We will briefly touch upon some important issues regarding challenges in adopting a responsive feedback approach, and ways to overcome these challenges. This is not meant to be an exhaustive review of a responsive feedback approach as much as an effort to generate dialogue among donors and practitioners among others. Our approach builds on earlier work in social and behavioral sciences that offered multiple frameworks for implementation and from literature on management that especially integrated implementation sciences with improvement sciences (Balasubramanian et al., 2015; Gaglio et al., 2013; Glanz et al., 2015; Green & Kreuter, 2005). We must also note that the RFM approach builds on and is complementary to similar dialogues in the development sector such as “Doing Development Differently” which also call for “rapid cycles of planning, action, reflection and revision” in implementing social change (Harvard Center for International Development, 2014).

How to label the new approach

To start with, one challenge worth noting is that there is a lack of consensus over terms and definition of what responsive feedback is. While developing a consensus is not the aim of this letter, we contend that it is critical to develop standard terminology and a broadly acceptable definition of what this approach may be called. We argue for the terms “responsive” and “feedback” to characterize the philosophy and goals of this approach. A responsive feedback approach reduces the tension between traditional monitoring and evaluation and traditional implementation and decision-making functions. It proposes that the design of research activities should be driven by the explicit intention of providing actionable insights to implementers. For the learning function to be effective (as judged by whether it leads to improved implementation and design decisions), interaction among researchers, implementers, and other decision-makers is important at defined moments throughout the project’s life cycle. In turn, the learning questions and the timing of these moments should be guided by the project’s Theory of Change.

Theories of Change and responsive feedback approach

A Theory of Change (ToC) is critical as, by identifying pathways and markers of course-correction, it can inform factors that influence the effectiveness of an intervention to ensure that short-term to long-term objectives are met. However, few interventions have an explicit ToC and hypotheses that are continuously tested and refined to improve the interventions. We argue that a ToC is an essential complement of a responsive feedback approach.

A ToC approach is about making our forward-thinking narrative explicit – and the assumptions that underlie one’s thinking. It clarifies how we see cause-effect relations between activities or actions and their intended changes, ensures that causal links and assumptions behind the links are explicit, and helps hone in on the relationship between activities and the achievement of long-term goals. Instead of becoming fixated on what the program is currently doing, it draws people’s minds to the activities that are needed to achieve the goals. This leads to better planning in that activities are linked to a detailed understanding of how change actually unfolds.

ToCs are largely focused on uncovering and critically appraising assumptions, with learning as a key goal of the process. Responsive Feedback Mechanisms (RFM) are a tool to support the practices of learning and adaptive thinking – making ToCs and RFM critical complements to one another. And, by using a collaborative and participatory process, developing a ToC should involve discussion about existing understandings of how change should happen and articulating the underlying assumptions. Through this process one may identify where there are structural inconsistencies or contradictions, particularly around cause-effect relations in the logic of change pathways, or where there is uncertainty, pinpointing where RFM can be most useful.

In summary, these points identify potentially powerful use of ToCs and RFM to drive program improvement – with the ToC setting the learning agenda and uncovering assumptions in need of validation, and RFM providing the tools and methods to do so.

Characteristics of RFM

What are some of the principal characteristics of responsive feedback approach and the interventions informed by RFM? There are five key features:

Challenges in implementing a responsive feedback culture

While the idea of responsive feedback is slowly being appreciated, it is by no means without its challenges, particularly in its operationalization. These challenges include the organizational structure and culture, capabilities of both researchers and implementers and resource constraints.

Opportunities

While there are challenges in adopting responsive approaches, there are a number of ongoing developments that make it conducive to accelerate their adoption.

One among them is the alacrity with which the social sector has adopted the tools stemming from the revolution in information and communication technologies. We are witnessing the emergence of the idea of digital health, “a set of activities and tools that encompass health, information and communication technology (ICT), including mobile health (mHealth), health information technology (IT), health information systems, wearable devices, telehealth, and telemedicine” (National Academy of Sciences, 2017). Development organizations are increasingly taking advantage of these changes particularly as barriers to adaption are lowering.

But perhaps more foundational is the shift among development practitioners to participatory, actionable, and adaptive MLE approaches. This cultural shift is accompanied by an expansion in the ways traditional monitoring and evaluation tools are being applied to support ongoing learning opportunities – by building in participation of key stakeholders throughout design and implementation or by supporting capacity development for organizational staff on MLE.

Implications

In this letter, we do not make any claims about how easy it is to adopt an RFM approach nor do we discuss in any depth the modes of observation for data collection to facilitate decision-making. Program planners and managers have to prioritize “what” to observe and “when” to observe based on the ToC to help in decision making based on such considerations as costs, order of importance, skills and practicality in decision making (Hornik, 1992).

A culture of responsive feedback begs commitment from all parties involved in project design, implementation and learning. Donors need to provide the impetus for interactions between implementers and researchers that result in opportunities for learning and adaptation. This may require an upfront commitment of time and resources to test areas of uncertainty within the ToC using formative research or small experiments. It may have implications for what functions are needed for project management, for the type of reporting needed and possibly even for what is admissible as learning. Requests for proposals would have to highlight the importance of returning to the ToC periodically and using it as a way of navigating improvements in implementation. At the same time, commitments would be needed of implementers, decision-makers, researchers, and other development practitioners to embrace the RFM approach, actively participate in continuous learning, and act on the evidence generated. Our organizations have each experimented in different ways with using RFM to enhance intervention effectiveness and believe in its power to effect development change on the pathway to impact.

Disclaimer

The views expressed in this article are those of the authors. Publication in Gates Open Research does not imply endorsement by the Bill & Melinda Gates Foundation.

Data and software availability

No data are associated with this article.