Criteria for Assessing a Program’s Certainty in Achieving Intended Outcomes
| Criterion and Rationale | Example Questions | Interpretation |
|---|---|---|
| Context: The context where an implementer implements an intervention has many and varied effects on intervention outcomes. When an implementer knows the context well, they can anticipate how it will affect their implementation activities. This predictability in how context influences an intervention influences certainty that the design of an intervention is appropriate for the context. |
• How similar or different is the setting for a program compared to the setting of previous program activities? • How much has the context changed over time? • Are conditions stable, or have they changed recently? |
Increased certainty: More similar settings and more stable conditions (less change in the context) mean more predictable outcomes, which increases certainty. Decreased certainty: Less similar setting and less stable conditions (more change in the context) mean less predictable outcomes, which decreases certainty. |
| Maturity: An implementer can more accurately anticipate how program design changes will influence outcomes when they have more experience carrying out the program or when the elements of the program design are older and more familiar to them. | • How much experience does the implementer have in implementing the existing program design? |
Increased certainty: With older, more mature programs, implementers have greater experience and familiarity with how they work in practice, which increases certainty. Decreased certainty: With newer, less mature programs, implementers have less experience and familiarity with how they work in practice, which decreases certainty. |
| Precision: Decision-makers have different standards for the robustness of the evidence they take into consideration when making decisions. Sometimes they want to be very certain in the evidence they are considering, while other times they are okay if the evidence requires a lot of caveats. From a research perspective, this translates into confidence intervals. If those intervals can be wider, the goal of the research would then be less about obtaining exact point estimates and more about understanding directionality (i.e., can we generally expect a positive outcome or a negative outcome?). The precision criterion is typically more salient when assessing the use of quantitative research methods. |
• How precise do the estimates of change caused by the adaptive learning activity need to be? • Do the confidence intervals on the estimate need to be very narrow—and therefore very defensible from a statistical point of view? • Or can they be wider—and less defensible? |
Increased certainty: The narrower confidence intervals that come with more precision increases certainty. Decreased certainty: The wider confidence intervals that come with less precision decreases certainty. |
| Urgency: Evidence generated through a learning activity will ideally be used to inform key decisions about program implementation. How soon those decisions must be made will dictate how certain we can be that a program’s design has fully run its course and produced its intended results. |
• How soon does the adaptive learning activity need to produce results? • Can the implementer wait for a year to receive the results? • Or will the implementer be making a decision in 1 month and so needs information as soon as possible? |
Increased certainty: Less urgency increases certainty because there will be more time to measure the outcomes from design changes. Decreased certainty: More urgency decreases certainty because there will be less time to measure the outcomes from design changes. |