Process Evaluation of Teaching Critical Thinking About Health Using the Informed Health Choices Intervention in Rwanda: A Mixed Methods Study

Michael Mugisha; Andrew D. Oxman; Laetitia Nyirazinyoye; Anne Marie Uwitonze; Clarisse Marie Claudine Simbi; Faith Chesire; Ronald Ssenyonga; Matt Oxman; Allen Nsangi; Daniel Semakula; Margaret Kaseje; Nelson K. Sewankambo; Sarah Rosenbaum; Simon Lewin

doi:10.9745/GHSP-D-23-00483

Figures & Tables

Figures

Tables
Additional Files

Download figure
Open in new tab
Download powerpoint
FIGURE
Process Evaluation Conducted as Part of the Intervention Arm of a Cluster-Randomized Trial of the Informed Health Choices Secondary School Intervention, Rwanda

View popup

TABLE 1.

Data Collection for Process Evaluation of Informed Health Choices Secondary School Intervention, Rwanda

Method and Timing	Source and Sampling	Data Collection Process and Tools
Workshop evaluation to assess teachers’ perceptions of training they received (immediately after teacher training workshop)	All teachers from intervention arm who attended the training (N=42)	Teachers completed an online questionnaire with 5-point Likert response options that assessed the quality of training, extent to which training goals were achieved, and their readiness to deliver the intervention as intended.
Lesson evaluations to assess the delivery of each lesson (immediately after each lesson)	All teachers in intervention arm (N=42) for each of the 10 lessons.	Teachers completed an online lesson evaluation form describing how they prepared for and taught the lesson and the extent to which the lesson objectives were achieved.
Non-participatory observation of lessons to observe how the lessons were taught (during intervention delivery)	Intervention-arm schools (N=16) that were purposively sampled to ensure variation in ownership (private, public, or government-aided) and performance (high or low), as defined by the National Examination and School Inspection Authority. We observed all 10 lessons at least once.	We sat in classes during lesson delivery and used a structured observation form to note how the lesson was taught. We recorded how the teacher delivered the lesson and how students responded.
KIIs to explore how participants experienced the intervention (after intervention delivery)	Purposively sampled teachers (N=10) from schools that varied by type (day or boarding), ownership (private, public, or government-aided) and performance (high or low). In each school, head teachers or director of studies (N=10). Policymakers from the Rwanda Basic Education Board with experience developing and implementing the intervention (N=2).	We used semistructured interview guides to conduct the interviews. We interviewed participants at their workplace in a convenient location that also ensured privacy and quality recording of discussions. Each interview lasted for 1–1.5 hours. Two researchers conducted each interview. One person led the discussion, and another took notes and audio-recorded the discussion. We transcribed verbatim all the recordings and translated to English if the interview was conducted in Kinyarwanda.
FGDs to explore how students and their parents or caregivers experienced the intervention (after intervention delivery)	Purposively sampled schools as described above for KIIs. In each of the 10 schools, we conducted 1 focus group for students (N=10 FGDs). Students varied in terms of age, sex, and performance. Each FGD included 8–10 students. For parents, we focused on 5 of the 10 selected schools that were day schools. In those schools, we used purposive sampling to select parents who were invited to the FGDs. We invited parents who had discussed the intervention with their children. Each FGD included 8–10 parents.	We used semistructured FGD guides to conduct discussions with students and their parents, respectively. We conducted the discussions at the students’ school, in a room where no teachers or school leaders were present. For all FGDs, 1 researcher moderated, and another took notes and audio-recorded the discussion. The duration of each FGD was 1–1.5 hours. We transcribed verbatim all recordings and translated to English.

Abbreviations: FGD, focus group discussion, KII, key informant interview.

View popup

TABLE 2.

Characteristics of Schools and Participants in Informed Health Choices Intervention Lesson Observations, Rwanda

	No.
Schools	(N=16)
School ownership
Government aided	10
Public	4
Private	2
School performance
Low	7
High	9
Participants
Students	(N=100)
Age, years
12–15	61
16–18	39
Sex
Male	42
Female	58
School staff	(N=20)
Age, years
45 or younger	13
Older than 45	7
Sex
Male	17
Female	3
Role
Science teacher	10
School leader	10
Parents	(N=40)
Age
45 or younger	17
Older than 45	23
Sex
Male	10
Female	30
Policymakers	(N=2)

View popup

TABLE 3.

Teachers’ Views of How They Experienced the Informed Health Choices Training, Rwanda

	Agree, No. (%)	Strongly Agree, No. (%)
N=42
The training gave me general understanding of the critical thinking about health.	11 (26.2)	31 (73.8)
The training gave me a clear overview and flow of all lessons.	14 (33.3)	28 (66.7)
I can navigate through the website, and I know where I can find all that I need.	17 (40.5)	25 (59.5)
Now I understand all teaching strategies relevant for teaching critical thinking about health.	21 (50.0)	21 (50.0)
The training gave me teaching tips that I need to consider while teaching CHOICE lessons.	13 (30.9)	29 (69.1)
I am confident that I understand and can teach all 10 lessons.	17 (40.5)	25 (59.5)
The training met my expectations.	24 (57.1)	18 (42.9)
I will be able to apply the knowledge learned.	11 (26.2)	31 (73.8)
Overall rating of the training	16 (30.1)	26 (61.9)

View popup

TABLE 4.

Teachers’ Feedback on How Lessons Taught in the Informed Health Choices Intervention Was Delivered and How the Learning Goals Were Achieved, Rwanda

	Lessons
(N=42^a)	1	2	3	4	5	6	7	8	9	10
Lesson preparation
Time taken to prepare the lesson (minutes), mean (SD)	50.5 (28.9)	47.4 (30.7)	47 (29.8)	44.8 (26.5)	35.5 (17.2)	44.7 (30.6)	42.2 (18)	48.6 (38.7)	42.6 (28.1)	46.4 (28.9)
Level of preparedness
Prepared or very prepared	41 (97.6)	40 (97.6)	42 (100)	41 (97.6)	40 (100)	41 (100)	42 (100)	41 (100)	42 (100)	41 (100)
Unprepared or very unprepared	1 (2.4)	1 (2.4)	0 (0)	1 (2.4)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)	0 (0)
Lesson delivery
Delivered on planned date
Yes	39 (92.9)	32 (78.1)	24 (57.1)	31 (73.8)	27 (67.5)	26 (63.4)	31 (73.8)	31 (75.6)	32 (76.2)	31 (76.6)
No	3 (7.1)	9 (21.9)	18 (42.9)	11 (26.2)	13 (32.5)	15 (36.6)	11 (26.2)	10 (24.4)	10 (23.8)	10 (24.4)
Mode of delivery
Projector based	40 (95.2)	39 (95.1)	38 (90.5)	40 (95.2)	38 (95)	38 (92.7)	39 (92.9)	41 (100)	40 (95.2)	38 (92.7)
Blackboard based	2 (4.8)	2 (4.9)	4 (9.5)	2 (4.8)	2 (5)	7 (7.3)	3 (7.1)	0(0)	2 (4.8)	3 (7.3)
Changed the mode of delivery as planned
Yes	3 (7.1)	3 (7.3)	4 (9.5)	4 (9.5)	4 (10)	6 (14.6)	4 (9.5)	3 (7.3)	3 (7.1)	3 (7.3)
No	39 (92.9)	38 (92.7)	38 (90.5)	38 (90.5)	36 (90)	35 (85.4)	38 (90.5)	38 (92.7)	39 (92.9)	38 (92.7)
No. students who attended lesson, mean (SD)	39.9 (9.1)	39.8 (9.1)	39.2 (9.5)	39.4 (8.7)	38.5 (9.8)	39.1 (9.3)	38.8 (9.3)	38.6 (8.9)	38.6 (9.4)	38.6 (9.2)
Time taken to complete lesson, median (IQR)	46.3 (8.9)	44.1 (8.7)	44.4 (9.8)	42.3 (5.6)	42.1 (6.2)	42 (4.9)	42.4 (6.1)	41.9 (4.2)	42.3 (4.9)	43.6 (6.2)
Level of ease or difficulty in teaching the lesson
Easy or very easy	37 (88.1)	39 (95.1)	39 (92.9)	40 (95.2)	38 (95.0)	31 (75.6)	41 (97.6)	41 (100)	41 (97.6)	39 (95.1)
Difficult	5 (11.9)	2 (4.9)	3 (7.14)	2 (4.8)	2 (5.0)	10 (24.4)	1 (2.4)	0 (0)	1 (2.4)	2 (4.9)
Extent to which lesson objectives were achieved
Achieved or very much achieved	41 (97.6)	39 (95.1)	41 (97.6)	41 (97.6)	40 (100)	38 (92.7)	41 (97.6)	41 (100)	42 (100)	39 (95.1)
Too little achieved or unachieved	1 (2.4)	2 (4.9)	1 (2.4)	1 (2.4)	0 (0)	3 (7.3)	1 (2.4)	0 (0)	0 (0)	2 (4.9)

Abbreviations: IQR, interquartile range; SD, standard deviation.
↵^a Items where numbers do not add up to 42 had missing data.