Process Evaluation of Teaching Critical Thinking About Health Using the Informed Health Choices Intervention in Uganda: A Mixed Methods Study

Time Used to Teach the Lessons

We planned for the intervention to include 10 40-minute lessons that were to be taught once a week in a single school term. We anticipated that teachers would take 30 minutes to prepare for each lesson and that students’ attendance rates for the IHC lessons would be similar to other subjects like biology, English, and mathematics.

As anticipated, teachers spent an average of 32 minutes (standard deviation [SD] 12 minutes) preparing for the IHC lessons (Table 3). One teacher reported that he prepared for one of the lessons during the taxi ride to school. Teachers who felt that preparation took too long (over 40 minutes) highlighted that some of the resources were lengthy and needed to be read several times. In addition, there were other school engagements that interfered with the preparation time. Most of those who felt it took too long to prepare thought that preparation time would be shorter if they were teaching these lessons for a second or third time. Most teachers (93.5%) reported that they were prepared to teach the lessons, and 80% found they felt confident to teach the IHC lessons (Figure).

• Download figure
• Open in new tab
• Download powerpoint

FIGURE

Teacher’s Feelings of Being Prepared and Confident to Teach the Informed Health Choices Lessons, Uganda

However, the delivery of IHC lessons took longer than we anticipated (Table 3). Despite repeated efforts during intervention development to simplify and shorten the lessons so they would fit into a 40-minute slot, lessons took, on average, 69 minutes to complete, and some took up to 80 minutes). Lessons 1 (Health actions) and 8 (Personal choices) took the least time, an average of 67 (SD 13) minutes. Lesson 3 (Unreliable claims) took the most time, an average of 74 (SD 12) minutes. The time used to teach these lessons was similar to what teachers initially recommended in the context analysis¹⁶ and what we found in a pilot study.²⁰

Teachers carried out nearly all the planned lessons (395/400 [98.8%]). One teacher taught only 5 of the 10 lessons, citing a change in the school program that did not allow her to complete the lessons. Twenty-two of the 40 intervention schools were able to teach 1 lesson per week as planned. The remaining schools needed to teach more than 1 lesson a week due to missed lessons. Fourteen (90%) of the 16 schools with a projector either used it for all or most of the lessons. Class attendance among students fluctuated from lesson to lesson (Table 3).

Despite an overly full school timetable after the long school closure due to the COVID-19 pandemic, all but 1 of the teachers were able to squeeze in the lessons. Nonetheless, this and several other factors may have adversely affected teachers’ and students’ motivation, the effort they put into these lessons, and what they got out of them. There also was a teachers’ strike that lasted 3 weeks at the beginning of the school term in which the intervention was delivered. In addition, this term included several extracurricular school activities, such as sports, music, and drama, as well as religious festivities, patriotic club trainings, and others.

Students came back more mature [after a 2-year school closure due to COVID-19], less interested in academics, and also they faced many interruptions during the lessons, for example, some had to go for sports, drama, and academic tours. —Teacher, public school, KII

Achievement of Lesson Objectives

Thirty-six of the 40 teachers reported having achieved the lesson objectives and mentioned that they used the number of correct responses from students to the wrap-up questions in the IHC lessons as a measure. This corresponded with our lesson observations. In FGDs, students correctly defined key terms and provided correct summaries of what they had learned. Teachers also reported this.

Learners were able to define the terms like health actions, health claims and also provide a correct summary of what they had learnt from class, for example, identify helpful and harmful health actions, and also appreciate the use of critical thinking in their daily lives. —Teacher, public school, teachers’ lesson evaluation forms

Accessing Resources

Only 1 of the 40 teachers did not use a smartphone to access the educational resources. This teacher’s phone was stolen after the teacher training workshop, so we provided him with a print-out of all 10 teacher summaries for the lessons. These 4–5-page summaries were part of the extra resources on the website and included the lesson goals, key terms, and the blackboard version of the lesson plan.

Delivery Description

We observed 67 lessons, with at least 1 lesson observed in each of the 40 schools that implemented the intervention (Table 4). In 97% of the observed lessons, teachers reviewed key messages from the previous lesson. They often asked students to say what they remembered from the previous lesson before the review. Our observations regarding students’ attendance, time spent delivering the IHC lessons, and teaching strategies did not differ from what teachers reported in the lesson evaluation forms. We observed that teachers tended to use class discussions rather than small group discussions or buzz groups when there was limited time to complete a lesson. This was also confirmed by teachers in interviews.

Perceived Overall Value

The intervention was aimed at equipping young people with skills and knowledge that could help them cope with the large amount of information available about the effects of health interventions. Several students mentioned that the IHC lessons improved their confidence to pause and reflect before deciding what to believe. A student reported that she used what she learned to help her cope with conflicting advice about health actions.

You find people telling you different things, for example, someone tells you use lemon and another person tells you why don’t you use Colgate [toothpaste], another person tells me use avocado because my sister uses avocado, so you find yourself getting stressed but after going through these be smart [IHC] lessons, I realized that I am not supposed to get so much stressed on what people tell you to use. I will find something that works for me by using the things [principles] we have learnt. —Student, FGD

It was evident from the lesson observation notes, the teachers’ lesson evaluation forms, interviews, and FGDs that this content is not taught in the curriculum, a finding that is consistent with results from our context analysis.¹⁶ Both the students and teachers appeared to appreciate the value of this additional educational content, expressing their inquisitiveness and a need to know how to teach and learn critical thinking skills.

When it was just introduced, I thought that I knew everything because we youth always pretend to know more. Teacher had just introduced health. I was like I know my health; I will just attend but without writing. Then [we] reached a moment when he introduced claims, and they were divided into 2: reliable and unreliable. I realized I know nothing. —Student, FGD

Learning Experiences Related to Individual Principles

As part of the intervention (Supplement 1 and Supplement 6), we introduced health actions (things that people do to care for their health or the health of others) and health claims (something that someone says as if it is true, but it may be wrong, about what will happen if you take a certain health action). Students listed many claims they had heard on the television, radio, online, on social media, or from friends or family. They also expressed that, before the IHC lessons, they had difficulties in deciding what to believe or do and that they were stressed by the overabundance of health information.

The 2 principles mentioned most often by students and teachers were: (1) the importance of weighing the advantages and disadvantages of health actions when deciding what to do, and (2) that personal experiences alone are a weak basis for claims about the effects of health actions. Teachers observed that students would highlight these 2 principles in other classes, and students reported these concepts in FGDs.

I learnt [from the IHC lessons] not to do things just because others are doing them. For example, if [student003] tells me to use something because it worked for him, I would remember that just because it worked for him does not mean it will work the same for me. I will also think, what if it harms me instead of helping me? —Student, FGD

The IHC lessons included 3 principles that intended to help people identify weak bases for health claims. These included claims based on personal experiences alone, on something being new or expensive, or on something being widely used or used for a long time. In both lesson observations and FGDs with students, several students explained how they applied these principles to claims that they had heard.

For example, a health claim that taking lemon juice cures flu – is it reliable or non-reliable? Then you find out it’s non-reliable because it has a missing basis. Yet, I thought it was reliable [for others too] because I have tried it out several times and it worked for me. Later, I found out that because it worked for me, it might not work for my friend, which is a weak basis. [For] a claim to be reliable, [it] must have a strong basis like be approved by health practitioners and with research by scientists through making randomized trials. —Student, FGD

At home, our paternal family, most of them are doctors, so whenever there was a strong disease and then they call the doctor, they used to tell us to buy only expensive drugs, that expensive drugs are better. Through these [IHC] lessons, I realized that even cheap drugs can work. —Student, FGD

The lessons were intended to help people question the basis of claims, recognize untrustworthy claims, and decide which treatment claims to believe and what to do. Several students applied the learned concepts to COVID-19 health interventions.

My parents support the opposition [political party] so they said that when you get that [COVID-19] vaccination, you won’t be able to elect in the year 2026 [national presidential elections] because you will be dead. And I replied that, you mean the president does not want his supporters [to be alive]. I also asked, have you seen anyone dying because of this vaccine? Then they answered yes, and I asked again how sure are you it was the vaccine and are they more than those that vaccine has saved [protected]? They said they didn’t know, and I told them that I checked and the number of those cured [protected] is many. Then they said we are going but are afraid …They went and came back. A month passed, and they were fine. —Student, FGD

The resources also taught people to be cautious of small studies, that the people being compared should be similar, and that random allocation was the best way to ensure this. Some students demonstrated they were able to recognize fair and unfair treatment comparisons when evidence was used to support a treatment claim.

For unreliable claims, just because something is new, costly/expensive or is being used by many people doesn’t mean that it is reliable. In order for one to know that it is a reliable claim, health researchers have to first carry out research on 2 groups that are large enough and compare them… researchers should take part in creating these groups so they should be created randomly. —Student, FGD

We have to think about the claims we make or hear. Like in holidays, I met a man who was selling herbal medicine like emumba [herbs mixed with clay] saying they prevent cancer. Now, if I was not studying “Be smart” [IHC] lessons, because I learnt to look at the basis of the claim, I would have bought them. I had to deny them because he could not explain very well how those herbs could cure cancer and how he knew without comparing to other medicines. —Student, FGD

Another principle that was taught in the intervention was for people not to assume that treatments were safe. A student reported how this helped decide about a treatment for pimples.

My friends told me that, there was a new tube which cures pimples, and I remembered what the teacher told us while he taught critical thinking, that even though a new or expensive medicine/drug comes on the market you don’t have to just buy it, you have to first get some information that shows [it] is better and also know whether this thing can cure or cause you more harm on your face, and I decided to leave that medicine. —Student, FGD

Other Perceived Benefits of the Intervention

Students, teachers, and parents reported several benefits from the IHC intervention. Students reported increased interest in learning, especially in relation to science, technology, engineering, and math subjects (moderate confidence). In addition, several students mentioned that they now aspire to become doctors or health researchers. Some teachers noticed a difference in their students’ desire to learn even in other subjects. Some teachers mentioned that the teaching strategies used in the resources improved their appreciation of the proposed strategies in the new curriculum.

They [fellow teachers] said that these learners nowadays have more interest towards sciences and have developed a positive attitude towards learning. We have been looking for ways to encourage these learners … so these [IHC] lessons can motivate these learners. —Teacher, public school, KII

Ever since I started teaching these lessons [to these learners], their interest in biology increased. Some of them started saying that they wanted to be doctors, their attitude improved. —Teacher, private school, KII

This term our children have changed, especially mine. When I look at her … you can see she is very interested in studying. She said that this term they have exposed them to many things which she may not have seen. —Parent, FGD

You ask yourself when am I becoming a doctor because if you know biology or chemistry it helps you to be a doctor and I got it [interest] from these [IHC] lessons. After learning this [IHC concepts], I am feeling compelled to become one because a number of people come to our communities deceiving/defrauding people with the wrong medicine. —Student, FGD

Students showed improved confidence in expressing their ideas to their peers, teachers, and parents. Teachers and parents observed some changes in student behavior that they attributed to the lessons. For example, teachers reported seeing increased engagement and reasoning in a group of students who were usually quiet before the IHC lessons. One of the parents mentioned that she saw the student in a drama skit leading scene, which she had never seen the child do before, and attributed this confidence to the IHC lessons.

The principles included in the lessons are applicable in other fields besides health, such as education, agriculture, and people’s social lives. Students were also able to apply what they had learned in different contexts, for example, balancing advantages and disadvantages.

There is a time we come to do health practices. For instance, I engage in sweeping this room. Yes, I am cleaning the room, but you have to look into the importance [advantages] and disadvantages because after cleaning, yes, the room will be clean, but I might end up with [a stuffy nose] from the dust as I sweep. So, you may find out that the disadvantages are more than the advantages. At times, because it is compulsory, we just take a risk and say let me just sweep the [room]. —Student, FGD

Potential Undesirable Effects of the Intervention

We reported the details of potential adverse outcomes associated with this intervention based on participants’ own perceptions and our observations in a separate qualitative evidence synthesis of the 3 trials.²⁸ The most common potential undesirable effects we found were partial understanding and misunderstanding of the concepts.

Perceived Value

Nearly all the students, teachers, and parents in FGDs and interviews could relate to the problem of overabundance of information about the effects of health interventions. They found it difficult to decide what to believe and do. Consequently, students, teachers, and policymakers recognized the importance of the lessons and valued them. The examples, content, and objectives of the lessons motivated the students to attend and actively participate in the lessons.

Since it [IHC lessons] had common examples, like the things they have heard of or things they have seen in [their] community but don’t know what is behind them like the adverts on TV, someone coming up and telling you my product is the best for you. Even the examples [in the IHC lessons] were day-to-day things that we use, and sometime we believed them. —Teacher, KII

In other lessons you may find someone dozing, but in those [IHC] lessons, nobody was dozing. Everybody was active and even members from other streams were coming to attend with us … We are in a new curriculum that involves critical thinking, yet even us, we wanted to know that critical thinking in the new curriculum … when we heard that there was critical thinking in these [IHC] lessons, I was so motivated to go for these lessons. —Student, FGD

My daughter told me about a new topic, where they were teaching them how to think … this excited her because they were given a chance to speak their mind … for example, she would tell us what she learnt like deciding whether to believe that new medicine is better than the old medicine. —Parent, FGD

Teacher Training

Another factor that might have positively influenced the intervention implementation was the teacher training workshop. All teachers reported in the workshop evaluation forms that they felt either confident or very confident teaching the IHC lessons after the 2-day teacher training workshop. However, in the interviews conducted after teaching the IHC lessons, some teachers reported feeling more confident teaching lessons about claims than those about research-related principles. Nonetheless, some teachers noted that the flow of the lessons made it easy for them and their students to understand the 2 lessons about assessing the trustworthiness of research evaluating the effects of health interventions.

I felt confident to teach lessons on personal choices, community choices and even health actions and claims. Mainly those first lessons, and last. Also, the content was just logically sequenced, so that once a student understands from the [beginning] of the discussion, the other information is easy to follow. —Teacher, KII

Flow of Lessons

Each lesson included a review of the previous lesson. Although several teachers found the review section good for the continuity in learning, a few teachers mentioned that if a student missed more than 1 lesson, they might not easily follow and comprehend the content of the lessons they missed.

There is that logical flow of the [IHC] lessons. For example, I review the previous lesson content in every new class. Although, I see that if a student has missed more than 1 lesson in between, they find it difficult to follow. —Teacher, KII

Absent From School Schedules, Curriculum, and National Exams

Several factors that may have hindered the impact and may also impede scaling up the intervention include the lack of allocated time to deliver the intervention and the intervention not being in the curriculum nor in the exams. The lessons were generally taught in place of students’ independent study time. Students found this inconvenient, especially when they learned that students in other streams and schools did not have these lessons.

Some students did not attend class when they learned that the IHC lessons were not taught in other streams and were not covered in the national exams. Even with the introduction of a competency-based curriculum in which a focus on examinations was discouraged, we found in the lesson observations, lesson evaluation forms, and interviews with teachers and students that teaching and learning still focused on exams. Teachers and students were less likely to spend time on lessons that were not included in the curriculum and exams, especially if these lessons took time away from subjects that were. We also observed that a few teachers did not seem to be well prepared when delivering the lessons, which may have affected the impact of the intervention. National examinations continue to be a key determinant of what students choose to study, what teachers choose to teach, and what school administrators choose to prioritize.

My group members never liked the [first 2–3] lessons. At first, I was also among those who never liked the lessons, but the teacher kept going on [teaching]. I started liking the lesson but for them they never wanted it, they used to say, you are showing off, the things you are doing are not going to be set in UNEB [national exams] and I thought about leaving [the class], but I had to be bold and stay. —Student, FGD

I think again right now the exams are still important because they [student] use them to change schools but it is very clear in the new curriculum that we are emphasizing acquisition of generic skills. —Policymaker, KII

Explanations in Local Languages

Students with low English proficiency reported that teachers and fellow students sometimes explained lesson content (e.g., definitions and instructions) in the local languages to improve understanding. This may have facilitated the impact of the lessons. Many students with low English proficiency attend under-resourced schools in rural areas. Thus, this strategy could potentially help to reduce inequities both within and between schools if the intervention is scaled up.

The teacher came and told us about health claims. These are claims about health actions. It was new to me health claims. When my friend explained in Luganda [the local language], I realized I have been coming across those things but when I don’t know what health claims are. I got to know that health claims can be reliable and unreliable. That the reliable ones are those ones that can be trusted, and the non-reliable claims are those ones which you can’t rely on. I got to know that the claim has 3 parts, the basis, and claim and health action. —Student, FGD

Easy to Access in Familiar Format

Teachers found the IHC digital lessons easy to access and presented in a familiar format to that in the new curriculum. This familiarity may have facilitated delivery of the lessons and their impact.

Potential Impact of Printable Materials on Effective Delivery

The IHC material included printable lesson summaries and posters, which may have increased the impact of the intervention. Most teachers that we interviewed preferred using their smartphones to access the lessons. However, we found that teachers still demonstrated a need for paper-based materials. For example, as mentioned earlier, 1 teacher requested printed lesson summaries upon losing his phone. We also observed 3 teachers using paper-based lesson summaries (printed at their respective schools) during the lessons, and 6 teachers made personal notes in a paper notebook to use as a back-up to their smartphones. A few teachers reported having to use personal notes because their phone’s screen froze during the lesson or they experienced a power outage during lessons taught using a projector. Furthermore, 3 teachers reported in interviews that they had printed lesson posters. Teachers and students reported finding the posters to be helpful when making personal notes and useful for students who had missed lessons.

Despite some teachers mentioning power outages during the school term, only 1 of the teachers reported that they had experienced challenges delivering lessons due to a power outage. Most mentioned the importance of adequate preparations, including ensuring fully charged phones before delivering the lessons and back-up hard copies of lesson summaries or personal notes.

Some teachers and policymakers expressed a desire for the entire set of printed resources to be provided (moderate confidence). They felt this would make it easier and quicker for both students and teachers to access and use the material. A few students also noted that like other subjects on the curriculum, the access to IHC printed resources in the school library would ease and improve their engagement with the resources.

Choice of School Term

The intervention may be better placed in a different school term. Some teachers suggested the first term of the year instead of the second or third term.

First term would be the best, because, in second term, we are so much engaged with the candidates as we are preparing for mocks [exams] and there is a fight to cover the whole syllabus by second term. So, it becomes a bit of a challenge to have enough spare time [for] this entire program. But, first term the pressure is not much, the teaching load is also not much. So, we recommend first term. —Teacher, KII

Relevance to Daily Life

Several students and teachers found the IHC lessons to be timely and beneficial, noting that the content related to their daily life experiences. This may motivate the students and teachers to continue engaging with this intervention and thus facilitate scaling it up.

The [IHC] project is good, it is hands on. It is highly informative, participatory in a way that it arouses students’ capacity to think, relate with health information and practices, connect with everyday life. And most important is if you went through this quality of teaching, if [teachers] really had this package [the intervention] very well, it would help both themselves and the student make informed decisions about the way they live. And that is what counts in their life … In a typical rural school, you could see some life in the students at that school as a result of this project. … I think it can touch many lives of people because we are looking at students getting out of school with something beyond passing exams. We want to produce a holistic child. —Policymaker, KII

Teaching Strategies That Facilitate Active Participation

During the activities section of the lessons, we observed that students were excited to participate in small groups and role-play. These teaching strategies and other elements of the intervention fit well with the new curriculum.

Clearly, there were a number of students who were able to think critically. That one could be seen. There was communication taking place, which is among our [new curriculum] competencies. There was cooperation among the small group discussions, some form of teamwork. Then, like I said, there is also what we call self-directed learning – that individuals are using their own knowledge. They are not just receiving information from the teacher. …This is the kind of teaching approach that we want, where the learners construct their learning, the teacher plays the facilitating role and, yes, it really reflects on what NCDC [National Curriculum Development Committee] is advocating for and the [IHC] lessons fit well with what we are hoping to see as lessons in lower secondary schools. —Curriculum specialist, KII

Perceived Credibility of Program Developers

The credibility of the institution associated with the intervention and how schools were approached may also have contributed to a positive reception and could potentially facilitate scaling up the intervention. Teachers at the training workshop reported that the presence of a government official from the curriculum office made it feel like a typical teacher training workshop that the government offers to introduce resources in the new curriculum. The credibility of Makerere University, letters of approval from the education ministry and district offices, and school consent by head teachers also contributed to teachers trusting the project and its objectives. Most students and teachers reported that they became interested when they heard that the educational resources were designed by researchers from Makerere University.

The factors influencing delivery and scale-up of the intervention are summarized in a logic model in Table 2. We highlight the intervention, immediate effects during implementation, and short- and medium-term impacts.