mLearning in the Democratic Republic of the Congo: A Mixed-Methods Feasibility and Pilot Cluster Randomized Trial Using the Safe Delivery App

Nancy E. Bolan; Larry Sthreshley; Bernard Ngoy; Faustin Ledy; Mano Ntayingi; Davis Makasy; Marie-Claude Mbuyi; Gisele Lowa; Lynne Nemeth; Susan Newman

doi:10.9745/GHSP-D-18-00275

Figures & Tables

Figures

Tables
Additional Files

Download figure
Open in new tab
Download powerpoint
FIGURE
CONSORT Flow Diagram
Abbreviations: CONSORT, Consolidated Standards of Reporting Trials; MD, medical doctor.

View popup

TABLE 1.

The Theoretical Domains Framework With Definitions and Component Constructs³⁰

Domain	Construct	Domain	Construct
1. Knowledge (an awareness of the existence of something)	Knowledge (including knowledge of condition/scientific rationale) Procedural knowledge Knowledge of task environment	8. Intentions (a conscious decision to perform a behavior or a resolve to act in a certain way)	Stability of intentions Stages of change model Transtheoretical model and stages of change
2. Skills (an ability or proficiency acquired through practice)	Skills Skill development Competence Ability Interpersonal skills Practice Skill assessment	9. Goals (mental representations of outcomes or end states that an individual wants to achieve)	Goals (distal/proximal) Goal priority Goal/target setting Goals (autonomous/controlled) Action planning Implementation intention
3. Social/professional role and identity (a coherent set of behaviors and displayed personal qualities of an individual in a social or work setting)	Professional identity Professional role Social identity Identity Professional boundaries Professional confidence Group identity Leadership Organizational commitment	10. Memory, attention, and decision processes (the ability to retain information, focus selectively on aspects of the environment, and choose between 2 or more alternatives)	Memory Attention Attention control Decision making Cognitive overload/tiredness
4. Beliefs about capabilities (acceptance of the truth, reality, or validity about an ability, or talent that a person can put to constructive use)	Self-confidence Perceived competence Self-efficacy Perceived behavioral control Beliefs Self-esteem Empowerment Professional confidence	11. Environmental context and resources (any circumstance of a person's situation or environment that discourages or encourages the development of skills and abilities, independence, social competence, and adaptive behavior)	Environmental stressors Resources/material resources Organizational culture/climate Salient events/critical incidents Person–environment interaction Barriers and facilitators
5. Optimism (the confidence that things will happen for the best or that desired goals will be attained)	Optimism Pessimism Unrealistic optimism Identity	12. Social influences (those interpersonal processes that can cause individuals to change their thoughts, feelings, or behaviors)	Social pressure Social norms Group conformity Social comparisons Group norms Social support Power Intergroup conflict Alienation Group identity Modeling
6. Beliefs about consequences (acceptance of the truth, reality, or validity about outcomes of a behavior in a given situation)	Beliefs Outcome expectancies Characteristics of outcome expectancies Anticipated regret Consequents	13. Emotion (a complex reaction pattern, involving experiential, behavioral, and physiological elements, by which the individual attempts to deal with a personally significant matter/event)	Fear Anxiety Affect Stress Depression Positive/negative affect Burn-out
7. Reinforcement (increasing the probability of a response by arranging a dependent relationship, or contingency, between the response and a given stimulus)	Rewards (proximal/distal, valued/not valued, probable/improbable) Incentives Punishment Consequents Reinforcement Contingencies Sanctions	14. Behavioral regulation (anything aimed at managing or changing objectively observed or measured actions)	Self-monitoring Breaking habit Action planning

View popup

TABLE 2.

Demographic Characteristics of Study Groups^a

	Intervention	Control
	(n=32)	(n=30)
Age, years, mean	41.3	44.2
Gender, No. (%)
Male	6 (18.8)	19 (63.3)
Female	26 (81.3)	11 (36.7)
Professional cadre, No. (%)
Nurses	22 (68.8)	14 (46.6)
Midwives	4 (12.5)	6 (20.0)
MDs	6 (18.8)	10 (33.3)
Years of experience in profession, No. (%)
1–5	3 (9.4)	6 (20.0)
6–10	12 (37.5)	12 (40.0)
>10	17 (53.1)	12 (40.0)
No. of deliveries past month, No. (%)
0–5	5 (15.6)	13 (43.3)
6–10	6 (18.8)	7 (23.3)
>10	21 (65.6)	10 (33.3)
Experience with smartphone, No. (%)
Tried using one	14 (43.8)	9 (30.0)
Never tried using one	18 (56.3)	21 (70.0)

Abbreviation: MD, medical doctor.
↵a The intervention and control group each comprised 4 health care facilities.

View popup

TABLE 3.

Within- and Between-Subject Differences in Mean Knowledge and Self-Confidence Scores Pre-and-Post Intervention

	Intervention	Control	Difference Between Intervention and Control	95% CI	P Value	Cohen's d
	(n=32)	(n=30)	Difference Between Intervention and Control	95% CI	P Value	Cohen's d
PPH knowledge scores (out of 100)
Pre, mean (SD)	47.8 (16.8)	47.5 (14.7)	0.3	(−7.8, 8.3)	.95
Post, mean (SD)	66.7 (14.8)	49.1 (15.6)
Pre-post difference, mean (SD)	18.9 (14.6)	1.6 (11.4)	17.4	(10.7, 24.0)	<.001	1.6
95% CI for pre-post difference	(13.7, 24.2)	(−2.6, 5.8)
P value for pre-post difference	<.001	.46
Cohen's d for pre-post difference	1.2
NR knowledge scores (out of 100)
Pre, mean (SD)	40.8 (17.5)	50.9 (16.6)	−10.1	(−18.7, −1.4)	.02	-0.6
Post, mean (SD)	57.7 (15.3)	48.3 (17.3)
Pre-post difference, mean (SD)	16.8 (14.6)	−2.5 (16.9)	19.4	(11.4, 27.4)	<.001	1.2
95% CI for pre-post difference	(11.6, 22.1)	(−3.8, 3.8)
P value for pre-post difference	<.001	.42
Cohen's d for pre-post difference	1.0
BEmONC self-confidence scores (out of 48)
Pre, mean (SD)	30.3 (8.7)	31.4 (10.8)	−1.1	(−6.1, 3.9)	.66
Post, mean (SD)	34.0 (8.9)	30.9 (8.7)
Pre-post difference, mean (SD)	3.8 (6.6)	−0.4 (7.2)	4.2	(0.7, 7.7)	.02	0.6
95% CI for pre-post difference	(1.4, 6.2)	(−8.6, 2.3)
P value for pre-post difference	.003	.74
Cohen's d for pre-post difference	0.4

Abbreviations: BEmONC, basic emergency obstetric and newborn care; CI, confidence interval; NR, neonatal resuscitation; PPH, postpartum hemorrhage, SD, standard deviation.

View popup

TABLE 4.

Differences in BEmONC Self-Confidence Scores (on a Scale of 4 Points) Among the Intervention Group, Pre- and Post-Intervention

BEmONC Procedures	Pre-	Post-	Pre-Post Difference	P Value
BEmONC Procedures	Mean (SD)	Mean (SD)	Pre-Post Difference	P Value
MVA	2.1 (1.3)	3.0 (1.2)	0.88	.003
D&C	2.7 (1.2)	2.6 (1.3)	−0.03	.90
Preeclampsia/eclampsia	2.0 (1.2)	2.4 (1.4)	0.41	.03
AMTSL	3.5 (0.7)	3.7 (0.5)	0.22	.09
Prolonged labor	2.7 (1.1)	3.1 (0.8)	0.47	.007
Vacuum extraction	1.7 (1.4)	1.8 (1.3)	0.19	.45
PPH	2.6 (1.2)	3.1 (0.9)	0.50	.03
Manual placental removal	2.8 (1.0)	3.1 (1.1)	0.31	.04
Septicemia	2.2 (1.2)	2.5 (1.3)	0.34	.08
NR	3.0 (0.8)	3.0 (0.9)	0.00	1.00
Danger signs in newborns	2.6 (1.0)	2.9 (1.0)	0.31	.11
Severe infection of newborn	2.4 (1.1)	2.6 (1.0)	0.19	.31

Abbreviations: AMTSL, active management of the third stage of labor; D&C, dilation and curettage; MVA, manual vacuum aspiration; NR, neonatal resuscitation; PPH, postpartum hemorrhage.

View popup

TABLE 5.

Within- and Between-Subject Differences in Knowledge Scores Pre- and Post-Intervention (Among Intervention and Control Groups Combined), Analyzed by Gender and Smartphone Experience

	Men (n=25)	Women (n=37)	Difference Between Men and Women	95% CI	P Value	Cohen's d	Never Used Smartphone (n=39)	Experienced Smartphone (n=23)	Difference Between Smartphone Experience	95% CI	P Value	Cohen's d
PPH knowledge scores (out of 100)
Pre, mean (SD)	53.9 (13.8)	43.4 (15.6)	10.5	(2.8, 18.3)	.008	0.72	43.0 (14.5)	55.6 (14.6)	12.6	(5.0, 20.3)	.002	.87
Post, mean (SD)	59.6 (15.3)	57.2 (18.9)	2.5	(−6.6, 11.6)	.588		53.8 (17.2)	65.6 (15.7)	11.8	(3.0, 20.5)	.009	.72
Pre-post difference, mean (SD)	5.7 (11.8)	13.8 (17.3)	8.1	(0.1, 16.0)	.046	0.56	10.9 (17.8)	10.0 (11.6)	0.85	(−7.5, 9.2)	.84
95% CI for pre-post difference	(0.9, 10.6)	(8.0, 19.5)					(5.1, 16.6)	(5.0, 15.0)
P value for pre-post difference	.02	<.001					.001	<.001
Cohen's d for pre-post difference	0.37	0.80					0.69	0.66
NR knowledge scores (out of 100)
Pre, mean (SD)	53.6 (15.9)	40.3 (16.9)	13.3	(4.8, 21.9)	.003	0.81	43.9 (17.5)	48.7 (18.0)	4.7	(−4.6, 14.0)	.31
Post, mean (SD)	54.4 (14.8)	52.3 (18.3)	2.1	(−6.7, 10.9)	.634		51.5 (17.8)	55.9 (15.0)	4.3	(−4.5, 13.2)	.33
Pre-post difference, mean (SD)	0.8 (15.7)	12.0 (19.0)	11.2	(2.1, 20.4)	.02	0.66	7.6 (19.4)	7.2 (17.0)	.40	(−9.4, 10.2)	.94
95% CI for pre-post difference	(−5.7, 7.2)	(5.6, 18.3)					(1.3, 13.9)	(−0.1, 14.6)
P value for pre-post difference	.81	<.001					.02	.05
Cohen's d for pre-post difference		0.68					0.43	0.44

Abbreviations: CI, confidence interval; NR, neonatal resuscitation; PPH, postpartum hemorrhage; SD, standard deviation.

View popup

TABLE 6.

Knowledge Scores Pre- and Post-Intervention (Among Intervention and Control Groups Combined), Analyzed by Health Worker Cadre

	Scores, Mean (SD)				F	P Value	P Value for Paired Comparisons^a
	Nurses (n=36)	Midwives (n=10)	MDs (n=16)	All Cadres (N=62)	Nurses–Midwives	Nurses–MDs	Midwives–MDs
Pre-tests
PPH	44.4 (16.8)	48.8 (14.1)	54.1 (12.3)	47.6 (15.7)	2.197	.12	—	—	—
NR	41.1 (18.5)	47.5 (11.8)	54.9 (15.5)	45.7 (17.7)	3.747	.03	—	.02	—
Post-tests
PPH	56.9 (17.6)	53.2 (17.5)	64.1 (16.7)	58.2 (17.5)	1.428	.25	—	—	—
NR	53.4 (15.1)	42.8 (21.2)	59.0 (15.6)	53.2 (16.9)	3.046	.055	—	—	.04

Abbreviations: MD, medical doctor; NR, neonatal resuscitation; PPH, postpartum hemorrhage; SD, standard deviation.
↵a Significant for Tukey's HSD (honestly significant test).

View popup

TABLE 7.

Qualitative Interview Results With SDA Users in Intervention Facilities (n=8)

Domains	Illustrative Quotes
Knowledge	“When we hear and see [the information] at the same time it teaches a lot.” “We did things blindly before with what we learned in school and it wasn't enough.”
Skills	“Training gave us the skills to use the app.”
Beliefs about capabilities	“We see that misoprostol is effective [for PPH].” “Had many PPH deaths before … PPH resolves if you use what is in the video.” “Now we see less fever in children after NR when we give antibiotics.” “With info in the video for NR, you see the newborn coming back. It's really encouraging.”
Reinforcement	“We watched videos about every 3 days during free time at maternity [ward] alone or with maternity team. Also used during a case of manual removal of placenta and MVA.” “One can re-watch the video as many times as one wants.”
Intentions	“It changed our old habits” “Now we take vital signs and use the partogram during delivery.” “Before for AMSTL we put the baby off to the side; now we put baby skin-to-skin and encourage breastfeeding.” “We aspirated all babies; now we only aspirate when we need to.” “Before we held the baby upside down after delivery and gave mouth-to-mouth brutally if needed; now we use the Ambu bag, which gives a good result. We learned that we must position the baby and the mask in order to do NR.” “Before for respiratory distress we did mouth to mouth and gave hydrocortisone IM, no antibiotics, and saw high rate of fever. Now, we give antibiotics, and we see less fever.” “Before we didn't do uterine massage or use misoprostol or IV fluids for PPH management; now we use massage, misoprostol, and IV fluids … with good results.” “Now we do uterine massage [with PPH] and use a urinary catheter, and we see the uterus contracts.” “Before we pushed the uterus down during 3rd stage; now we support the uterus and use controlled traction on the cord.” “Now with premature rupture of membranes we give antibiotics.”
Memory, attention, and decision processes	“mLearning with the app is good, the learner sees the information, hears it and then can do it themselves. It helps participants to remember the visual images or auditory information.”
Environmental context and resources	“Only 2 of us were trained back in 2012 but need others to be trained, and we need formative supervision more often.” “Need uniforms, tops, shoes, eye protective equipment, aprons, soap. We work in our own clothes and shoes, we risk to contaminate our children.” “We earn nothing–12,700 Francs per month. Put yourself in our place. We work hard for nothing.”
Emotion	“It's amusing and relaxing. It's good for educating adults. There is variety.” “App should be made more widely available–in pediatrics and the operating room.” “Animated graphics were interesting. Other trainings, they talk and talk.” “We are very happy with the intervention. It's very encouraging.”

Abbreviations: AMSTL, active management of the third stage of labor; IM, intramuscular; IV, intravenous; MVA, manual vacuum aspiration; PPH, postpartum hemorrhage; NR, neonatal resuscitation; SDA, Safe Delivery App.

View popup

TABLE 8.

Qualitative Interview Results With Key Stakeholders (n=10)

Domains	Illustrative Quotes
Knowledge	“Trainings are theoretical via lectures. They teach too many things at the same time and they rush through the material. Learners have trouble prioritizing and leave with confusion; meanwhile the essential notions aren't mastered and they aren't able to apply the knowledge to a case. Providers have been trained before but it's as if they have never been trained. It would be better to have more practical training.” “CE should incorporate adult learning principles, and it should be continuous/regular.” “Hospital/workers are often 'subjected' to trainings that they haven't planned … should be included in planning in response to priority gaps/needs.” “Must present things that are relevant to what they [health workers] do–where they get practical info and can see the gestures.” “SDA renders learning operational.”
Skills	“There is a weak development of competence of personnel.” “Care has become mechanized and based on memorized protocols, so they [health workers] have a difficult time analyzing situations.” “Providers were able to put into practice new things learned [from the SDA] such as using the side of the hand for manual extraction [of the placenta] and bimanual compression for PPH.” “Should be conducted in real work conditions to combat the gap between what one knows and what one does.” “mLearning also teaches people how to use technology.”
Social/professional role and identity	“[Health workers] use what they have to treat patients, but they are not going to look something up or get additional info.” “Must create a way for people to share information. Maybe form a club (to discuss with an animator/trainer). If the program is personalized, even better–get points, get certificate [must be linked to the employer].”
Beliefs about capabilities	“Imposing CE is only of limited value, when the boss isn't there … they won't do it.” “Inspiring them to the benefits of CE would be more motivating then sanctions.”
Beliefs about consequences	“No accountability related to malpractice.” “CE regulations/requirements are where the country needs to go to make a change.” “[Employers or] Ministry should have tracking capability related to CE to be able to identify persons in need of training and needed training.” “Must assure career path for health workers based on regular evaluation.” “Must lead to a change in employment status/have a concrete change.”
Reinforcement	“mLearning is interesting to reinforce learning since the tool is available at all times and the provider can view the information many times.” “Ideal if someone [trainer] follows the process to enrich the application [give retro-information and ensure that needs are covered in the app], and to answer questions.” “SDA should be made available in all health zones with post-training supervision … would decrease maternal and neonatal mortality.”
Intentions	“For remote settings, eLearning is interesting–self-directed learning, and should measure if they acquire competencies/can be certified, and put in place a system for them to be encouraged to do this.”
Memory, attention, and decision processes	“The images allow people to learn with more stimulation and attention. The visual memory can fix the memory of the information for longer.” “When time passes after a training, you forget.” “If people are motivated they may retain info better.”
Environmental context and resources	“Better working conditions [salaries, supervision, environment] would push people.” “Poor distribution of opportunity to get CE training. Limited training for hard-to-reach areas and for lower health worker cadres (A3, A2).” “For CE, programs/partners decide subject, may repeat subjects already covered, same people always go to the trainings and are missing from work.” “No clear CE policy. Who chooses the subjects? And must be defined who participates and what one gets from it. Needs a clear policy, training should be programmed and budgeted … so that employers recognize the training and has it planned/budgeted for their facility.” “eLearning can train more people at lower cost.”
Emotion	“Seeing the images is more interesting than just talking theoretically.” “Approaches must change from written info to interactive self-learning options.” “New technology fascinates/attracts people and they want to try it; it responds to a need.”