Implementation Research for the Evaluation of the Child Health Education and Surveillance Tool Application: A Randomized-Controlled Trial

doi:10.21203/rs.3.rs-995871/v1

Download PDF

Research

Implementation Research for the Evaluation of the Child Health Education and Surveillance Tool Application: A Randomized-Controlled Trial

https://doi.org/10.21203/rs.3.rs-995871/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

When community health workers (CHWs) are effective, they can help teach essential knowledge of healthy child rearing practices in their communities. An effective mHealth tool can potentially improve the performance of CHWs and create greater child health and development outcomes. The Child Health Education and Surveillance Tool (CHEST App) was developed and implemented in a CHW program in the Amazon of Peru to improve CHW performance and early childhood development. The intervention was developed and implemented with the guidance of several implementation research tools. This study presents the results of the intervention

METHODS

The CHEST App intervention was evaluated using an effectiveness-implementation hybrid type 2 study design. A group of communities (8) in Loreto, Perú were matched by propensity scores and then randomly assigned to an arm of the study. The implementation outcomes were evaluated with a mixed-methods approach to how well the intervention was installed into the local CHW program. The effectiveness of the intervention was determined by assessing knowledge scores, hemoglobin gm/dL, early childhood development motor scores and cognitive scores. An independent samples t-test and one way ANOVA analysis were conducted with each intervention outcome to assess differences between the intervention and control groups. The results of the study are presented within the framework of the Implementation Research Logic Model.

RESULTS

The COVID-19 pandemic created significant interruptions in the intervention, significantly decreasing the dosage of the intervention received by the participants. However, the group of caregivers that received home visits with the CHEST App had significantly higher knowledge scores (+1.26 standard deviations) than those in the control group (t(184)=-4.39, p<0.001). The intervention group did not display significant differences in hemoglobin levels and early childhood development scores. Implementation of the CHEST App intervention was successful with high degrees of acceptability, adoption, and fidelity. However, adoption and fidelity of the surveillance function of the CHEST App by program coordinators was not achieved.

CONCLUSION

The CHEST App intervention is a promising tool to improve the performance of CHWs during their home visits and their ability to teach caregivers healthy child rearing practices.

Trial Registration

Trial registered on 11/29/2018 at https://doi.org/10.1186/ISRCTN43591826.

Maternal & Fetal Medicine

Technical Communication

Implementation science

implementation research

mhealth

early childhood development

amazon

hybrid type II

anemia

Peru

community health

community health workers

The current study is exemplary for how implementation research methods and tools can be used to design, evaluate, and report a pilot study, to improve learning and replicability.
The research presented here provides important generalizable knowledge about the feasibility and value of providing community health workers a digital tool to improve their capacity to teach caregivers healthy childrearing practices.
Animated videos, automated message selection, and the novelty of an electronic device are associated with increase knowledge acquisition by caregivers in rural Amazonian communities.

Children around the world continue to suffer from ailments such as malnutrition and developmental delay due to unhealthy practices in the household.^1–4 A large portion of the ailments could be alleviated by the adoption of better sanitation practices, diet, and disease prevention, however, caregivers’ knowledge of healthy practices can be limited due to poor education and health promotion efforts in impoverished settings.^5–9 Effective transmission of knowledge of healthy practices has been difficult to achieve at scale in rural communities, but when accomplished it can substantially improve child health and development.¹⁰ The utilization of community health workers (CHWs) to conduct health promotion and education in their communities has been shown to be effective, though outcomes vary greatly.^11–14 Effective, consistent, and scalable implementation of CHWs programs have been elusive.^14,15 The utilization of information and communication technology (ICT) and mHealth technology have also been shown to be effective at improving health behavior in the household.^16–24 However, effective implementation of ICT tools in rural settings has also been varied, and most populations in impoverished settings have not received the benefits of knowledge transmission through modern technology.²⁵

Ineffective implementation of evidence-based interventions, such as CHW programs and ICT tools for behavior change, is partly due to an imprecise understanding of what has been done, what has worked, and what has failed. The implementation of innovations needs to be mapped, evaluated, and reported with sufficient detail to support continuous learning and improvement.^26–28 The current study reports on the implementation and impact of an mHealth innovation to improve CHW performance in the Amazon of Peru. The implementation research was conducted to generate data on the success and failures of the implementation strategies to install the innovation and the feasibility of the intervention to function effectively in the rural setting. The complete study protocol is described in the article, Westgard, et. al., 2018.²⁹ The specific stages of implementation (based on the Active Implementation Frameworks^30,31) and implementation strategies that were used to install the intervention are described in the article, Westgard, et. al., 2020.³²

The intervention was implemented in the northern Amazon region of Peru, in the department of Loreto. In Loreto, 57% of children under 3 have anemia, 20% under 5 have chronic malnutrition, and 26.7% of children are developmentally delayed.^33,34 Caregivers’ knowledge of healthy childrearing practices are limited as they transition from traditional practices to modern medicine.^{5–7, 9} In many rural communities in Peru, CHWs conduct home visits with new mothers to teach them health topics such as sanitation, diet, disease prevention, and early childhood development. However, the impact of the CHW programs has been hindered by poor performance of CHWs and a lack of effective educational material. The CHWs are often unequipped and under-trained to successfully transmit the information to the caregivers and convince them the importance of the behavior change.^15,35,36 To address the underperformance, the research team at the Peruvian research organization, Elementos, developed a tablet-based application and animated videos to support the CHWs in their task to teach caregivers key health messages and collect child health indicators during home visits. The innovative tool is titled, The Child Health Education and Surveillance Tool Application (The CHEST App). A video of the CHEST App and the animated videos that accompany the App can be viewed online.³⁷ The primary objectives of the CHEST App intervention are to improve early childhood development (ECD) scores, reduce anemia rates, and reduce chronic malnutrition rates. The theory of change of the intervention is displayed in Appendix 1 and the Conceptual Model of the intervention is displayed in Appendix 2.

The pilot study was interrupted by the COVID-19 pandemic, causing CHW program shutdowns and thus a significant loss of program participants and sample size. After 6 months of shutdowns, one community in the intervention group and one community in the control group reactivated their CHW programs. The two communities were matched with propensity scores before assignment to the intervention and control study group.

Study Design

The implementation process and CHEST App intervention are evaluated using a Hybrid Type II evaluation study design.³⁸ In a Hybrid Type 2 study, the implementation process is evaluated alongside the intervention (the CHEST App) to determine what mechanisms are responsible for the changes associated with the intervention.³⁸

The study took place in the Amazon of Peru, department of Loreto. The study initially included 8 communities that were matched 1-1 with propensity score matching and then randomly assigned to the intervention or control group. Both communities are similar in size (8-10,000), access to health facility (Level I-2 in community), distance to province capital (1.5 hours in fast boat), rates of anemia (25-32%), and structure of CHW program. The study was reduced to one intervention group (Indiana) that continued to use the CHEST App after COVID and one control group (Tamshiyacu) pair that continued their CHA activities. The intervention group’s CHWs received the CHEST App and the control group’s CHWs did not. The CHW programs had to randomly select children in their communities to receive home visits when they were not able to visit all children aged 0-3 years. The intervention group included the children and caregivers that received a home visit with the CHEST App. The control group included all children in both communities that did not receive the CHEST App. Most of the participants at endline were not the same as the participants at baseline, although they lived in the same community. Many of the children that participated in the baseline assessment aged out of the program by the endline assessment. Therefore, a ANCOVA analysis could not be used to assess impact.

A baseline survey was conducted prior to implementation of the intervention and an endline survey was conducted 16 months following implementation, in the intervention and control communities. The research team conducted a consensus in each community to survey all children ages 6-35 months, regardless of if they received home visits by the CHWs. A detailed description of the selection process and study population is included in the published study protocol.²⁹

Implementation Outcomes

The evaluation of the implementation of the intervention was conducted with a mixed methods approach to identify the extent to which the intervention was successfully adopted and incorporated into the local CHW program. The implementation process, including the barriers and adaptations, is described in the publication by Westgard, et. al., 2020.³² Reporting the implementation process was done to allow others (researchers and program implementers) to understand what activities took place during the implementation of the intervention that may have influenced the success or failure of the intervention.³⁹ The implementation research reported here follows the C.A.R.E. guidelines (Consolidated Advice on Reporting ECD Implementation Research) to ensure the necessary information is included in the evaluation.⁴⁰ An Implementation Research Logic Model (IR Logic Model) was used to design the evaluation process and specify the relationship between the determinants of implementation, implementation strategies, and the implementation, service, and clinical outcomes (Appendix 3).²⁶ The implementation outcomes were selected based on the conceptual framework presented by Proctor, et. al., 2011.⁴¹ The implementation outcomes serve as a precondition for attaining the desired changes in service outcomes and clinical outcomes. The implementation outcomes relevant to this study include, acceptability, adoption, fidelity, and cost. Other implementation outcomes were not measured because the intervention had not yet reached the point in its maturation to measure them with confidence, including, feasibility, penetration, and sustainability. An additional implementation outcome, dosage, was also measured.

Acceptability was measured to identify the extent to which the implementation stakeholders (caregivers and providers) perceive the intervention to be satisfactory.⁴¹ Both the CHWs and caregivers in the intervention group received a semi-structured interview to determine their level of acceptance of the CHEST App. The quantitative portion of the survey was analyzed and reported using descriptive statistics. The qualitative responses were analyzed by identifying a set of sub-themes from the responses then assessed for similarities and differences in perspectives of the participants. Key quotes that best reflect the position of each actor group are reported.

Adoption was measured to determine the uptake of the intervention and continued use by the providers throughout the study period. Adoption indicates that when the CHW realized a home visit, they used the CHEST App to do it. It was measured by analyzing data that was collected with the CHEST App and then uploaded to the server. The number of home visits conducted with the CHEST App was compared to the number of home visits the CHWs were reported to complete each month, to determine a percentage of adoption. Also, the CHWs received a semi-structured interview to provide them the opportunity to describe the extent to which they use the CHEST App during their home visits, and the components of the CHEST App they use with consistency. The caregivers in the intervention group also received a semi-structured interview to determine the number of home visits with an electronic tablet they have received. This indicator was used as the independent variable of the study. The responses from the CHWs and caregivers are reported with descriptive statistics. The program coordinator of the CHW program received a semi-structured survey pertaining to the adoption of the CHEST App and the use of each of its components. The response to the survey is summarized.

Dosage is a measure of the total number of home visits realized with the CHEST App during the study period. The program was expected to conduct home visits with at least 100 children and provide an average of 10 home visits with each child. Dosage is be measured to determine to what extent the intervention reached the recipients. It will be calculated by dividing the number of home visits conducted during the program by the expected number of home visits (1000) to be conducted.

Fidelity was measured to determine the degree to which the intervention was delivered as prescribed. The research team conducted observations of home visits with the CHEST App to determine if each component of the CHEST App was being delivered with quality and as intended in the intervention protocol. The observers had an observation checklist to mark the completion, or incompletion, of each component. The results of the checklist are reported as the measure of fidelity. The fidelity measure instrument (in Spanish) can be seen in Appendix 4.

Implementation cost is reported to provide the financial impact of the implementation process. The costs include equipment, training, and supervisory services. The costs are broken down to provide a per CHW cost and the cost to implement the intervention with 20 CHWs. The costs to development of the App and educational material are not included.

Intervention Outcomes

The effectiveness of the intervention was determined by measuring differences in the service outcome (intermediary) and clinical outcomes between the intervention and control group. The service outcome was the knowledge scores by caregivers. The clinical outcomes were the hemoglobin gm/dL levels and early childhood development (ECD) scores (motor and cognitive). To determine if the intervention and control groups were similar at baseline, an independent t-test analysis was conducted on each intervention outcome. To assess the impact of the CHEST App, two analyses were conducted on the outcome variables. First, an intention-to-treat analysis was conducted with the treatment variable coded as 1 or 0 (1= 1 or more home visits with the CHEST App, 0= no home visits with the CHEST App). An independent t-test was used to determine if the mean of the outcome variables (knowledge, hemoglobin, and ECD scores) are significantly different in the two groups (received CHEST App vs. did not receive CHEST App). The following null hypothesis was tested with the independent t-tests for each of the intervention outcome variables (H₀1.1: knowledge scores, H₀1.2: hemoglobin levels, H₀1.3: ECD motor scores, H₀1.4: ECD cognitive scores):

H ₀ 1: The outcome variable for those in the intervention group is significantly different then the outcome variable for those in the control group at endline.

Secondly, a dose-response relationship analysis was conducted with the treatment variable coded as three categories: 0 home visits with the CHEST App, 1-2 home visits with the CHEST App, 3+ home visits with the CHEST App. The relationship was tested to identify if an increase level of exposure to the intervention is associated with an increasing or decreasing risk of each outcome.⁴² A one-way analysis of variance (ANOVA) was conducted to determine if the mean of the outcome variables are different in the three groups (0, 1-2, or 3+ home visits). If the one-way ANOVA showed a statistically significant difference between groups than a Tukey post hoc test was conducted to determine where any differences lie. The following null hypothesis was tested by the one-way ANOVA for each of the intervention outcome variables (H₀2.1: knowledge scores, H₀2.2: hemoglobin levels, H₀2.3: ECD motor scores, H₀2.4: ECD cognitive scores):

H ₀ 2: The outcome variable for those in each of the three groups (0 home visits, 1-2 home visits, 3+ home visits) will be significantly different at endline.

The knowledge score represents the service outcome of the study, as shown in IR Logic Model (Appendix 3). The service outcome of the study reflects the effectiveness of the intervention to improve CHW performance.⁴¹ If CHWs are effective at delivering their intended service, the caregivers will have greater knowledge of healthy childrearing practices. Improved knowledge is expected to lead to improvements in the clinical outcomes (anemia and ECD scores).⁴³ Knowledge scores were measured by an opened-end questionnaire with caregivers, at baseline and endline. The questionnaire was designed to give the participant ample opportunity to describe what they know about each question. The surveyors were trained to ask the participant the survey question, then probe the participant to provide further information. For example, the participant was asked, what are the benefits of breast milk for a child, then follow-up with statements such as, “what other benefits” and “have you heard of any other benefits”. Probes continued until the participate indicated that they do not know any further information. For each correct answer, the participant received a point. The points were totaled to provide the knowledge score for the participant. The questionnaire included 15 questions with a total possible score of 91. The topics included, nutrition, sanitation and hygiene, disease prevention, and early childhood development. The knowledge questionnaire (in Spanish) can be seen in Appendix 5.

Hemoglobin levels were measured by local health officials independent of the pilot study. The hemoglobin analysis was conducted during the child´s growth monitoring check-up at various times of the year. The health official writes the result of the hemoglobin analysis on the child’s growth monitoring/vaccine card. During the baseline and endline surveys, the surveyor recorded the result of the last hemoglobin analysis written on the card.

ECD was evaluated with the Caregiver-Reported Early Development Instrument (CREDI), an instrument that utilizes caregiver-reported data to assess the ability of the child to perform age-related developmental activities.^44–49 The CREDI has been validated as an effective instrument to evaluate ECD in 17 low- and middle-income countries.⁴⁹ The CREDI is conducted by asking the caregiver if the child is able to complete a list of age-appropriate activities that become progressively more difficult. The instrument was used to assess motor development and cognitive development. As recommended by the CREDI manual, a raw score was calculated for each development measure using the statistical software, R⁵⁰, and the script package provided by the CREDI instrument. The raw scores were used as the ECD motor and ECD cognitive scores for the analysis.

A SPIRIT Checklist was completed to ensure the manuscript includes all research reporting components. The SPIRIT Checklist is included in Appendix 6.

Implementation Outcomes

The results of the implementation research can be seen in the IR Logic Model in Appendix 3. The evaluation of the implementation outcomes included 41 caregivers and 6 CHWs that received the intervention.

Acceptability by Caregivers

Most caregivers (92%) that received home visits with the CHEST App expressed that they prefer the App than use of traditional methods (pen, paper, and flipcharts). 5% expressed that they had no preference and 3% expressed that they prefer traditional methods of home visits.

When caregivers were asked if they learned the health messages delivered during the home visits better with the CHEST App or without the CHEST App, 84% expressed that they learned better with the CHEST App, 13% expressed that they learned the same with or without it, and 3% (n=1) expressed that they learned better with traditional methods. When asked what they liked most about receiving home visits with the CHEST App, all caregivers answered that they most enjoyed the educational component of the App. Many (43%) specifically mentioned the animated videos as a reason why they prefer the CHEST App to traditional methods. For example, a mother in the community of Indiana said,

“I like that they show use the videos and how to feed the children. You learn better because you can see how to do it through the animations”.

(Me gusta que nos hacen mirar los videos y la alimentación de los niños. Se aprende mejor porque se ven como se hace para aprender mediante dibujos)

Acceptability by CHWs

All CHWs expressed that they prefer to conduct the home visits with the CHEST App than their traditional methods (pen, paper, and flipcharts). When asked what they liked about using the App, the CHWs reported that they most liked learning from the educational material, the animated videos, and the child health indicators displayed in red or blue. The animated videos were the most cited reason that they liked the CHEST App and how it helped them better conduct their home visits. A CHW in the community of Indiana said,

Yes, it (the App) has everything summarized and is faster. They like the videos. They laugh and understand more quickly. The child points. They like it a lot.

(Si. Tiene todo resumido y mas rápido. Les gustan los videos. Se ríen. Aprenden más rápido y el niño apunta. Les gusta mucho)

The CHWs expressed that the most difficult part about using the CHEST App was sending the data and registering the information in the tablet.

Adoption by CHWs

The CHWs were registering health indicators of 140 children per month with the CHEST App, the same number of children they were assigned to visit. Thus, adoption was confirmed to be 100%., meaning the App was used by the CHWs during every scheduled home visit. After 18 months of use, the CHWs expressed that they continue to use the CHEST App during all their home visits. The caregivers verified adoption of the CHEST App by CHWs by reporting the number of home visits they have received, and number of home visits received with an electronic tablet. The results were similar (8.6 vs. 7.8 visits with tablet), indicating that the CHWs use the CHEST App during their home visits. All the CHWs reported that they use all the App’s functions when conducting a home visit (share health indicators with caregiver, register health indicators, scheduling, educational images, and animated videos).

Adoption of the CHEST App was hindered due to the cancelation of the CHW programs in several communities, both before COVID and during COVID. The instability of the local CHW programs makes adoption and sustainability of the CHEST App difficult to measure because without the infrastructure of the CHW program the CHEST App cannot be utilized.

Adoption by CHW Program Coordinators

CHW Program Coordinators were tasked to upload and utilize the data collected with the CHEST App. Adoption of this practice was not achieved. The CHW Program Coordinators were not interested in uploading the data to the server. The task was completed during the pilot study, but only to satisfy the external research team at Elementos. Therefore, the research team is not able to monitor the data from the tablet unless they visit the community and upload the data from the tablets to the server. The Program Coordinator and local municipality continue to desire a paper-based list of results of the surveillance data and did not have a printer readily available to print the data from the tablets. The CHW Program Coordinator wrote the data displayed in the tablet on paper and submitted the paper report to the municipality. Thus, reflecting low adoption and poor fidelity of the surveillance function of the CHEST App.

Dosage

A total of 51 children received home visits with the CHEST App. The CHWs conducted a total of 286 home visits while the program was operating. The program fell significantly short of reaching its dosage objective of 1000 home visits, at 5.7% of the objective.

The intervention group community continues to use the CHEST App in their program at the time of writing this manuscript, 26 months post implementation.

Fidelity

Fidelity of the CHEST App by the CHWs, as observed during home visits, remained high throughout the pilot. At 18 months, each intended activity associated the CHEST APP showed fidelity scores between 80-100%, as shown in Table 1. The score indicated a high degree of fidelity and high quality of home visits with the CHEST App.

Table 1

Fidelity scores for the CHEST App
Intended Activities with CHEST App	Fidelity Score (N=6)
Registered Health Indicators	80%
Used App to discuss child health indicators with caregiver	80%
Used content in App to explain health messages	100%
Explained health messages with sufficient information	100%
Asked caregiver what they understood from the health messages	100%
Showed animated video to caregiver	80%

Cost

The intervention was implemented at a scale of 20 CHWs in 4 communities. The cost of incorporating the CHEST App into the established CHW program included the following: providing an electronic tablet to each CHW ($120), training the CHWs on how to use the tablet ($184/CHW) and the cost of a Field Supervisor to visit the CHWs for continued training. The Field Supervisor can cover a greater number of CHWs (max 40) than listed, decreasing the cost per CHW. The cost of the Field Supervisor can also decrease after the first 2-3 months as less visits and attention are needed. The cost of App development and creation of the educational material is not included. The incremental implementation costs are displayed in Table 2.

Table 2

Implementation Cost of the CHEST App intervention
	USD	For 20 CHWs and 3 months of supervision
Electronic Tablet	$120 / CHW	$2,400
Training	$184 / CHW	$3,680
Field Supervisor	$900 / Month	$2,700
	Total	$8,780

Service Outcome

Knowledge Scores by Caregivers

The average knowledge scores by the caregivers, at baseline and endline, are shown in Table 3. The intervention and control group showed no significant difference in knowledge scores at baseline (p=0.9216). The independent samples t-test (intention to treat) found a significant increase in knowledge scores (M=5.53, SD=1.26) by those in the intervention group compared to those in the control group (t(184)=-4.39, p<0.001). The results of the independent t-test analysis can be seen in Table 4. A one way ANOVA showed that the effect of the CHEST App on knowledge scores yielded significant variation among groups, F(2,185)=12.9, p<0.000. The results of the one-way ANOVA analysis can be seen in Table 5. A post hoc Tukey test indicated that the average knowledge score for those that received 1-2 home visits (M=36.05, SD=6.11) was significantly higher than those that received no home visits (M=27.66, SD=7.38), p<0.000. The average knowledge score of those that received 3+ home visits (M=30.77, SD=8.57) was significantly different than those that received 1-2 home visits, p=0.040. The comparison between those that received no home visits and those that received 3+ home visits was not significant, p=0.125. The results of the post hoc Tukey test can be seen in Table 6.

Table 3

Mean Intervention Outcomes
	Indicators	Baseline	Endline
Intervention Group	Knowledge Scores	23.25	33.19
	Hemoglobin gm/dL	11.26	11.63
	ECD Motor Scores	47.44	50.85
	ECD Cognitive Scores	48.09	50.84
Control Group	Knowledge Scores	23.4	27.66
	Hemoglobin gm/dL	11.24	11.57
	ECD Motor Scores	49.61	50.24
	ECD Cognitive Scores	49.7	50.57

Table 4

Independent t-test analysis results to compare group means of intervention outcomes
Outcome	Intervention Group		Control Group		P value
Outcome	Sample size	Mean	Sample size	Mean	P value
Knowledge Scores	48	33.19	138	27.66	<0.000
Hemoglobin mg/dL	45	11.63	97	11.57	0.711
ECD Motor	45	50.85	97	50.24	0.111
ECD Cognitive	45	50.84	97	50.57	0.232

Table 5

One-way ANOVA results to compare group means of intervention outcomes
Outcome	No home visits	1-2 Home Visits	3+ Home Visits	Sum of Squares	Df	F	P value
Outcome	Mean (N)			Sum of Squares	Df	F	P value
Knowledge Scores	27.66 (138)	36.05 (22)	30.77 (26)	1420.06	2,183	12.9	<0.000
Hemoglobin mg/dL	11.57 (97)	11.94 (19)	11.41 (26)	3.26	2, 139	1.86	0.160
ECD Motor	50.24 (97)	50.88 (22)	50.83 (26)	10.27	2, 142	1.28	0.282
ECD Cognitive	50.57 (97)	50.93 (22)	50.78 (26)	2.32	2, 142	0.79	0.457

Table 6

Tukey Post hoc comparison of groups on Knowledge Scores.
Groups	Mean Difference	P value
No home visits vs 1-2 home visits	8.39	<0.000
No home visits vs 3+ home visits	3.11	0.125
1-2 home visits vs 3+ home visits	-5.28	0.040

Clinical Outcomes

The intervention and control group communities showed no significant difference in any of the clinical outcomes at baseline.

Hemoglobin

The average hemoglobin gm/dL of the children in intervention group and control group, at baseline and endline, are shown in Table 3. The independent samples t-test found no significant difference in hemoglobin levels in the intervention and control group at endline (t(140)=-0.37, p<0.711). The results of the t-test analysis can be seen in Table 4. A one way ANOVA showed that there was no significant difference between the three analysis groups, No home visits, 1-2 home visits, and 3+ home visits, in terms of hemoglobin levels at endline, F(2,139)=1.86, p=0.160. The results of the one-way ANOVA analysis can be seen in Table 5.

Early Childhood Development, Motor

The average ECD Motor scores for the children in the intervention and control group, at baseline and endline, can be seen in Table 3. Although the mean ECD motor score was 0.38 standard deviations higher in the intervention group at endline, the independent samples t-test found no significant difference (t(133)=-1.60, p<0.111). The one way ANOVA analysis found no significant difference between the three analysis groups, F(2,141)=1.28, p=0.282.

Early Childhood Development, Cognitive

The average ECD cognitive scores for the children in the intervention and control group, at baseline and endline, can be seen in Table 3. The independent samples t-test showed that there was no significant difference in ECD cognitive scores at endline (t(135)=-1.20, p<0.232). The one way ANOVA analysis also found no significant difference between the three analysis groups, F(2,141)=0.79, p=0.457 .

The results suggest that the CHEST App improved the performance of CHWs to teach caregivers important knowledge pertaining to healthy child rearing. The caregivers that received visits from CHWs that utilized the CHEST App produced greater knowledge scores (1.26 standard deviation increase) than caregivers that received visits from CHWs with traditional material (flip charts and brochures). Improvement in knowledge scores by the caregivers reflects the immediate impact of the CHEST App. As displayed in other studies, the mHealth tool improved the ability of the CHWs to conduct health promotion and education in the household.²¹

The CHEST App intervention failed to create statistically significant change in the clinical outcomes of the participants. It is believed to be due to the low dosage outcome that was ultimately achieved. The dosage implementation outcome revealed that the intervention reached only 5.7% of its objective, the number of home visits needed to make the hypothesized change to the clinical outcomes. Failure to meet the dosage objective created a breakdown in the theory of change. Full implementation of the intervention was not achieved during the study time frame. The reduced sample size and disruption in CHW services was due to the COVID-19 pandemic. The improved knowledge by caregivers is expected to contribute to improved clinical outcomes, after more time passes and more home visits can be conducted. The authors will review the health indicators in the intervention community in one year to again test the null hypotheses.

Prior to the implementation of the intervention, applied implementation research was conducted to identify effective strategies to install the intervention in the local context. The research team first assessed the key implementation determinants in the local context to identify potential barriers and facilitators to success. The determinants were based on the categorization from the Consolidated Framework for Implementation Research/CFIR; Damschroder et al., 2009.⁵¹ To help determine which implementation strategies should be included, the research team used the CFIR ERIC Matching Tool.⁵² The strategies that were used during the implementation process are listed in the IR Logic Model. Not all strategies suggested by the Matching Tool were utilized. For instance, funding strategies were not used due to a lack of receptibility by the local government to discuss changes to funding structures. For each implementation strategy, the research team implemented a set of discrete activities to operationalize the strategies.^53,54 Methods from Intervention Mapping (and Implementation Mapping) were used to develop the implementation strategies and evaluate the implementation outcomes.⁵⁵ The specific implementation activities are described in Westgard, et al., 2020.³² Finally, the implementation outcomes, service outcomes, and clinical outcomes were defined, monitored, evaluated, and reported, as displayed in the IR Logic Model (Appendix 3).

The implementation outcomes that were evaluated suggest that the implementation strategies were affective at installing the intervention in the established CHW program, with high degrees of acceptability, adoption, and fidelity. Acceptability and adoption by the CHWs were greatly influenced by the novelty of the CHEST App technology. The CHWs felt empowered from using the modern technology and more prepared to teach the caregivers due to the extensive educational material they had at hand. Fidelity rates of the CHEST App by the CHWs were consistently high, which is especially promising given the dynamic nature of home visits. Registering the child health indicators with the CHEST app was the most likely to be omitted during the home visit. For instance, if a home visit occurred and the caregiver did not have new information written on their child’s growth monitoring card, the CHWs would not register child health indicators during that visit. This led to missed opportunity to record cases of infections that may have occurred since the last home visit.

Full adoption and fidelity by CHW program coordinators were not achieved because there was little desire to digitize the surveillance data. The coordinators and local municipality did not have the desire to rely on technology to track the health of children in their communities. Because the programs are not integrated into a larger, regional program, the number of children managed by the program remains relatively small. The CHW program and municipality value the educational component of the CHEST App but do not value the digital surveillance component. The digital surveillance component will be more important for CHW programs that are integrated into a regional or national level program, which demands greater management of data.

Outcomes related to sustainability and penetration need to be assessed after the communities have had more time with the CHEST App and following the end of support from the external research team from Elementos. Sustainability of the CHEST App intervention is greatly influenced by the sustainability of the CHW programs. The local municipalities have been inconsistent in their support of the CHW programs. The three municipalities in the CHEST App pilot study each changed the CHW program during the 18-month trial. Two of the municipalities temporarily canceled the CHW program and one of the municipalities cut back the number of CHWs from 10 to 5, due to a lack of available funds. The National CHW program (Cuna Mas)⁵⁶ that also conducts home visits throughout the poorest regions of Peru is more stable, and thus a better alternative to incorporate the CHEST App. The implementation outcomes, service outcomes, and clinical outcomes from this pilot study are expected to be generalizable to the context of the National CHW program.

Limitations

The reduced sample size of the study due to program shutdowns caused by COVID-19 diminished the ability of the study to detect changes in the clinical outcomes. The restriction makes it difficult to conclude that the intervention is, or is not, effective at improving ECD or anemia.

The study design may have also been compromised by adaptations to the randomization process (to intervention and control group). Although assignment of the communities to the intervention and control group was randomized by the research team, the CHW program than led the randomization of individuals within the intervention community to receive the intervention. It was expected that the CHWs would be able to visit all children in the intervention community, however, reduction in the number of CHWs in the program hindered that possibility. The CHWs may had choose families to be involved in the program for reasons other than randomization. By checking for balance between the two groups at baseline in the key outcomes, and finding no statistically significant difference, the authors believe the objective of randomization was maintained.

An additional limitation was caused by the method to select who received the hemoglobin test. Hemoglobin levels were measured by the local health center and were not random nor a census of all children in the community. There may be selection bias of those that received the hemoglobin test and reported hemoglobin levels to the study. The anemia rates reported in the study suggests that there was a selection bias present in the sampling because the anemia rates are significantly lower than the regional average. To accurately detect the effect of the CHEST App on anemia rates a more rigorous method of hemoglobin assessment will be needed.

The effectiveness-implementation hybrid type 2 study provided the following results:

The CHEST App intervention can be effectively incorporated into a CHW program with high degrees of acceptability, adoption, and fidelity.

Adoption and fidelity of the surveillance function of the CHEST App by program coordinators was not achieved.

The CHEST App intervention is associated with improvements in knowledge of healthy child rearing practices by caregivers.

Program interruptions caused by COVID-19 created a decrease in final dosage of the intervention, interrupting or delaying the theory of change.

The improved knowledge scores by the caregivers are theorized to contribute to a reduction in anemia rates and improve ECD scores, with more time passed with the intervention. The CHEST App is a promising tool to improve the performance of CHWs during their home visits, to accomplish their objective of teaching caregivers healthy childrearing practices and improving childhood health and development in their communities.

C.A.R.E Guidelines: Consolidated Advice on Reporting ECD Implementation Research Guidelines

CHEST App: Child Health Education and Surveillance Tool Application

ECD: Early Childhood Development

ICT: Information and Communication Technology

IR Logic Model: Implementation Research Logic Model

Ethics approval and consent to participate

The investigation was approved by the Institutional Ethics Committee of the National Hospital ‘San Bartolomé' in Peru on November 8, 2018 (Exp. Number 15 463–18, Oficio N. 0744–2018- OADI- HONADOMANI- SB) and the Institutional Review Board, Office of Human Research Ethics at the University of North Carolina – Chapel Hill (IRB Number: 19-3097). The participants provided their written informed consent to participate in this study.

Consent for Publication

Not Applicable

Availability of Data and Material

The dataset supporting the conclusions of this article is available in the figshare repository, 10.6084/m9.figshare.16725271.

Competing Interests

The author declares that they have no competing interests

Funding

Project undertaken with the ﬁnancial support of a grant from Saving Brains, Grand Challenges Canada and the Government of Canada, through Global Aﬀairs Canada (GAC). Project made possible with the administrative and research support of Elementos and funding from Grand Challenges Canada, Saving

Brains Grant.

Authors Contributions

CW designed, implemented, analyzed, and reported the study. LOF designed, analyzed, and reported the study.

Acknowledgements

Special thanks to the team at Elementos whom helped develop the program material and implement the program in the field; Mayra Young, Liz Franco Calderòn, Milagros Alvarado Llatance, and Gabriela Palacios Rojo. Special thanks to the community health workers, program coordinators and municipality workers in Indiana, Mazan, and Fernando Lores, that work so hard for the betterment of the children in their communities.

Mosler H-J. A systematic approach to behavior change interventions for the water and sanitation sector in developing countries: a conceptual model, a review, and a guideline. Int J Environ Health Res. 2012;22(5):431–49. doi:10.1080/09603123.2011.650156.
Black MM, Walker SP, Fernald LCH, et al. Early childhood development coming of age: science through the life course. Lancet Lond Engl. 2017;389(10064):77–90. doi:10.1016/S0140-6736(16)31389-7.
Bhutta ZA, Black RE. Current and Future Challenges for Children Across the World. JAMA. 2019;321(13):1251–2. doi:10.1001/jama.2019.1840.
Black RE, Levin C, Walker N, Chou D, Liu L, Temmerman M. Reproductive, maternal, newborn, and child health: key messages from Disease Control Priorities 3rd Edition. The Lancet. 2016;388(10061):2811–24. doi:10.1016/S0140-6736(16)00738-8.
Williamson J, Ramirez R, Wingfield T. Health. Healthcare Access, and Use of Traditional Versus Modern Medicine in Remote Peruvian Amazon Communities: A Descriptive Study of Knowledge, Attitudes, and Practices. Am J Trop Med Hyg. 2015;92(4):857–64. doi:10.4269/ajtmh.14-0536.
Westgard CM, Rogers A, Bello G, Rivadeneyra N. Health service utilization, perspectives, and health-seeking behavior for maternal and child health services in the Amazon of Peru, a mixed-methods study. Int J Equity Health. 2019;18(1):155. doi:10.1186/s12939-019-1056-5.
Gyorkos TW, Maheu-Giroux M, Blouin B, Casapia M. Impact of Health Education on Soil-Transmitted Helminth Infections in Schoolchildren of the Peruvian Amazon: A Cluster-Randomized Controlled Trial. PLoS Negl Trop Dis. 2013;7(9). doi:10.1371/journal.pntd.0002397.
Asaolu SO, Ofoezie IE. The role of health education and sanitation in the control of helminth infections. Acta Trop. 2003;86(2-3):283–94. doi:10.1016/s0001-706x(03)00060-3.
Zavaleta N. Anemia infantil: Retos y Oportunidades al 2021. Rev Peru Med Exp Salud Publica. 2017;34(4):588–89.
Janmohamed A, Sohani N, Lassi ZS, Bhutta ZA. The Effects of Community Home Visit and Peer Group Nutrition Intervention Delivery Platforms on Nutrition Outcomes in Low and Middle-Income Countries: A Systematic Review and Meta-Analysis. Nutrients. 2020;12(2). doi:10.3390/nu12020440.
Perry HB, Sacks E, Schleiff M, et al. Comprehensive review of the evidence regarding the effectiveness of community–based primary health care in improving maternal, neonatal and child health: 6. strategies used by effective projects. J Glob Health. 7(1). doi:10.7189/jogh.07.010906.
Lewin S, Munabi-Babigumira S, Glenton C, et al. Lay health workers in primary and community health care for maternal and child health and the management of infectious diseases. Cochrane Database Syst Rev. 2010;(3). doi:10.1002/14651858.CD004015.pub3.
Lassi ZS, Bhutta ZA. Community-based intervention packages for reducing maternal and neonatal morbidity and mortality and improving neonatal outcomes. Cochrane Database Syst Rev. 2015;(3):CD007754. doi:10.1002/14651858.CD007754.pub3.
Gilmore B, McAuliffe E. Effectiveness of community health workers delivering preventive interventions for maternal and child health in low- and middle-income countries: a systematic review. BMC Public Health. 2013;13(1):847. doi:10.1186/1471-2458-13-847.
Westgard C, Naraine R, Paucar Villacorta DM. Performance Evaluation of Community Health Workers: Case Study in the Amazon of Peru. J Community Health Published online March 26, 2018. doi:10.1007/s10900-018-0503-3.
Ilozumba O, Dieleman M, Kraamwinkel N, Belle SV, Chaudoury M, Broerse JEW. “I am not telling. The mobile is telling”: Factors influencing the outcomes of a community health worker mHealth intervention in India. PLOS ONE. 2018;13(3):e0194927. doi:10.1371/journal.pone.0194927.
van Heerden A, Sen D, Desmond C, Louw J, Richter L. App-Supported Promotion of Child Growth and Development by Community Health Workers in Kenya: Feasibility and Acceptability Study. JMIR MHealth UHealth. 2017;5(12). doi:10.2196/mhealth.6911.
Lee S, Chib A, Kim J-N. Midwives’ Cell Phone Use and Health Knowledge in Rural Communities. J Health Commun. 2011;16(9):1006–23. doi:10.1080/10810730.2011.571344.
Källander K, Tibenderana JK, Akpogheneta OJ, et al. Mobile Health (mHealth) Approaches and Lessons for Increased Performance and Retention of Community Health Workers in Low- and Middle-Income Countries: A Review. J Med Internet Res. 2013;15(1):e17. doi:10.2196/jmir.2130.
Prinja S, Nimesh R, Gupta A, Bahuguna P, Gupta M, Thakur JS. Impact of m-health application used by community health volunteers on improving utilisation of maternal, new-born and child health care services in a rural area of Uttar Pradesh, India. Trop Med Int Health TM IH. 2017;22(7):895–907. doi:10.1111/tmi.12895.
Ilozumba O, Van Belle S, Dieleman M, Liem L, Choudhury M, Broerse JEW. The Effect of a Community Health Worker Utilized Mobile Health Application on Maternal Health Knowledge and Behavior: A Quasi-Experimental Study. Front Public Health. 2018;6. doi:10.3389/fpubh.2018.00133.
Chib A. The Aceh Besar midwives with mobile phones project: Design and evaluation perspectives using the information and communication technologies for healthcare development model. J Comput-Mediat Commun. 2010;15(3):500–25. doi:10.1111/j.1083-6101.2010.01515.x.
Limaye NP, Rivas-Nieto AC, Carcamo CP, Blas MM. Nuestras Historias- Designing a novel digital story intervention through participatory methods to improve maternal and child health in the Peruvian Amazon. PLOS ONE. 2018;13(11):e0205673. doi:10.1371/journal.pone.0205673.
Piette JD, Lun KC, Moura LA, et al. Impacts of e-health on the outcomes of care in low- and middle-income countries: where do we go from here? Bull World Health Organ. 2012;90(5):365–72. doi:10.2471/BLT.11.099069.
Mookherji S, Mehl G, Kaonga N, Mechael P. Unmet Need. Improving mHealth Evaluation Rigor to Build the Evidence Base. J Health Commun. 2015;20(10):1224–9. doi:10.1080/10810730.2015.1018624.
Smith JD, Li DH, Rafferty MR. The Implementation Research Logic Model: a method for planning, executing, reporting, and synthesizing implementation projects. Implement Sci. 2020;15(1):84. doi:10.1186/s13012-020-01041-8.
Peterson HB, Haidar J, Fixsen D, Ramaswamy R, Weiner BJ, Leatherman S. Implementing Innovations in Global Women’s, Children’s, and Adolescents’ Health: Realizing the Potential for Implementation Science. Obstet Gynecol. 2018;131(3):423–30. doi:10.1097/AOG.0000000000002494.
Peterson HB, Haidar J, Merialdi M, et al. Preventing maternal and newborn deaths globally: using innovation and science to address challenges in implementing life-saving interventions. Obstet Gynecol. 2012;120(3):636–42. doi:10.1097/AOG.0b013e3182632cc1.
Westgard CM, Rivadeneyra N, Mechael P. mHealth tool to improve community health agent performance for child development: study protocol for a cluster-randomised controlled trial in Peru. BMJ Open. 2019;9(11):e028361. doi:10.1136/bmjopen-2018-028361.
An Overview of Active Implementation Frameworks | NIRN. National Implementation Research Network. Accessed November 29. 2019. https://nirn.fpg.unc.edu/module-1.
Fixsen D, Naoom S, Blase K, Friedman R, Wallace F. Implementation Research. A Synthesis of the Literature. University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; 2005.
Westgard C, Fleming WO. The Use of Implementation Science Tools to Design, Implement, and Monitor a Community-Based mHealth Intervention for Child Health in the Amazon. Front Public Health. 2020;8:411. doi:10.3389/fpubh.2020.00411.
Encuesta Demográfica y de Salud Familiar 2019 - Nacional y Regional. Instituto Nacional de Estadística e Informática, Peru; 2019. https://proyectos.inei.gob.pe/endes/2019/ppr/Indicadores_de_Resultados_de_los_Programas_Presupuestales_ENDES_Primer_Semestre_2019.pdf.
Westgard C, Alnasser Y. Developmental delay in the Amazon: The social determinants and prevalence among rural communities in Peru. PloS One. 2017;12(10):e0186263. doi:10.1371/journal.pone.0186263.
Cruzado de la Vega V, Cavero Arguedas D, Araujo MC, Dormal M, Rubio-Codina M. Resultados de La Evaluacion de Impacto Del Servicio de Acompañamiento a Familias Del Programa Nacional Cuna Mas. Ministerio de Economia y Finanzas, Intermericano de Desarrollo; 2016. https://www.mef.gob.pe/contenidos/presu_publ/ppr/eval_indep/informe_resultados_cuna_mas.pdf.
Josephson K, Guerrero G, Coddington C. Cómo Apoyar al Personal Que Trabaja En Programas a Gran Escala Dirigidos a La Primera Infancia: El Caso Del Servicio de Acompañamiento a Familias de Cuna Más En Perú. Results for Development; 2017.
Elementos. Child Health Education and Surveillance Tool Application CHEST App. Accessed June 15, 2020. https://www.youtube.com/watch?v=ChuGwXTvs58&list=PLO-rrzBuEiYJxE_yF92ERqJk0X7X5oC5A&index=10&t=0s.
Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation Hybrid Designs. Med Care. 2012;50(3):217–26. doi:10.1097/MLR.0b013e3182408812.
Yousafzai AK, Aboud FE, Nores M, Kaur R. Reporting guidelines for implementation research on nurturing care interventions designed to promote early childhood development. Ann N Y Acad Sci. 2018;1419(1):26–37. doi:10.1111/nyas.13648.
Yousafzai AK, Aboud FE, Nores M, Kaur R. Reporting guidelines for implementation research on nurturing care interventions designed to promote early childhood development. Ann N Y Acad Sci. 2018;1419(1):26–37. doi:10.1111/nyas.13648.
Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. doi:10.1007/s10488-010-0319-7.
Boslaugh S. Dose-Response Relationship. Encyclopedia of Epidemiology. SAGE Publications, Inc.; 2008:pp. 283–3. doi:10.4135/9781412953948.
Morrison AK, Glick A, Yin HS. Health Literacy: Implications for Child Health. Pediatr Rev. 2019;40(6):263–77. doi:10.1542/pir.2018-0027.
Rubio-Codina M, Araujo MC, Attanasio O, Muñoz P, Grantham-McGregor S. Concurrent Validity and Feasibility of Short Tests Currently Used to Measure Early Childhood Development in Large Scale Studies. PLOS ONE. 2016;11(8):e0160962. doi:10.1371/journal.pone.0160962.
Fernald LCH, Kariger P, Hidrobo M, Gertler PJ. Socioeconomic gradients in child development in very young children: evidence from India, Indonesia, Peru, and Senegal. Proc Natl Acad Sci U S A. 2012;109(Suppl 2):17273–80. doi:10.1073/pnas.1121241109.
Fernald L, Kariger P, Engle P, Raikes A. Examining Early Child Development in Low-Income Countries: A Toolkit for the Assessment of Children in the First Five Years of Life. The World Bank; 2009.
Rubio-Codina M, Araujo MC, Attanasio O, Grantham-McGregor S. Validez concurrente y viabilidad de pruebas cortas comunmente usadas para medir el desarrollo infantil temprano en estudios a gran escala: Metodologia y resultados. Banco Interam Desarro. 2016;(723). https://publications.iadb.org/bitstream/handle/11319/7823/Validez-concurrente-y-viabilidad-de-pruebas-cortas-comunmente-usadas-para-medir-el-desarrollo-infantil-temprano-en-estudios-a-gran-escala-metodologia-y-resultados.pdf.
Rubio-Codina M, Attanasio O, Grantham-McGregor S. Mediating pathways in the socio-economic gradient of child development: Evidence from children 6–42 months in Bogota. Int J Behav Dev. 2016;40(6):483–91. doi:10.1177/0165025415626515.
McCoy DC, Waldman M, Fink G. Measuring early childhood development at a global scale: Evidence from the Caregiver-Reported Early Development Instruments. Early Child Res Q. 2018;45:58–68. doi:10.1016/j.ecresq.2018.05.002.
R: What is R? Accessed September 14. 2021. https://www.r-project.org/about.html.
Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50. doi:10.1186/1748-5908-4-50.
Waltz TJ, Powell BJ, Fernández ME, Abadie B, Damschroder LJ. Choosing implementation strategies to address contextual barriers: diversity in recommendations and future directions. Implement Sci IS. 2019;14(1):42. doi:10.1186/s13012-019-0892-4.
Bunger AC, Powell BJ, Robertson HA, MacDowell H, Birken SA, Shea C. Tracking implementation strategies: a description of a practical approach and early findings. Health Res Policy Syst. 2017;15(1):15. doi:10.1186/s12961-017-0175-y.
Waltz TJ, Powell BJ, Matthieu MM, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10(1):109. doi:10.1186/s13012-015-0295-0.
Fernandez ME, Ten Hoor GA, van Lieshout S, et al. Implementation Mapping: Using Intervention Mapping to Develop Implementation Strategies. Front Public Health. 2019;7:158. doi:10.3389/fpubh.2019.00158.
Acompañamiento a Familias. Programa Nacional Cuna Mas. Accessed April 6. 2017. http://www.cunamas.gob.pe/?page_id=57.

Download PDF

Version 1

posted

You are reading this latest preprint version

Implementation Research for the Evaluation of the Child Health Education and Surveillance Tool Application: A Randomized-Controlled Trial

Status:

Version 1

Abstract

Contributions to the Literature

Background

Methods

Study Design

Implementation Outcomes

Intervention Outcomes

Results

Acceptability by Caregivers

Acceptability by CHWs

Adoption by CHWs

Adoption by CHW Program Coordinators

Dosage

Fidelity

Cost

Service Outcome

Knowledge Scores by Caregivers

Clinical Outcomes

Hemoglobin

Early Childhood Development, Motor

Discussion

Limitations

Conclusions

Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1