Between February and April 2017, FELTP used quantitative, semi-quantitative, and qualitative methods to evaluate all FETP-F activities. Groups 1–6 formed the population for the process evaluation because they graduated ≥ 18 months before the impact evaluation began. We followed-up with graduates of these 6 groups to gather the data for the quantitative portion of the summative evaluation. Field project data provided baseline information for the DQA, DCA, and OTR measures. By February 2017, we had 103 graduates who had provided all pieces of the quantitative information. We randomly selected 35 graduates from the 103 for on-site follow-up for 1) face-to-face interviews, 2) observation of their work environment, 3) interviews with their supervisors, 4) interviews with their colleagues, 5) verification of the DQA and DCA and OTR data reported on the past 18 months.
Quantitative measures. We used interrupted repeated measures on 3 quantitative values. Because all participants had to complete these activities, we used the field project values as the baseline values. Afterward, during the process evaluation at 6 months, we gauged their measures at that time. One year post-graduation, we gauged the same measures again. The final measure was taken at least 18 months post-graduation. All measures were self-reported via online survey. For all quantitative measures, we conducted one-way ANOVAs using MS-Excel’s data toolpak. The quantitative measures are described below.
DQA scores. The participants had to complete a DQA for their field project, and we used these scores as the baseline scores. The DQA tool was designed to: (1) verify the quality of health facility data, (2) assess the system that produces that data, and (3) develop action plans to improve both [Cheburet et al., 2016]. We subsequently asked graduates to measure DQA scores from the same data source as the field project, except it should be 6 months beyond the baseline data. We then asked them to repeat the procedure at month 12 and finally at month 18 post-graduation. We used one-way ANOVA to determine if there were differences in the mean DQA scores over time.
DCA scores. The DCA is an end-to-end data integrity process. Because DCA focuses on the entire surveillance network, we did not ask the graduates to cross-check all data points in the system. We asked them to do a check of the indicators and counts used in their field projects. The first end is the generation of data at the health facility level. The middle is the county record, where the health facilities report their weekly and monthly tallies to the county health department (CHD) using MoH form 753. Then those data are entered into the district health information system (DHIS) by the county health records and information officer (HRIO). The DCA process is outlined in Fig. 2.
The goal is to detect inconsistencies as data travel through the surveillance system and identify root causes for these inconsistencies and to develop solutions, at the most granular level of the surveillance system – the health facility. The DCA score indicates the depth of the inconsistencies. A low score indicates high inconsistencies, whereas a high score indicates low inconsistency levels of the data between the 3 repositories of surveillance data. We used one-way ANOVA to determine if there were differences in the mean DCA scores over time.
Timeliness of reporting. Timeliness is a key performance measure of public health surveillance systems. Timeliness can vary by disease, intended use of the data, and public health system level. The participants, as part of their field projects, had to evaluate the timeliness of reporting for the condition, disease, or health priority that was the focus of their field project. We used the results from the field project as baseline OTR measures. Then we followed up at 6 months to assess the proportion of reports submitted on time for the previous quarter. We repeated the procedure at 12 months post-graduation and then a final query at least 18 months post-graduation to examine the proportion of OTRs for the prior quarter. We used one-way ANOVA to determine if there are differences in the mean OTR scores over time.
Semi-quantitative measures. At the beginning of each training course, we asked participants to score their knowledge and skills in 8 key competencies on a Likert scale from 1 to 5, with 1 representing limited knowledge/skills and 5 representing expertise (Table 3). At the end of the 3-month training, we asked them to gauge their knowledge skills in each of those areas now that they have sat through 30 hours of didactic training, received hands-on coaching and mentoring from FELTP faculty, and completed a 5-week field project. We use the pre-post difference as our comparison point when we followed up after 18 months and asked them to rate their knowledge and skills now in terms of practical applications to their day-to-day work. We also asked their supervisors and colleagues to score the graduates’ skills and knowledge and practical application in each of those competencies. We used those scores to gauge the impact of FETP-F training on knowledge, skills, and change in work methods. We conducted a one-way ANOVA to determine if there was a difference in the scores between the 3 groups.
Table 3
Pre- and post-self-assessment score sheet, public health competencies, n = 103
Before Training | Self-assessment of Your Knowledge and Skills Related to: | After Training |
1 | 2 | 3 | 4 | 5 | I can calculate basic measures of central tendency: mean, median, & mode | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I can calculate basic statistical measures of dispersion: range, variance, & standard deviation | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I have an understanding of descriptive epidemiology | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I have an understanding of basic disease surveillance | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I can use MS-Excel to enter and manipulate basic data | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I know how to use formulas in MS-Excel | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I know how to look at data and analyze it by person, place, and time | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I know the basic components of a functional disease surveillance system | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I know the difference between active and passive surveillance systems | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I know the difference between qualitative and quantitative data | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I can develop a case definition to use during a field investigation | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I can analyze field and surveillance data and apply appropriate statistics to describe what the data show | 1 | 2 | 3 | 4 | 5 |
1 | 2 | 3 | 4 | 5 | I can audit health facility data for accuracy and completeness | 1 | 2 | 3 | 4 | 5 |
Qualitative measures. The qualitative portion of the evaluation used grounded theory to determine the impact of the training, mentoring, and supervision on behavior, work method, and application of training to work duties [Reeves et al., 2015]. The grounded theory approach allowed us to develop our inquisitive instruments and then draw theory from them as we analyzed the interview transcripts, the interviewers’ field notes from observations, and the previous responses to other evaluation tools as the graduates went through the 3-month training.
Semi-structured interviews were conducted with randomly selected graduates from groups 1–6. Because we wanted to examine the impact of the training at least 1.5 years post-graduation, so that we could, at most, look at the first 6 groups to go through the FETP-F process. Those groups enrolled between July 2014 and July 2015. All interviews were recorded with the consent of the interviewees. All interviewees had to provide written and verbal consent to the interview.
Participant observation. Participant observation allowed the field investigators to establish rapport with the person being interviewed so that the interviewee would provide more honest answers and opinions (vs answering what they think the interviewer wants to hear) [Laurier, 2016]. We used a checklist for field workers to note the presence of monitoring charts, active use of the DQA and DCA tools, and presence of log books, interactions with colleagues, presence of operational tools such as laptops, printers, case definition posters. Field investigators had to submit their field notes and completed checklists along with the consent forms and audio files to the evaluation cloud site. Audio files were sent out for transcription to parties unaffiliated with FELTP, the participants, CDC-Kenya, and other stakeholders.
Developing the interview instrument. In September 2016, FETP-F initiated development of the semi-structured interview and guide that would provide data to support the impact evaluation. The questionnaire was field tested in South Sudan for clarity and then tested with medical education partnership initiative (MEPI) graduates in Kenya. We did not develop questions that could use a Likert scale to measure these 2 attributes of the graduates. Instead, we used open-ended questions that would help us getting a fuller picture of the impact on multiple areas of their personal and work lives [Melovitz et al., 2018].
Trainings to prepare for evaluation. In September 2016, FETP-F hosted a one-week workshop on qualitative research methods in Nairobi, Kenya, facilitated by CDC/DGHT (Atlanta, Georgia, USA). The workshop participants were FELTP faculty and CDC-Kenya staff who supervised the evaluation field workers, who would conduct the interviews for the evaluation.
Field training. In December 2016, 17 MEPI graduates underwent one week of residency-based training by CDC-Kenya staff on interview and observation methods. Kenya FELTP staff also showed them how to conduct the confirmatory DQA and DCA exercises. Each MEPI graduate had to complete the online FHI-360 ethics training module before graduating from the training course [Aalborg et al., 2016].