Step 1: Delphi Process. We will use a three-round Delphi process to create the PREFER training. Based on a review of current training topics for CHWs in this area, we have developed a set of topics for the PREFER course. We will develop a prototype for the CHW Curriculum Committee to review (Table 1). The PREFER training uses the social cognitive theory (SCT) to guide the determinants that CHWs need to achieve to support patients in genomics. According to SCT, individuals are more likely to adopt a new behavior if they have a sense of self-efficacy, gain knowledge and skills to overcome obstacles, and have positive expectations about adopting the new behavior.10 Thus, the training is designed to increase knowledge, skills, and self-efficacy for CHWs to have discussions about hereditary conditions and genomics with their communities. The program will incorporate Adult Learning Theory methods throughout to ensure practical applications of the topics for CHWs. These include: learning should capitalize on the participants’ experience, learning should be adapted to the limitations of the participant, adults should be challenged to move to advance stages of personal development, adults should have a choice in organizing the learning program.11
Table 1
Topics for Inclusion in PREFER Training
Topic | Description |
Importance of family history (1 hour) | Introduce family history, sharing personal story of someone who tested positive, group discussion with question and answers |
Genetics 101 (1 hour) | Basic concepts of genetics |
Tier 1 conditions (1 hour) | Practice drawing family histories, case examples and role play, practice drawing family histories in pairs, learn about referrals for affected and unaffected family members |
Genetic counseling and testing (30 minutes) | Introduce topic of genetic testing, navigation of genetic testing and counseling services |
Ethical, Legal and Social Issues (1 hour) | Discuss ethical, legal and social issues including communicating about family history and potential stigma of passing along deleterious mutation, sharing results, cost of testing, confidentiality |
Navigating genetic resources (1 hour) | Describe different types of genetic services, group discussion about genetic counseling, demonstrate how to navigate national database of genetic resources |
Communication about hereditary conditions (1 hour) | Group discussion about how to apply information discussed to day-to-day work, including motivational interviewing |
Curriculum is based on Adult Learning Theory which focuses on sharing personal experiences, learning being adapted to limitations of participants, learners being challenged to move to advanced stages of personal development and having a choice in organizing the learning program. Examples include: scripted role play, group discussion, link to previous knowledge, and case examples. |
The three-round Delphi process will follow the RAND/UCLA Appropriateness Method (RAM) for conducting expert panels with the Delphi method, we will include two rounds of rating with a round of discussion between the two rounds.12 Round 1: Assessment Round. The expert panel will be instructed to review the curriculum prototype and rate and comment on its different aspects using a 5-point Likert scale; participants will also provide qualitative feedback. Round 2: Feedback and Discussion Round. Experts will receive a personalized report showing how their individual responses to the rating questions compare to the responses of other participants. The report will include a distribution of all responses, a group median response and its interquartile range, and a statement that explains whether the group reached agreement. The report will also include a summary of participant’s qualitative comments during Round 1. In Round 2, participants will discuss the results of Round 1 using an anonymous, asynchronous threaded discussion board hosted on a learning management platform. We will use randomly generated usernames such as Expert01. We anticipate that this discussion will focus on areas where there may be disagreement or potential confusion. Round 3: Reassessment Round. After the discussion round is complete, we will provide experts with the opportunity to revise their Round 1 answers based on Round 2 feedback and discussion or leave them unchanged. Any modifications made to Round 1 questions will be clearly identified at the end of Round 3. Each round will be open for 7-10 days. Participants will receive an invitation email at the start of each round that will include a description of what they are expected to do, how to access the learning management platform, and how long each round will be open. We will send up to three reminders during each round to encourage all participants provide their input. Upon completing the Delphi process, we will update the curriculum and prepare for delivery.
Measures for Delphi Process. We will use the Kirkpatrick Model of Training Evaluation to develop questions for the Delphi process (Table 2). Participants will be asked to review the PREFER curriculum and rate their level of agreement with each statement using a 5-point Likert agreement scale (strongly disagree to strongly agree) as part of Round 1 and during Round 3. Qualitative assessment questions will be used to help explain the quantitative ratings and be provided to experts as part of Round 2 (feedback and discussion).
Table 2
Measures used during the Delphi Process
Kirkpatrick Model | Quantitative Assessment | Qualitative Assessment |
Reaction: experience with the training | • Course materials are aligned with adult learning theory • The length of training is appropriate for materials covered • The materials are presented in a way that is accessible to CHWs | How can we improve participant’s response to the training? |
Learning: what was learned as part of the training | • Participants learn the desired knowledge about genetics and genomics • Participants learn the desired skills related to hereditary conditions • There is clear integration of topics • Sequencing of topics are appropriate | How much do you think participants will learn from the training? How will their skills improve? How can we improve the learning opportunities? |
Behavior: changes in behavior following training | • Course will help CHWs identify individuals at high risk of hereditary conditions • The course content is relevant to CHW practice | How do you see participant’s applying their learning in practice? |
Results: longer-term impact of training | • Will this empower CHWs to help change client’s behavior? • Will this empower CHWs to change their behavior? | Are there ways to enhance the impact of this training? |
All quantitative assessments will be assessed using 5-point Likert scale (strongly disagree, disagree, neutral, agree, strongly agree) |
Analysis for Delphi Process. We will use descriptive statistics to present results of Round 1 and 3 ratings for each panel separately, focusing on the frequency distributions of responses to each question, as well as measures of central tendency (median) and dispersion (interpercentile range). Explicit assessment of inter-rater agreement will be performed using kappa coefficients. Significance of these coefficients will be assessed using threshold assessment. To assess patterns of agreement, we will extend this analysis to log linear models for multiple categories. Power assessment for these models follows from the log linear formulation.
Delivery and Evaluation of PREFER Training. After finalizing the curriculum, we will deliver and evaluate the training among a cohort of 30 CHWs nationally. CHWs will be eligible if they are employed by or volunteer for an organization, clinic, institution, hospital or agency providing direct education, outreach or health services to minority populations Specific recruitment strategies include: email outreach, distribution through relevant list servs, and word of mouth. Interested trainees will submit an online application including information about educational and professional background, workplace setting and demographics of the patient and community populations they serve, current educational and referral practices related to genomics, two personal and professional goals for the training, and assurance they can attend all components of the training.
All materials will be delivered virtually using the Zoom platform. Participants will receive a pre-course survey delivered through REDCap two days prior to beginning the course to assess initial knowledge, skills, and self-efficacy in key topics. Participants will also receive a post-course survey immediately following all training to assess change in competencies as well as an evaluation of participant experience with the course. Finally, participants will receive a 6-month after follow-up survey to determine change/attrition in competencies, as well as how the skills gained in this curriculum are being used.
Measures for Course Delivery. Measures are guided by the RE-AIM AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework to evaluate the course, demonstrate its effectiveness, and gather data prior to future delivery.13,14 This organizing framework helps identify the essential program elements that can improve the adoption and implementation of programs (see Table 3). Data sources will include: course records, quantitative surveys delivered pre-course, immediately post-course, and 6-months following the training.
Table 2
Measures to Assess PREFER CHW Training
RE-AIM Construct | Measures | Source |
Reach into the target population. How can we improve the reach of PREFER? | • # of participants • % of participants who completed the course • # of people on wait list • Log of recruitment strategies (list servs, types of outreach) • Log of questions from potential participants (FAQ) | Course records Tracking logs |
Effectiveness of the intervention on participant knowledge. Did PREFER improve trainee’s competence in key learning objectives? | • Genetic health literacy: use REAL-G which captures functional genetic literacy through key word familiarity and comprehension measures15 • Genetic knowledge: 8-item multiple choice questionnaire on general genetic knowledge.15 | Pre-course survey Immediate post-course survey Six-month post-course survey |
Adoption of intervention components into CHW practice. How confident are trainees in using approaches taught in the program? | • Self-efficacy for identifying at risk women, providing education, referral, and navigation services as assessed with items adapted from scales of self-efficacy for genetic counseling. 7 item measure assesses on 10-point Likert scale and summed16 • Attitudes toward genetic counseling and testing: 17 item validated measure where 10 items assessed perceived benefits for genetic testing for cancer and 9 items assessed perceived barriers to genetic testing for cancer risk, and 5 items assessed concerns related to genetic counseling and testing17 | Pre-course survey Immediate post-course survey Six-month post-course survey |
Implementation of training. How well was PREFER implemented? | • Kirkpatrick Model Measures to assess reaction, learning, behavior and results (e.g., training objectives were clearly articulated, information presented in an understandable way, provided personal value). Measured on 5-point Likert scale.18 • Rank order of preferred learning sessions • Open ended questions about the aspects of training sessions that were most helpful, least helpful, and how they intended to use the information within community | Immediate post-course survey |
Maintenance of effects of program over time. Do trainees retain knowledge learned from PREFER? | • Participant log of number of: # of people discussed genetics, # identified, referred, or navigated at high-risk for genetic services 19,20 • Perceived relevance of training: Agreement with statement: “This training is relevant to my work as a CHW.” Measured on 5-point Likert scale. | Immediate post-course survey Six-month post-course survey |
Analysis for PREFER Training. For individual question responses we will assume a categorial logistic model and employ item response theory (IRT) to assess the latent responses underlying the data. These will be longitudinal in form and we plan to test hypotheses focused on a) differences between different time points for individual questions, b) common across-participant groupings on the longitudinal responses. For groups of questions, we will assume a multivariate logistic longitudinal modeling framework whereby we will assess both inter participant and inter-question group latent groupings. In terms of power and sample size we assume a logistic linear model with fixed correlation over time. For power of 0.815 and alpha level of .05 we can achieve a two-period regression (trend) effect of 0.33, assuming and intercept of -0.5 and adjusted predictor of 0.2, based on a simulation of 1000 samples of size 100. For a sample size of 30 with power 0.548 and adjusted predictor of 0.3, the achievable effect size is 0.5. Drop out from the study will be examined and we plan to use intention to treat (ITT) so that all data arising will be considered, with missing data imputed where appropriate. For non-random missingness we will assume a survival model for the drop-out mechanism.