Cluster-Randomized Trial
Aim 1: Compare two implementation strategies using a CRT.
Sample Frame – County Selection and Randomization
For our sample frame, we selected counties with an estimated number of YMSM greater than 1,500(45) (N = 113 counties). Counties in which KIU has been studied previously were excluded. We randomized 66 counties 2:1 to the CBO and DTC strategies, respectively. Our team opted to randomize 2:1 :: CBO:DTC because we expected that some counties would not have any CBOs apply, or if they applied, some counties’ applicant CBOs would not meet minimal requirements for implementing KIU. Further, we randomized 22 counties to receive the KIU DTC strategy because we expected no selection loss(46) from counties where we will advertise online directly to YMSM. KIU will be delivered in each county for two years. Figure 1: KIU! 3.0 Trial Design illustrates our primary design elements. Figure 2 is a CONSORT diagram of the flow of the trial; a CONSORT checklist was provided as an additional file in submitting this manuscript. See the section entitled County Selection Iterative Process in Additional File 1 – Supplemental Information for more details.
Participant Inclusion Criteria and Recruitment
YMSM are eligible for participation in the research components of KIU if they meet the following criteria: 1) test negative for HIV at the time of registration, 2) report that they are assigned male at birth, 3) identify their current gender identity as male or identify as non-binary, 4) are between the ages of 18–29 at registration, 5) report CAS with a male partner in the previous six months, and 6) are not on PrEP, have been on PrEP for less than six months, or report missing a PrEP dose in the previous six months.
Participants will be recruited into KIU using different strategies based on the arm of the study into which they are recruited. For the CBO arm, participants will be recruited at CBO sites after testing negative for HIV. CBO staff will explain KIU and register interested YMSM, who will receive an email with a link that will take them to the application. Once there, participants will complete a baseline survey where their eligibility for the KIU research surveys is verified. In the DTC arm, participants will be recruited primarily through social media, geospatial dating apps, and supplemental approaches (e.g. referrals and print ads). Participants will click through or enter a URL to complete an initial screener for eligibility. When a participant meets all of the other indicators for eligibility, they will be sent an at-home HIV test kit to verify the HIV negative enrollment criterion as well as materials to self-administer swabs and a urine collection cup for oral, urethral, and rectal gonorrhea and chlamydia. See the section entitled DTC Recruitment in Additional File 1 – Supplemental Information for additional details on how remote recruitment, screening, and testing will take place in the DTC arm.
Intervention Delivery
KIU 3.0 is a custom mobile, tablet, and desktop responsive web-based application that is developed using multi-tenant architecture. KIU consists of five user roles with separate login pages and varying levels of access to the KIU application: Superusers, Project Directors, Coordinators, Recruiters, and YMSM participants. Superusers are comprised exclusively of Northwestern University project staff who oversee the application’s functionality. CBO Project Directors oversee coordinator and recruiter efforts, have access to participant data for intervention retention, and act as primary contacts with KIU project staff at Northwestern University. Non-YMSM user role distinctions are most salient for the CBO arm as DTC project staff functionally have the same level of access as CBO Project Directors. Following completion of the baseline survey, YMSM participants must complete KIU content in sequence, though they are able to return to previously viewed material via a learning map. For more details on intervention delivery, including data collected by the application and the experience of YMSM, see the section entitled KIU Intervention Delivery in Additional File 1 – Supplemental Information.
Implementation Strategies - Community-Based Organization Arm – RFP Process
To select CBO recruitment sites, we wrote a request for proposals (RFP) modeled on various RFPs created by CDC and other funders of HIV prevention. Following, we developed an objective scoring rubric and trained objective reviewers to score 3–4 applications on nine criteria. Each application was scored by three distinct review panelists. Once scores were submitted, we averaged each application’s scores and selected the highest scoring applicants for implementation. See the section entitled RFP Process in Additional File 1 – Supplemental Information for more details.
Implementation Strategies - Community-Based Organization – Training
Northwestern project staff developed and disseminated online training to CBO implementers. Training modules covered topics ranging from using the implementer dashboard to recruitment strategies. Hosted on the KIU platform, the training includes seamless integration of KIU content. To ensure continuous improvement, Northwestern project staff offer ongoing technical assistance to meet emergent needs of the trial. See the section entitled Training in Additional File 1 – Supplemental Information for more details.
Implementation Strategies – Direct to Consumer Arm – Incentives
In the DTC arm, participants will receive a $10 or $25 pre-loaded virtual Visa gift card for completing the first three sessions of the intervention. Incentive amount is randomized by county in order to determine if a threshold amount must be met for incentives to be effective. DTC participants will also receive a discount code for an online store that sells adult toys and products. Participants will be entered into an e-raffle for various prizes donated by vendors as an incentive for completing the booster content at 3- and 6-months post-intervention. This strategy is based on those successfully used by CBO partners in previous KIU service implementations to keep their clients engaged.(34) Participants across both arms of the trial are entered into raffles for $200 gift cards when they complete research surveys at baseline, 3, 6, and 12 months post-intervention.
Measures
The RE-AIM model(47, 48) broadly guided our outcome measurement framework. Aim 1 will use quantitative data on Reach, Effectiveness, and Implementation, while Aim 2 will pull from mixed-methods data on Adoption and qualitative data to explore Maintenance and Sustainability. Outcomes will be measured across four domains: (1) reach to high risk YMSM; (2) effectiveness at reducing HIV risk among engaged YMSM;(47, 49, 50) (3) implementation variability;(51) and (4) cost for delivery.(52, 53) See Table 2: Reach to High Risk YMSM; Table 3: Effectiveness at Reducing HIV Risk among Engaged YMSM; Table 4: Variability in Implementation Success; and Table 5: Cost Elements for a list of outcomes measured by assessment time point.
Table 2
Level | Construct | Measure/ Operationalization | Measurement Schedule |
Screening | Baseline | Quarterly |
YMSM Participant | Reach | Proportion of YMSM in county screened for KIU | X | | |
Proportion of invited YMSM who begin KIU | X | X | |
Proportion of KIU Participants that are Black or Latino | | X | |
Proportion of KIU Participants with an STI at enrollment | | X | |
Proportion of KIU Participants who engaged in unprotected sex (no condom or PrEP) in prior 6 months | | X | |
CBO | Reach | Total number of HIV tests conducted | | X | X |
Number of HIV tests conducted with YMSM | | X | X |
Number of YMSM who tested non-reactive | | X | X |
Number of enrolled KIU Participants | | X | X |
Number of clients tested for Gonorrhea and/or Chlamydia | | X | X |
Number of Gonorrhea and/or Chlamydia tests conducted with YMSM | | X | X |
Table 3
Effectiveness at Reducing HIV Risk among Engaged YMSM
Outcome Type | Construct | Measure/ Operationalization | Measurement Schedule |
Baseline | 3- month | 6- month | 12- month |
Primary | HIV Risk Behavior | Number of condomless male assigned at birth (MAAB) sex partners | X | X | X | X |
Number of condomless MAAB sex partners who are casual | X | X | X | X |
Risk behavior associated with most recent sexual encounter: condoms, PrEP, substance use | X | X | X | X |
STI Incidence | Urethral and rectal Chlamydia and Gonorrhea - Aptima Combo 2 Assay on the Panther system(1) | X | | | X |
Secondary: Prevention Continuum | HIV/STI Testing | Assessing past 3-month HIV and STI testing history | X | X | X | X |
PrEP Use & Adherence | Current & past 3-month PrEP use; adherence | X | X | X | X |
Secondary: Substance Use | Alcohol Problems | AUDIT Alcohol Consumption Questions (AUDIT-C)(2); Full Alcohol Use Disorders Identification Test (AUDIT) branched to those scoring > 3(3) | X | X | X | X |
Other Drug Use | Past 3-month use of marijuana and illicit drugs | X | X | X | X |
1. Incorporated G-P. Aptima Combo 2 Assay for Panther System Package Insert. San Diego, CA2016. |
2. Bush K, Kivlahan DR, McDonell MB, Fihn SD, Bradley KA. The AUDIT alcohol consumption questions (AUDIT-C): an effective brief screening test for problem drinking. Ambulatory Care Quality Improvement Project (ACQUIP). Alcohol Use Disorders Identification Test. Arch Intern Med. 1998;158(16):1789-95. |
3. Saunders JB, Aasland OG, Babor TF, de la Fuente JR, Grant M. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption-II. Addiction. 1993;88(6):791–804. |
Table 4
Variability in Implementation Success
Level | Construct | Measure/ Operationalization | Measurement Schedule |
Base-line (YMSM) or Pre-Launch (CBO) | 3- month (YMSM) or 4-month (CBO) | 6- month | 12-month | 24-month |
YMSM Participant | Self-Efficacy | Confidence in HIV testing, condom use, and ability to get on PrEP | X | X | X | X | |
Resource Access | Access to HIV testing, HIV prevention services, and other sexual health services | X | X | X | | |
Stigma | Sexual health care avoidance due to LGBTQ stigma | X | | | | |
Satisfaction | Satisfaction with DTC or CBO staff and services | X | X | X | | |
Technology Center | Engagement | Mean number of KIU modules completed by Participants | | | | | |
System analytics; Metrics of activity within modules | | | | | |
CBO | Demographics | | X | | | | |
Effectiveness and Satisfaction with Training | Effectiveness and Satisfaction with Training | X | X | | | |
Normalization Process Theory | NoMAD: Normalization Process(1) | X | X | | X | X |
Organizational Readiness for Change | OCRBS: Organizational Change Recipient Beliefs Scale(2) | X | | | | |
Leadership Engagement | ILS: Implementation Leadership Scale(3) | X | X | | X | X |
Implementation Climate | ICS: Implementation Climate Scale(4) | X | | | X | X |
Readiness for eHealth | Readiness for eHealth (Developed from cited sources)(5–7) | X | | | | |
Sustainability | Program Sustainability Assessment Tool(8) | | | | X | X |
Adaptation | Items based on the Frame measure(9) | | | | X | X |
CFIR domains | Administrative Data | X | X | X | X | X |
Qualitative Interviews | X | | | X | |
1. Finch TL, Girling, M., May, C.R., Mair, F.S., Murray, E., Treweek, S., Steen, I.N., McColl, E.M., Dickinson, C., Rapley, T.. Nomad: Implementation measure based on Normalization Process Theory. [Measurement instrument]. Retrieved from http://www.normalizationprocess.org. 2015 [ |
2. Armenakis AA BJ, Pitts JP, Walker HJ. Organizational Change Recipients Beliefs Scale: Development of an Assessmetn Instrument. The Journal of Applied Behavioral Science 2007;42:481–505. |
3. Aarons GA EM, Torres EM, Finn NK, Roesch SC. Validation of the Implementation Leadership Scale (ILS) in substance use disorder treatment organizations. Journal of substance abuse treatment 2016;68:31 − 5. |
4. Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). 2014;9:157. |
5. Weichelt B. Health in Your Hand: Assessment of Clinicians’ Readiness to Adopt Mhealth into Rural Patient Care. Theses and Dissertations2016. |
6. Khatun F HA, Ray PK, Hanifi SMA, Bhuiya A, Liaw S-T. Determinants of readiness to adopt mHealth in a rural community of Bangladesh. International Journal of Medical Informatics. 2015;84(10):847 − 56. |
7. Khatun F HA, Ray PK, Bhuiya A, Liaw S-T. Community Readiness for Adopting mHealth in Rural Bangladesh: A Qualitative Exploration. International Journal of Medical Informatics. 2016;93:49–56. |
8. Luke DA CA, Robichaux CB, Elliott MB, Moreland-Russell S. The Program Sustainability Assessment Tool: A new instrument for public health programs. Preventing Chronic Disease. 2014;11. |
9. Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci. 2019;14(1):58. |
Table 5
Cost Type | Construct | Measures | Source | Measurement Schedule |
Start-Up | Recruitment | Staff time, consultant and other costs | Staff activity logs; study records and CBO interviews/surveys | Monthly for start-up period; one-time |
Initial Training | Number and duration of trainings | Study records | One-time |
Technology development | Staff time, content development costs | Staff activity logs | Monthly for start-up period; one-time |
Variable | Shipping and processing HIV and STI test kits | Number of units shipped, STI lab cost per test, staff time per shipment | Study records; staff interviews | Ongoing; one-time |
Participant contacts | Number of Participant contacts, staff time per contact | Participant contact logs; staff interviews | Ongoing; one-time |
Time-Dependent | Social Media Advertisement | Total spending | Study records | Ongoing |
Booster Training and Technical Assistance | Staff time by arm | CBO and DTC Participant tracking logs; staff activity logs | Ongoing; monthly |
Other staff activities | Staff time by arm | Staff activity logs; CBO interviews/surveys | Monthly; one-time |
Space and occupancy costs | Square feet | Staff interviews | One-time |
Data Analysis – Public Health Impact
Our primary outcomes are public health impact (PHI), defined as reach X effectiveness(47, 49, 50), and cost per infection averted.(54, 55) For this trial, PHI will be assessed by (1) reach into the county’s YMSM community, weighted by HIV risk, and (2) effectiveness at reducing HIV risk.(47, 49, 50) By measuring individual-level change on these modifiable factors, our index of PHI will allow for heterogeneity in response to KIU across individuals, race and age groups, counties, and implementation condition. Effectiveness will be marked by the estimated change in that person’s risk for HIV from baseline to follow-up surveys, determined by observed changes in target risk behavior: CAS, STIs, and adherent PrEP use, all of which have major impacts on HIV transmission in MSM.(56–58) We will base our modeling of HIV risk on published single-exposure probabilities and account for multiple exposures using binomial modeling.(59) Because these are individual-based measures, we will use two-level mixed-effect modeling.(60) Measuring cost is described below, and from these we will compute cost per infection averted, which is analogous to the measure used by CDC to decide which effective HIV prevention intervention would be supported as part of high impact prevention.(54, 55)
We will follow established guidelines for collecting cost data and conducting economic evaluations and will conduct cost analyses for each arm from the perspective of the healthcare sector.(52, 53) We will follow a micro-costing approach, a technique in which all inputs consumed in an intervention are identified and quantified in detail and then converted into fiscal terms to produce a cost estimate.
Data Analysis – Compare CBO and DTC Implementation Strategies
We will employ Multivariable Generalized Linear Mixed Models (GLMM) for our analyses, as they account for nested individual observations within counties and are commonly used in CRTs.(53, 61) The multivariable aspect of the model will allow for the control of factors that are unbalanced between arms either because they were not accounted for in the randomization process or because they may have become unbalanced due to loss to follow-up. To account for potential selection bias in constructing an efficient (1 degree of freedom) adjustment for measured differences in county-level covariates, we will adjust for the linear combination forming the first canonical covariate that maximally distinguished the sites in the two arms.(46, 62) Similar to propensity score analysis, we will also formally include the model that predicts selection effects.(63) We will estimate standard errors using nonparametric bootstrapping techniques within the multivariable framework treating arm as a dichotomous indicator.(53) The differences between the intervention arms will be evaluated by examining the statistical significance of the level-2 (i.e., cluster level) dichotomous indicator for the intervention arms. We will estimate and compare the difference in predicted mean cost per participant between the arms, and will also use those estimates of mean cost to characterize the cost per HIV infection averted in each arm of the study.
For cost-effectiveness analysis, parameters obtained from bootstrapping will be used to estimate cost-effectiveness acceptability curves which will indicate the probability that either intervention is a good value for different willingness-to-pay thresholds (i.e., incremental cost per infection avoided).(64) Finally, all analysis of outcomes will conform to best practices in analysis of randomized trials, including intention-to-treat analysis and sensitivity analysis of missing data and multilevel multiple imputation in order to examine the potential effect of missing responses on the results.(65–67)
Study Status
The KIU trial commenced in October 2019. As of October 2020, the DTC arm has begun recruitment in all 22 of its counties. The CBO arm has launched 20 counties for recruitment; the remaining two sites will initiate their programs in fall 2020. The CBO arm has gathered both qualitative and quantitative baseline data on implementation from 20 CBOs. See the section entitled CBO and DTC Study Statuses in Additional File 1 – Supplemental Information for additional details on study status for each arm.