Study Setting: NIATx200
The NIATx200 initiative built on prior successful NIATx research [4, 27-30]. NIATx200 evaluated the effectiveness of implementation strategies commonly used a QIC. To achieve this objective, NIATx200 recruited 201 addiction treatment clinics in five states (Massachusetts, Michigan, New York, Oregon, and Washington). Clinic eligibility criteria included: 60+ admissions per year, outpatient or intensive outpatient levels of care as defined by the American Society of Addiction Medicine (ASAM); and received some public funding in the past year [31]. Clinics, randomized within states, were stratified by size (number of patients per year) and management score [32] and assigned to one of four implementation strategies: (1) interest circle calls (n=49), (2) learning sessions (n=54), (3) coaching (n=50), or (4) a combination of all three implementation strategies (n=48). The NIATx200 initiative consisted an 18-month active implementation timeframe. During three distinct implementation periods lasting 6 months, participating clinics implemented organizational changes designed to improve wait time (mean days between first contact and first treatment), retention in treatment (percent of patients retained from first to fourth treatment session), and annual admissions. Data was also collected at the staff level about their perceptions associated with organizational readiness for change and sustainability propensity. The structure of the NIATx200 initiative and the description of the implementation strategies are described in more detail elsewhere [31, 33, 34].
Mixed-effect regression models determined which implementation strategy was most effective in improving outcomes, as well as being most cost-effective [31]. Improvements in the wait time and admission outcomes for clinics assigned to the coaching and combination strategies significantly differed from clinics assigned to the interest circle strategy and the coaching strategy was the more cost-effective as compared to interest circles [34]. Although no NIATx implementation strategy significantly improved treatment retention (as defined), an exploratory analysis, accounted for early treatment drop-off (i.e., a client not making it to the first treatment session) when measuring retention, showed clinic-level improvements for providers assigned to the coaching, combination and learning session implementation strategies which suggest that how retention was defined impacted the findings [34]. Results from this exploratory analysis clearly indicated that clinic participation in the three intervention (i.e., learning sessions, coaching, and the combination arm) improved the outcomes, with coaching being the most cost-effective strategy.
Although differences in clinic attributes did not affect improvements in the outcomes examined in other studies, organizational characteristics were included in these secondary data analyses. Organizational characteristics comprised: (1) non-profit status, (2) whether the clinic was free-standing Alcohol and Drug Abuse Treatment Program or part of a healthcare system, (3) whether the clinic had received accreditation from a national organization such as the Joint Commission on Accreditation of Healthcare Organizations or the Commission on Accreditation of Rehabilitation Facilities, and (4) the metropolitan statistical area (rural or urban status).
Implementation Strategies
The structure of the four NIATx200 implementation strategies represented clinic participation levels. Interest circles involved monthly multi-clinic teleconferences for a total of 18 direct contact hours (18 calls, each one hour in length), and allowed change teams from participating clinics to receive advice from peers and learn new skills. The learning session strategy consisted of three face-to-face multi-day sessions held approximately every six months, which were led by a core faculty team and utilized a common curriculum to offer didactic and experiential learning opportunities. The first learning session consisted of 8.5 hours of content delivered over a single day, while another 13 content hours were delivered over 1.5 days during each of the second and third learning sessions, resulting in a potential for 34.5 total direct contact hours. Clinics assigned to the coaching strategy received a one-day, 4-hour, site visit, as well as participated in monthly one-hour coaching calls; 22 direct contact hours were possible for the coaching strategy. On the calls, the coach and change leader, executive sponsor, and change team reviewed the impact of organizational changes to improve the study outcomes, discussed successes, and identified ideas for future change projects. The combination strategy involved the interest circle calls, coaching, and learning sessions, and consisted of a cumulative possibility of 74.5 direct contact hours. NIATx200 results indicated that clinics assigned to interventions with higher participation hours, where interest circles were the referenced intervention, showed greater improvements in wait time and admissions. As such, staff in the clinics assigned to the interventions with more opportunities to participate in the intervention and be exposed to sustainability concepts would have higher perceptions about the likelihood that changes would be sustained.
Outcomes and Measurement
The NIATx200 initiative utilized the British National Health Services Sustainability Index (BNHS-SI) to assess staff perceptions about the likelihood that a change will be sustained in the organization [23, 24]. The BNHS-SI has been utilized across multiple healthcare settings to assess staff perceptions about the sustainability of an organizational change [24, 35-45] and as a qualitative framework to qualitatively identify factors associated with the concept of sustainability [44, 46-48].
The tool (see Additional File 2 for questions) consists of 10 factors designed to assess overall staff perceptions about sustainability as well as their perceptions across three domains:
- Process– benefits beyond helping patients, credibility of the benefits, adaptability of the improved process, and effectiveness of systems to monitor progress.
- Staff– staff involvement and training to sustain the process, staff attitudes toward sustaining the change, senior leadership engagement, and clinical leadership engagement.
- Organization– fit with organization’s strategic aims and culture, and infrastructure for sustainability.
The BNHS-SI utilizes an additive, multi-attribute, utility model to summarize the scores across the three domains (see Additional File 3) which are then totaled to arrive at an overall organization sustainability propensity score [24].
In the NIATx200 initiative, a staff sustainability survey [31] was developed and distributed at baseline and at every subsequent 9-month period (see Figure 1) to prospectively assess clinic staff perceptions about sustainability capacity. The BNHS-SI measures the likelihood that a change will be sustained; therefore, it does not rely on a set sustainability definition (e.g., clinic continued to maintain the intervention after funding ended) when asking staff to complete the instrument. Instead, survey instructions stated that the BNHS-SI was “designed to gauge your organization’s propensity for sustaining changes”. As such staff were asked to “think about one specific change implemented as part of the NIATx200 project”; and then select one of four options for each of the 10 factors that best describes sustainability in their organization. Our team utilized a similar approach when assessing sustainability capacity within the Veterans Administration [38-40].
For this analysis, the cumulative extent of staff beliefs that changes implemented as part of the NIATx200 initiative would be sustained (called the Total Sustainability Score) was the primary outcome. Three secondary outcomes also were evaluated – representing scores from the process, staff, and organization domains from the BNHS-SI tool (called Process, Staff, and Organization Domain Scores, respectively).
Data Collection
Staff were invited to complete a paper survey or use a link in the invitation letter to complete the survey online. The survey also collected staff demographic information related to job function, employment status, and tenure within the organization. Two additional questions (i.e., “What is the first initial of your mother’s maiden name?” and “On what day of the month is your birthday?”) were combined with staff demographic characteristics and the clinic ID to create a unique identifier for individual staff members that allowed matching of individual survey responses to be tracked over time.
Clinic participation (direct contact hours) in the assigned implementation strategy and the number of persons from the clinic participating in the assigned implementation strategy were recorded in real time by NIATx200 research staff and coaches.
Design and Sample
The unique identifier was utilized to match individual survey responses across the four different time points (Figure 1). As a result, all analyses are based on responses from the same staff members (n=908, representing 2,329 total cases) across the evaluation timeframe.
An important variable for this analysis is each clinic’s cumulative level of staff participation in QIC activities throughout the 27-month intervention interval (at baseline and approximately every nine months) (called Total Participation). Participation in each of the four study interventions was measured separately but, for the purpose and this study, was aggregated into a Total Participation metric. This variable is used to determine the influence of the number of encounters with the implementation strategy during the 27-month period. Although this factor does not represent the total number of staff who took part in each activity, and therefore reflects a clinic-level influence, it remains dependent on overall staff involvement. As such, degree of staff participation is considered appropriate and relevant, and is retained for this sample.
Analysis
Analysis comprised both simple descriptive statistics and multivariate model building. Descriptive statistics were calculated for all variables used for this study. The type of descriptive values depended on whether the variables were continuous or categorical. For continuous variables, the mean, standard deviation (SD), and min/max are reported, while the frequencies of each category are provided for the categorical variables. Bivariate analyses were conducted on the primary outcome measure (Total Sustainability Score) and each anticipated study variable before entry into model; all variables used in the model demonstrated a significant independent association with the sustainability total.
The multivariate method was a linear mixed model repeated measures analysis that fit three separate statistical models to assess potential predictors of staff-level Total Sustainability Score, as well as on Process, Staff and Organization Domain Scores. For each model, a Repeated Covariance Type – A1(1): Heterogeneous – was used, which assumes different variances at each measurement time as well as correlations across time points that become weaker over those successive assessment times. All variables were entered into the models as fixed effects, and a maximum likelihood method was used to estimate the variable parameters. The statistical models are as follows:
- Model I – containing only the variable representing the four time points during the NIATx200 initiative (Time),
- Model II – containing Time, plus NIATx200-provided Implementation Strategies (i.e., learning sessions, interest circle calls, coaching sessions, or service combinations) and Job Function (i.e., administrative vs. clinical), and
- Model III – containing the variables from Models I and II, plus organizational characteristics and the cumulative extent of participation in NIATx200-provided strategies (i.e., total number of hours).
IBM SPSSv26® was used to calculate all descriptive statistics and to estimate each model by calculating the parameter estimates for fixed effects at 95% confidence intervals. This study is reported in full accordance with the StaRI checklist [Additional File 4] [49].