Participants and Setting
Data were collected as part of an ongoing service contract between the UC San Diego team and the Indiana Division of Mental Health and Addictions (DMHA) to engage in the Leadership and Organizational Change for Implementation (LOCI) strategy to facilitate an implementation climate for evidence-based practice (EBP) in Indiana CMHCs [7–9]. Participants were providers (n = 93) from 6 CMHCs that are also contracted with Indiana DMHA to improve upon implementation of combined motivational enhancement therapy and cognitive behavioral therapy (MET/CBT) and other EBPs. Providers identified mostly as female (n = 77; 84.6%), non-Hispanic (n = 81; 89%), White (n = 79; 86.8%), and were 41 years old on average (sd = 14.8 years). A majority of providers had completed master’s level education (n = 65; 71.4%) and identified social work as their primary discipline (n = 44; 48.4%). On average, providers reported having worked with their present agencies for 4.7 years (sd = 7.7 years), and being in their current positions for 3.2 years (sd = 5.9 years). Providers reported spending the greatest percentage of their work time in psychotherapy and/or counseling (\(\stackrel{-}{x}\) = 44.1%) and reported an average caseload of 52.1 (sd = 39.5) clients per month. Administrative work (e.g., documentation, billing) (\(\stackrel{-}{x}\) = 18.1%) and case management (\(\stackrel{-}{x}\)= 12.0%) comprised the next greatest percentages of providers’ work time. See Table 1 for more information regarding provider demographics.
Table 1
Age (years; \(\stackrel{-}{x}\)±sd) | 41.0 ± 14.8 |
Gender | n | % |
Female | 77 | 82.8 |
Male | 13 | 14.0 |
Other | 1 | 1.1 |
Missing | 2 | 2.2 |
Race | n | % |
White | 79 | 84.9 |
Black or African American | 3 | 3.2 |
Asian | 2 | 2.2 |
American Indian/Alaska Native | 1 | 1.1 |
More than one race | 6 | 6.5 |
Missing | 2 | 2.2 |
Ethnicity | n | % |
Non-Hispanic | 81 | 87.1 |
Hispanic | 10 | 10.8 |
Missing | 2 | 2.2 |
Highest level of Education | n | % |
Some college | 1 | 1.1 |
College graduate | 14 | 15.1 |
Some graduate work | 5 | 5.4 |
Master’s degree | 65 | 69.9 |
PhD, MD, or equivalent | 6 | 6.5 |
Missing | 2 | 2.2 |
Primary Discipline | n | % |
Drug/Alcohol Counseling | 13 | 14.0 |
Social Work | 44 | 47.3 |
Child Development | 2 | 2.2 |
Marriage and Family Therapy | 2 | 2.2 |
Psychology | 16 | 17.2 |
Other | 14 | 15.1 |
Missing | 2 | 2.2 |
Providers per Agency | n | % |
Agency 1 | 7 | 7.9 |
Agency 2 | 15 | 16.9 |
Agency 3 | 11 | 12.4 |
Agency 4 | 11 | 12.4 |
Agency 5 | 43 | 48.3 |
Agency 6 | 2 | 2.2 |
Years at present agency (\(\stackrel{-}{x}\)±sd) | 4.7 ± 7.7 |
Years in present position (\(\stackrel{-}{x}\)±sd) | 3.2 ± 5.9 |
Percentage of your work time doing the following… | \(\underset{\_}{x}\)±sd |
Standardized assessments | 7.1 ± 10.6 |
Case management | 12.0 ± 18.6 |
Psychotherapy and/or counseling | 44.1 ± 28.6 |
Administrative work (e.g., documentation, billing) | 18.1 ± 11.9 |
Meeting with your supervisor | 7.2 ± 10.6 |
Supervising others | 4.3 ± 13.4 |
Travel | 2.1 ± 5.0 |
Other | 5.1 ± 14.5 |
Procedures And Measures
CMHC providers completed surveys via the Qualtrics web-based platform that included measures targeting their reactions to the COVID-19 outbreak, and transition to providing services via telehealth. The measures that were utilized are described below. When appropriate, the internal consistency of each measure was assessed for this sample using Cronbach’s α and is included in the description of each measure.
Perceptions of Personal Risk. This 9-item measure was adapted from Wu et al.’s measure assessing perceptions of personal risk around SARS [10]. Items were adapted to assess participants perceived risk of being exposed to, and getting infected with, COVID-19. Response options were also adapted such that participants responded using a 5-point response scale ranging from 0 = “Strongly Disagree” to 4 = “Strongly Agree.” Internal consistency for this measure was high at α = .87.
COVID-19 Rumination. This 3-item measure was developed by LeNoble and colleagues to explore the extent to which participants’ rumination about COVID-19 is interfering with their work [11]. Participants responded using a 5-point response scale ranging from 0 = “Strongly Disagree” to 4 = “Strongly Agree.” Internal consistency for this measure was high at α = .81.
Work Changes due to COVID-19. These three items were developed by the study authors through an iterative process of item generation, discussion, and refinement until consensus on item wording was achieved. The resulting three items assess the changes in tasks, settings, and teams that mental health providers experienced following the COVID-19 outbreak. Participants responded using a 5-point response scale ranging from 0 = “Not at all” to 4 = “Very great extent.” These items were developed to measure different types of changes that providers may experience, and not an underlying construct of work changes. Combining item responses into a single scale was not appropriate, and as such, internal consistency was not assessed. Items were analyzed individually to better understand the impact of COVID-19 on each individual type of change.
Burnout. The Copenhagen Work Burnout Inventory is 3-item measure developed to assess the extent to which participants have experienced emotional exhaustion and work-related frustration within the past two weeks [12]. Participants responded using a 5-point scale ranging from 0 = “Never” to 4 = “Always.” Internal consistency for this measure was high at α = .91.
Perceived Organizational Support. The 3-item perceived organizational support scale was developed to assess the extent to which respondents believe help is available to them from their agency, that their agency cares about their well-being, and their agency shows concern for them [13]. Participants responded using a 5-point response scale ranging from 1 = “Strongly Disagree” to 5 = “Strongly Agree.” Internal consistency for this measure was high at α = .93.
Telehealth Self-Efficacy. This 4-item measure was adapted from a measure developed by Lau and Brookman-Frazee to assess participant’s confidence, knowledge, understanding, and preparation to deliver therapy via telehealth [14, 15]. Participants responded using a 5-point response scale ranging from 0 = “Strongly Disagree” to 4 = “Strongly Agree.” Internal consistency for this measure was high at α = .92.
Collective Efficacy. This 3-item measure was adapted from Jex and Bliese to assess efficacy beliefs targeting the agency’s transition to telehealth [16]. Participants responded using a 5-point response scale ranging from 0 = “Strongly Disagree” to 4 = “Strongly Agree.” Internal consistency for this measure was moderately high at α = .76.
Telehealth Beliefs. This 5-item measure was adapted from the University of Michigan’s Behavioral health Workforce Research Center to assess beliefs regarding the importance of telehealth [17]. Participants responded using a 5-point response scale ranging from 0 = “Strongly Disagree” to 4 = “Strongly Agree.” Internal consistency for this measure was high at α = .85.
Transition to Telehealth. Seven items evaluating the transition to telehealth were developed by the study authors through an iterative process of item generation, discussion, and refinement until consensus on item wording was achieved. These items asked participants to indicate the extent to which different aspects of treatment were better or worse when serving clients via telehealth as opposed to in-person treatment; see Table 4 for the individual items. Participants responded using a 5-point response scale ranging from 0 = “Significantly worse with telehealth relative to in-person” to 5 = “Significantly better with telehealth relative to in-person.” Internal consistency for this measure was high at α = .82.
Table 4
Transition to telehealth descriptive statistics.
| Minimum | Maximum | Mean | Std. Deviation |
Relationships between you and your patients/clients. | 1 | 5 | 2.80 | 0.71 |
Quality of communication between you and your patients/clients. | 1 | 5 | 2.57 | 0.81 |
Rate of no-shows with fewer being better. | 1 | 5 | 2.81 | 1.22 |
Patient/client focus during sessions. | 1 | 5 | 2.38 | 0.85 |
Patient/client engagement in treatment. | 1 | 5 | 2.71 | 0.88 |
Confidentiality of discussions with patients/clients. | 1 | 5 | 2.82 | 0.78 |
Patient/client willingness to schedule sessions. | 1 | 5 | 3.12 | 1.0 |
*Responses ranged from 0 = “Significantly worse with telehealth relative to in-person” to 5 = “Significantly better with telehealth relative to in-person.” |
Open-ended survey questions. Participants also responded to two open-ended survey items to explore the impacts of COVID-19. The first asked “What have been the major impacts of COVID-19 on your work?” The second asked “What have been the major impacts of COVID-19 on the implementation of MET/CBT and other EBPs?”
Analysis
Participant responses were aggregated across all items to obtain an overall scale mean with the exception of the Work Changes and the Transition to Telehealth items, which were analyzed individually. Descriptive statistics of quantitative measures were assessed to explore providers’ responses and/or reactions to the COVID-19 outbreak, and subsequent transition to providing services via telehealth. Potential between agency differences in provider responses were explored using univariate analysis of variance (UNIANOVA). Provider responses to open-ended survey questions were first reviewed by authors (MS, KR, and KC) to gain familiarity with the content and to isolate broad themes. Text was then sorted and organized in accordance with broad themes, and new themes were generated when appropriate. All authors held meetings to review “chunks” [18] of text and develop summaries of findings.