Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

doi:10.21203/rs.3.rs-20758/v1

Download PDF

Short report

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

https://doi.org/10.21203/rs.3.rs-20758/v1

This work is licensed under a CC BY 4.0 License

Journal Publication

published 08 Feb, 2021

Read the published version in Implementation Research and Practice →

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Background Usability – the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction – is a key determinant of implementation success. However, usability is rarely assessed in implementation research and no instruments have been developed to measure the design quality of complex health interventions, such as the evidence-based psychosocial interventions that characterize the majority of effective practices in mental and behavioral health services. This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale for digital technologies, when measuring the usability of complex health interventions. Prior studies of the original System Usability Scale have found both one- and two-factor solutions, both of which were examined in the current study of the IUS.

Methods A survey was administered to 205 healthcare professionals working at 11 primary care sites. Surveys collected demographic information, including each participant’s professional role (i.e., medical provider, mental/behavioral health provider, pharmacist), and IUS ratings for one of six common evidence-based psychosocial interventions (e.g., cognitive behavioral therapy, motivational interviewing) that they reported using most regularly. Factor analyses replicated the procedures used in prior research on the System Usability Scale, and a sensitivity analysis using analyses of variance compared IUS scores across different groups of respondents and interventions assessed.

Results Analyses indicated that a two-factor solution (with “usable” and “learnable” subscales) in which one item was removed best fit the data. This solution accounted for 52.6% of the variance observed. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .82, and α = .63, respectively. Resulting scores indicated that usability ranged from below acceptable standards to good, depending on the intervention. On average, behavioral health providers found the interventions to be more usable that other types of healthcare providers.

Conclusions The current study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability and sensitivity to role and intervention. Future directions for implementation research evaluating the usability of complex health interventions are discussed.

Psychology

Complex health interventions

Psychosocial interventions

Mental health

Usability

Human-centered design

The ease with which interventions can be readily adopted by service providers is a key predictor of implementation success, but very little implementation research has attended to intervention usability.
No instruments exist to evaluate the usability of complex health interventions, such as the evidence-based practices that are commonly used to integrate mental and behavioral health services into primary care.
The current study evaluated the first instrument for assessing the usability of complex health interventions and found that its factor structure replicated some research with the original version of the instrument, a scale developed to assess the usability of digital systems.

Although complex health interventions (CHIs)—those with several interacting components—are common in contemporary healthcare worldwide,[1, 2] clinicians often face considerable challenges implementing these interventions, owing to their design complexity.[3, 4] For this reason, intervention usability – the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction[5] – has been identified as a key “upstream” determinant of successful implementation.[6] This is particularly true in behavioral health (consistent with literature in the United States, we use “behavioral health” to include mental health and substance abuse services,[7]) where most effective practices are complex evidence-based psychosocial interventions (EBPIs).[8] Frequently, EBPIs are delivered in integrated or non-specialty care settings (e.g., primary care, schools) that differ markedly from the contexts where they were originally developed (e.g., academic medical settings). For instance, mismatches between the design of EBPIs and the constraints of primary care make intervention fidelity and patient outcomes difficult to sustain.[9–11] Despite being critical to implementation, aspects of intervention design quality, such as usability, have been insufficiently assessed.[12] No instruments exist to assess EBPI usability, greatly limiting implementation research.[13]

Human-centered Design (hcd)

Human-centered design (HCD; a.k.a., user-centered design) provides methods to evaluate the usability of CHIs in general, and EBPIs in particular.[12, 14] HCD is an approach focused on developing compelling and intuitive products, grounded in knowledge about the people and contexts where an innovation will ultimately be deployed.[15, 16] No instruments exist to evaluate the usability of CHIs.

System Usability Scale (SUS). Originally designed for assessing digital systems, the System Usability Scale (SUS) is among the most widely applied usability assessment instruments[17]; it has been found to be significantly related to task success in both laboratory and field studies[18]. Brooke[19] initially developed the 10-item scale as a brief and reliable instrument to allow comparisons among products and across versions of a product (e.g., a computer software program). Bangor et al.[20] evaluated 2324 SUS questionnaires and found high internal reliability (a = .91) and good sensitivity and concurrent validity. Although the instrument was originally considered unidimensional, Lewis & Sauro[17] assessed the SUS factor structure using 2648 questionnaires assessing products such as computer hardware, software, and websites. They identified an eight-item “Usable” subscale and a two-item “Learnable” subscale, a finding which provoked several replication studies,[21–25] including one analyzing over 9,000 surveys.[26] Unfortunately, these studies did not replicate the initial findings. Most studies found a two-factor solution, with factors defined by whether the item was positively worded or negatively worded, which Lewis and Sauro interpreted as practically unidimensional.[26] The current study replicates these factor analyses, but using a different population (healthcare professionals instead of consumers) and different target (EBPIs instead of computer hardware/software).

Current Aims

In light of the criticality of CHI usability, and the paucity of psychometrically-sound instruments for its evaluation, we revised the SUS to evaluate EBPI usability. The current study assessed the revised instrument’s structural validity using confirmatory factor analytic methods. This study was carried out with primary care providers given the importance of primary care for the delivery of contemporary mental health services worldwide[27–29] and the likelihood that many traditional EBPIs may demonstrate problematic usability in that intervention context. Consistent with Lewis and Sauro[17], we also conducted a sensitivity analysis to determine if scores on the IUS varied by types of EBPI and professional role.

Participants

Table 1 provides sample demographics. The study sample consisted of 205 healthcare professionals, including 144 medical providers, 40 behavioral health providers (including psychologists, counselors, social workers, and therapists), and 11 pharmacists.

Table 1

Study sample demographics.
	N	%
Age
18–29	47	22.9%
30–39	78	38.0%
40–49	34	16.6%
50–59	21	10.2%
60–69	15	7.3%
Missing	10	4.9%
Gender (self-reported)
Female/woman	113	55.1%
Male/man	73	35.6%
Genderqueer/non-binary	3	1.5%
Missing	16	7.8%
Race/Ethnicity*
American Indian/Alaska Native	6	2.9%
Asian	17	8.3%
Black/African American	2	1.0%
Hispanic/Latinx	3	1.5%
Native Hawaiian/Pacific Islander	1	0.5%
White	169	82.4%
Other	5	2.4%
Professional role
Medical provider	144	70.2%
Behavioral health provider	40	19.5%
Pharmacist	11	5.4%
Missing	10	4.9%
Degree(s) held*
MD	124	60.5%
DO	12	5.9%
NP	5	2.4%
PA	3	1.5%
PhD	10	4.9%
PsyD	19	9.3%
MSW	19	9.3%
MPH	11	5.4%
MS/MA	13	6.3%
PharmD	11	5.4%
BS/BA	54	26.3%
Other	4	2.0%
Years in practice
0–5	25	12.2%
6–10	26	12.7%
11–20	24	11.7%
21–30	18	8.8%
31 or more	7	3.4%
Missing	105	51.2%
Intervention most commonly used
Acceptance and Commitment Therapy	2	1.0%
Behavioral Activation	5	2.4%
Cognitive Behavioral Therapy	19	9.3%
Motivational Interviewing	172	83.9%
Problem Solving Therapy	4	2.0%
Trauma Treatment	3	1.5%
N = 205. *Racial/ethnic categories and degrees held are not mutually exclusive.

Procedures

The data in the current study were collected as a component of a larger survey completed by the WWAMI (Washington, Wyoming, Alaska, Montana, Idaho) region Practice and Research Network (WPRN) in January 2019, a group of primary care clinics committed to conducting research that informs primary care practice. At the time of the survey, WPRN clinics were spread across 25 diverse parent organizations in both urban and rural communities. This survey was originally conducted for quality improvement purposes of the WPRN; this study’s analyses of the dataset was approved by the University of Washington Institutional Review Board. The full survey consisted of questions in four areas: (1) clinician role/demographics; (2) perspectives on behavioral health in primary care; (3) anxiety treatment; and (4) provider burnout/resilience. Survey completion took approximately ten minutes.

All WPRN member organizations (n = 25) were invited to participate. Consistent with standard practices in the WPRN, staff and faculty held webinars with WPRN clinics and a preview of the full survey was distributed to build engagement among WPRN site champions and secure approval from leadership at each clinic. Eleven clinic sites chose to participate. The survey was programmed into the REDCap[30] digital data collection system and distributed by WPRN clinic champions to 491 providers representing three categories of primary care clinicians: medical providers (MD/DO, PA, NP), behavioral health providers (social workers, licensed addiction counselors, psychologists, marriage and family therapists), and clinical pharmacists. Participation was voluntary and no incentives were provided. Participating sites received aggregated results. The overall survey response rate was 56.8% (n = 279). 205 (73.4%) participants were qualified to answer the IUS questions, because they reported that they used at least one of the six identified behavioral health interventions (see below).

Measures

Intervention Usability Scale (IUS). The 10-item SUS[19] was adapted to create the IUS.[14, 31] The term “system” was replaced with the word “intervention” in each item, but no other modifications were made. Items are rated on a Likert scale from 0 (strongly disagree) to 4 (strongly agree), with half of the items reverse-scored. A total score is normally calculated by multiplying the sum of these scores by 2.5 (range: 0-100). However, as described below, we ultimately removed one item from the measure as a result of factor analyses and therefore multiplied the overall IUS by 2.78, so that the possible range of scores remained 0-100. This ensured consistency with prior studies using the SUS.

Intervention type. Participants were asked to identify the type of behavioral health intervention that they most commonly used from a list of frequently used interventions in the region. The list of possible approaches included Acceptance and Commitment Therapy (ACT), Behavioral Activation (BA), Cognitive Behavioral Therapy (CBT), Motivational Interviewing (MI), Problem-Solving Therapy (PST), and trauma treatment.

Analyses

To allow for direct comparability of studies, we replicated the procedures of Lewis & Sauro[17, 26]. To replicate the 2009 study[17], we used principal axis factoring (a.k.a., common factor analysis) and, to replicate the 2017 study[26], we used principal components and unweighted least squares factor analysis to examine the factor structure of the IUS and evaluate possible subscales. All analyses used varimax rotation. We then assessed scale correlations and internal consistency for each subscale and the total score. We compared IUS descriptive statistics with prior research on the SUS. Finally, we examined the sensitivity of the IUS by comparing IUS scores across different groups using analyses of variance (ANOVAs) based on provider type and intervention type. Assuming that certain intervention types are inherently more usable than others, and/or that certain types of providers are more prepared and oriented toward the delivery of these interventions, differences between these groups would support IUS construct validity.

Item Descriptives and Correlations

Scores on IUS items included the full range (0–4) for all items besides item 5 (range: 1–4). The mean scores for items ranged from 0.95–3.19; median scores were 3 for all regularly scored items and 1 for all reverse scored items. See Table 2 for more detail on individual item descriptive statistics. Inter-item correlations absolute values (half of the IUS items are reverse scored) ranged from 0.06 to 0.60. Only three pairs of items were not significantly correlated: items 1 and 7, items 1 and 10, and items 5 and 10. See Table 3 for full item correlation results.

Table 2

IUS individual item descriptives.
Item	Min	Max		Mean	SD
1. I like to use [X] frequently.	0	4	3	3.18	0.71
2. I find [X] unnecessarily complex.	0	4	1	1.04	0.78
3. I think [X] is easy to use.	0	4	3	2.68	0.90
4. I need the support of an expert consultant to be able to use [X].	0	4	1	0.95	0.82
5. I find the various components of [X] are well integrated.	1	4	3	2.80	0.73
6. I think there is too much inconsistency in [X].	0	4	1	1.12	0.84
7. I would imagine that most people would learn to use [X] very quickly.	0	4	3	2.45	0.98
8. I find [X] very cumbersome to use.	0	4	1	1.04	0.82
9. I feel very confident using [X].	0	4	3	2.68	0.88
10. I needed to learn a lot of things before I could get going with [X].	0	4	1	1.45	0.92
N = 205 for all items. [X] refers to the mental health intervention the participant indicated they deliver most frequently.
For all items, 0 = strongly disagree, 1 = disagree; 2 = neither agree nor disagree; 3 = agree; 4 = strongly agree.
Items in bold are reverse scored.

Table 3

Item Correlations
	1. I like to use [X] frequently.	2. I find [X] unnecess-arily complex.	3. I think [X] is easy to use.	4. I need the support of an expert consultant to be able to use [X].	5. I find the various components of [X] are well integrated	6. I think there is too much inconsistency in [X].	7. I would imagine that most people would learn to use [X] very quickly.	8. I find [X] very cumbersome to use.	9. I feel very confident using [X].	10. I needed to learn a lot of things before I could get going with [X].
1. I like to use [X] frequently.	-	-	-	-	-	-	-	-	-	-
2. I find [X] unnecessarily complex.	-0.39***	-	-	-	-	-	-	-	-	-
3. I think [X] is easy to use.	0.32***	-0.47***	-	-	-	-	-	-	-	-
4. I need the support of an expert consultant to be able to use [X].	-0.18**	0.37***	-0.29***	-	-	-	-	-	-	-
5. I find the various components of [X] are well integrated.	0.34***	-0.37***	0.51***	-0.17**	-	-	-	-	-	-
6. I think there is too much inconsistency in [X].	-0.15*	0.46***	-0.28***	0.33***	-0.36***	-	-	-	-	-
7, I would imagine that most people would learn to use [X] very quickly.	0.11	-0.28***	0.57***	-0.19**	0.36***	-0.19**	-	-	-	-
8. I find [X] very cumbersome to use.	-0.35***	0.60***	-0.52***	0.36***	-0.36**	0.53***	-0.41***	-	-	-
9. I feel very confident using [X].	0.40***	-0.40***	0.54***	-0.41***	0.44***	-0.25***	0.38***	-0.42***	-	-
10. I needed to learn a lot of things before I could get going with [X].	-0.06	0.16*	-0.16**	0.42***	-0.09	0.18***	-0.21**	0.22**	-0.22**	-
N = 205.

Table 4

Rotated Component Matrices for three exploratory structure analyses.
Item	Principal axis		Principal components		Unweighted least squares
	Factor 1	Factor 2	Factor 1	Factor 2	Factor 1	Factor 2
1. I like to use [X] frequently.	.450	− .129	.608	.030	.450	− .127
2. I find [X] unnecessarily complex.	− .572	.380	− .651	.340	− .575	.375
3. I think [X] is easy to use.	.782	− .157	.789	− .157	.781	− .156
4. I need the support of an expert consultant to be able to use [X].	− .182	.751	− .209	.785	− .182	.759
5. I find the various components of [X] are well integrated.	.646	− .080	.738	.001	.646	− .078
6. I think there is too much inconsistency in [X].	− .384	.383	− .427	.450	− .387	.377
7. I would imagine that most people would learn to use [X] very quickly.	.534	− .138	.579	− .173	.534	− .137
8. I find [X] very cumbersome to use.	− .622	.405	− .671	.393	− .626	.400
9. I feel very confident using [X].	.596	− .306	.667	− .279	.596	− .305
10. I needed to learn a lot of things before I could get going with [X].	− .087	.478	.023	.805	− .089	.477

N = 205.

Ius Factor Structure

Examining the point of inflection on a scree plot indicated that a two-factor solution for the IUS best fit the data. Factor 1 had an eigenvalue of 4.0 and accounted for 40.5% of the variance, and Factor 2 had an eigenvalue of 1.2 and accounted for 12.1% of the variance, for a total of 52.6% of variance accounted for in the two-factor solution. For all three approaches to analysis, items 1, 2, 3, 5, 7, 8, and 9 loaded onto the first factor, whereas items 4 and 10 loaded onto the second. Item 6, “I think there is too much inconsistency in [intervention],” loaded almost equally onto both factors, and was therefore removed from later subscale analyses. See Table 4 for the rotated component matrix. The item-factor alignment was nearly identical to that of Lewis and Sauro[17], which found a two-factor structure, but inconsistent with studies that followed[26]. Therefore, we named these factors the same as in the 2009 study[17]; “Usable” and “Learnable.” To place the Usable and Learnable scores on a comparable 0 to 100 scale as the Overall IUS score, we multiplied their summed score contributions by 3.57 and 12.50, respectively.

Scale Correlations

The correlations between the subscales and the Overall IUS score were r = .957 for Usable (p < .001) and r = .630 for Learnable (p < .001). The strong scale-to-total correlation for Usable was expected, given that 7 out of 9 items were included in this subscale. The correlation between Usable and Learnable was moderate at r = .377 (p < .001).

Reliability/internal Consistency

The overall IUS items had good internal consistency (α = .83). Coefficient alphas for Usable and Learnable were α = .82 and α = .63, respectively. Only two items loaded onto Learnable, contributing to its low alpha. The Learnable subscale did not have sufficient reliability to meet the typical minimum standard of .70[32, 33].

Sensitivity

Intervention type. The majority of providers (83.9%) reported most commonly using MI, with the next largest group (9.3%) reporting that they used CBT (Table 1). We conducted an ANOVA with most commonly delivered intervention as an independent variable with 3 levels (MI, CBT, and other). ANOVAs revealed significant differences by type of intervention on the Total IUS score and the Usable subscale. MI and CBT appeared to be less usable than the aggregated list of other types of therapy. The average adjusted IUS score for those who delivered MI was 69.1, CBT was 71.7, and “other” was 81.8 (F_{(2, 202)} = 6.06; p = .003). The average score on the adjusted Usable subscale for those who deliver MI was 68.8, CBT was 73.5, and “other” was 83.9 (F_{(2, 202)} = 7.93; p < .001). There were no significant differences between intervention types on the Learnable subscale: for those delivering MI the average score was 70.0; CBT was 65.1, and “Other” was 74.1 (F_{(2, 202)} = 0.93; p = .397).

Professional role. There were significant differences on the overall IUS and the Usable subscale depending on the respondent’s professional role. Behavioral health providers (M = 77.2) had higher scores than both medical providers (M = 68.5) and pharmacists (M = 66.0) on the overall IUS (F_(2,192) = 7.30; p = .001). Behavioral health providers (M = 79.0) also scored significantly higher on the Usable subscale compared to medical providers (M = 68.0) and pharmacists (M = 66.2; F_{(2, 192)} = 9.93; p < .001). There were no significant differences between behavioral health providers (M = 70.6), medical providers (M = 69.9) and pharmacists (M = 64.8) on the Learnable subscale (F_{(2, 192)} = 0.40; p = .670).

Distribution Of Ius Subscale Scores

Table 5 shows descriptive statistics on distributions of scores on the IUS, alongside a comparison with prior SUS data[17]. Of note, the current sample had significantly higher overall scores (t = 8.58; p < .001) and Usable scores (t = 10.63; p < .001) on average, whereas the 2009 SUS study had slightly yet significantly higher Learnable scores (t = -2.16; p = .032).

Table 5

Comparison with prior System Usability Scale data.
	Lewis & Sauro SUS (comparison sample)			Current Dataset
Statistic	Overall	Usable	Learnable	Overall	Usable	Learnable
N	324	324	324	205	205	205
Minimum	7.5	0.0	0.0	30.58	32.13	0.00
Maximum	100	100	100	100	100	100
Mean	62.10	59.40	72.70	70.33**	70.25**	69.81*
Variance	494.38	531.54	674.47	184.11	212.68	365.24
Standard Deviation	22.24	23.06	25.97	13.57	14.58	19.13
Standard Error	1.24	1.28	1.44	0.95	1.01	1.32
Skewness	− .42	− .38	− .80	− .11	− .18	− .55
Kurtosis	N/A	N/A	N/A	− .11	− .18	− .55
Significantly different from comparison sample at p = .05; *Significantly different from comparison sample at p = .001.

No psychometrically-sound instruments exist to evaluate the usability of CHIs. Our study of the IUS revealed a factor structure that was nearly identical to a SUS study from 2009[17] with exception of item 6 (“I think there is too much inconsistency in [Intervention]), which we removed. However, our results differed from subsequent studies[21–26]. The moderate correlation between the subscales indicates that the measure can be used as a total scale score, as well as decomposed into Usable and Learnable subscales.

Comparisons of IUS scores across interventions and providers yielded notable differences, especially that “Other” interventions were found to be more usable than MI or CBT. This may be because some of the interventions combined into the “Other” category (such as BA)

are simpler than MI and CBT (e.g., omitting a focus on cognitions), but further research should apply more extensive usability assessment methods[14] to better understand the EBPI qualities that result in higher and lower scores. Future research also may assess the extent to which differences in implementation supports (e.g., training, consultation) impact experiences of EBPI usability. Additionally, behavioral health providers found all interventions more usable than other types of providers. This is consistent both with SUS findings that greater experience with products results in higher scores[34, 35] and with a prior application of the IUS in which more expertise in a clinical service domain was associated with higher scores[14]. However, there was no difference by provider type on intervention learnability, with all disciplines finding the learnability of EBPIs to be slightly below the SUS cutoff of 70 for “passable”[20]. It may be that while behavioral health providers found the interventions more usable over time, given that they had expertise in the domains and potentially more supports for practice, they still found them initially complex to learn. It is also possible that learnability did not demonstrate significance because it contained only two items with lower reliability.

This study had several limitations. First, it occurred only in primary care and may be difficult to generalize to other contexts. Second, due to a survey programming error, we did not obtain data about the amount of training respondents had received for the intervention they indicated delivering most frequently. Finally, as indicated above, we did not collect any information in the survey regarding what EBPI qualities drove ratings.

Overall, the adapted IUS demonstrated good psychometric quality and a structure consistent with some prior research. This consistency may bode well for growing collaborations between implementation and HCD researchers[6]. Given that the Learnable subscale was composed of two items and did not demonstrate differences by provider type, additional items may be indicated to create a more robust subscale. Overall, intervention usability has been conceptualized as a key determinant of both perceptual (e.g., appropriateness, feasibility) and behavioral (e.g., adoption, fidelity, reach) implementation outcomes, as well as patient outcomes[36]. Application of the IUS to a broader range of EBPIs, settings, and professional roles would allow this proposition to be explicitly tested.

ACT

Acceptance and Commitment Therapy

Behavioral Activation

CBT

Cognitive Behavioral Therapy

CHI

Complex health intervention

EBPI

Evidence-based psychosocial intervention

HCD

Human-centered design

IUS

Intervention Usability Scale

Motivational Interviewing

PST

Problem-Solving Therapy

SUS

System Usability Scale

WPRN

WWAMI region Practice and Research Network

WWAMI

Washington, Wyoming, Alaska, Montana, and Idaho

Ethics Approval and Consent to Participate

This project was approved by the University of Washington Institutional Review Board (IRB).

Consent for Publication

Not applicable.

Availability of data and material

Please contact the lead author for more information.

Competing Interests

All authors declare that they have no competing interests.

Funding

This publication was supported in part by grants P50MH115837 (Areán) and R34MH109605 (Lyon), award by the National Institute of Mental Health. This publication was also supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1 TR002319. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Authors’ Contributions

ARL developed the Intervention Usability Scale. ARL, KO, BNR, SAM, and PAA developed the study survey. KO programmed the online survey and served as the liaison to the participating WPRN organizations. JJ and MDP conducted all data analyses. ARL drafted the initial manuscript outline. ARL, JJ, MDP, and KO drafted sections of the manuscript. ARL, JJ, MDP, KO, BNR, SAM, MAA, and PAA all drafted sections of the manuscript and/or participated in reviewing and approving the final version.

Acknowledgements

We thank Ryan Allred for checking manuscript references and manuscript formatting.

Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, et al. Designing and evaluating complex interventions to improve health care. Br Med J. 2007;334:455–9.
Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: The new Medical Research Council guidance. Int J Nurs Stud. 2013;50:585–7.
Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of Innovations in Service Organizations: Systematic Review and Recommendations. Milbank Q. 2004;82:581–629.
Perez Jolles M, Lengnick-Hall R, Mittman BS. Core Functions and Forms of Complex Health Interventions: a Patient-Centered Medical Home Illustration. J Gen Intern Med. Springer New York LLC; 2019;34:1032–8.
International Organization for Standardization. Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability. Geneva, Switzerland; 1998.
Dopp AR, Parisi KE, Munson SA, Lyon AR. Integrating implementation and user-centred design strategies to enhance the impact of health services: Protocol from a concept mapping study. Heal Res Policy Syst. BioMed Central Ltd.; 2019;17.
Blount A, Schoenbaum M, Kathol R, Rollman BL, Thomas M, O’Donohue W, et al. The Economics of Behavioral Health Services in Medical Settings: A Summary of the Evidence. Prof Psychol Res Pract. 2007;38:290–7.
Institute of Medicine. Psychosocial Interventions for Mental and Substance Use Disorders: A Framework for Establishing Evidence-Based Standards. Washington, DC: National Academies Press; 2015.
Areán PA, Ayalon L, Jin C, McCulloch CE, Linkins K, Chen H, et al. Erratum (Retracted article): Integrated specialty mental health care among older minorities improves access but not outcomes: Results of the PRISMe study (International Journal of Geriatric Psychiatry (2008)). Int J Geriatr Psychiatry. 2008;23.
Alexopoulos GS, Raue PJ, Gunning F, Kiosses DN, Kanellopoulos D, Pollari C, et al. Engage Therapy: Behavioral Activation and Improvement of Late-Life Major Depression. Am J Geriatr Psychiatry. 2016;24:320–6.
Barrett JE, Williams JW, Oxman TE, Katon W, Frank E, Hegel MT, et al. The treatment effectiveness project. A comparison of the effectiveness of paroxetine, problem-solving therapy, and placebo in the treatment of minor depression and dysthymia in primary care patients: Background and research plan. Gen Hosp Psychiatry. 1999;21:260–73.
Lyon AR, Munson SA, Renn BN, Atkins DC, Pullmann MD, Friedman E, et al. Use of Human-Centered Design to Improve Implementation of Evidence-Based Psychotherapies in Low-Resource Communities: Protocol for Studies Applying a Framework to Assess Usability. JMIR Res Protoc. JMIR Publications Inc.; 2019;8:e14990.
Lewis CC, Stanick CF, Martinez RG, Weiner BJ, Kim M, Barwick M, et al. The society for implementation research collaboration instrument review project: A methodology to promote rigorous evaluation. Implement Sci. BioMed Central Ltd.; 2015;10.
Lyon AR, Koerner K, Chung J. Usability Evaluation for Evidence-Based Psychosocial Interventions (USE-EBPI): A methodology for assessing complex intervention implementability. Under review.
Norman DA, Draper SW. User Centered System Design; New Perspectives on Human-Computer Interaction. Hillsdale: L. Erlbaum Associates Inc.; 1986.
Rubin J, Chisnell D. Handbook of usability testing: how to plan, design, and conduct effective tests. Wiley Pub; 2008.
Sauro J, Lewis JR. Correlations among Prototypical Usability Metrics: Evidence for the Construct of Usability. Proc 27th Int Conf Hum Factors Comput Syst April 4–9, 2009, Boston, MA, USA. ACM Press; 2009. p. 1609–1618.
Kortum P, Peres SC. The Relationship Between System Effectiveness and Subjective Usability Scores Using the System Usability Scale. Int J Hum Comput Interact Taylor Francis Inc. 2014;30:575–84.
Brooke J. SUS - A quick and dirty usability scale. Reading, UK; 1986.
Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact. 2008;24:574–94.
Kortum P, Sorber M. Measuring the Usability of Mobile Applications for Phones and Tablets. Int J Hum Comput Interact Taylor Francis Inc. 2015;31:518–29.
Lewis JR, Brown J, Mayes DK. Psychometric Evaluation of the EMO and the SUS in the Context of a Large-Sample Unmoderated Usability Study. Int J Hum Comput Interact Taylor Francis Inc. 2015;31:545–53.
Lewis JR, Utesch BS, Maher DE. UMUX-LITE - When there’s no time for the SUS. Conf Hum Factors Comput Syst - Proc. 2013. p. 2099–102.
Lewis JR, Utesch BS, Maher DE. Measuring Perceived Usability: The SUS, UMUX-LITE, and AltUsability. Int J Hum Comput Interact Taylor Francis Inc. 2015;31:496–505.
Sauro J, Lewis JR. When designing usability questionnaires, does it hurt to be positive? Conf Hum Factors Comput Syst - Proc. 2011. p. 2215–23.
Lewis JR, Sauro J. Revisiting the Factor Structure of the System Usability Scale [Internet]. J. Usability Stud. 2017. Available from: https://uxpajournal.org/revisit-factor-structure-system-usability-scale/.
World Health Organization Regional Office for Europe. Integrated care models: an overview Working document [Internet]. Copenhagen, Denmark; 2016. Available from: http://www.euro.who.int/pubrequest.
Centers for Medicare & Medicaid Services. Behavioral Health Integration Services Booklet [Internet]. 2018. Available from: https://www.cms.gov/Outreach-and-Education/Medicare-Learning-Network-MLN/MLNProducts/Downloads/BehavioralHealthIntegration.pdf.
World Health Organization. World Organization of Family Doctors. Integrating mental health into primary care: a global perspective. World Health Organization; 2008.
Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)-A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42:377–81.
Lyon AR. Intervention Usability Scale (IUS). University of Washington: Unpublished instrument; 2016.
Landauer T. Behavioral Research Methods in Human-Computer Interaction. Handb Human-Computer Interact [Internet]. North-Holland; 1997 [cited 2020 Mar 18]. p. 203–27. Available from: https://www.sciencedirect.com/science/article/pii/B9780444818621500753?via%3Dihub.
Nunnally JC. Psychometric theory. New York: McGraw-Hill; 1978.
Mclellan S, Muddimer A, Peres SC. The Effect of Experience on System Usability Scale Ratings. J Usability Stud [Internet]. 2012;7:56–67. Available from: http://uxpajournal.org/the-effect-of-experience-on-system-usability-scale-ratings/.
Sauro J. Does prior experience affect perceptions of usability? [Internet]. 2011. Available from: https://measuringu.com/prior-exposure/.
Lyon AR, Bruns EJ. User-Centered Redesign of Evidence-Based Psychosocial Interventions to Enhance Implementation - Hospitable Soil or Better Seeds? JAMA Psychiatry. American Medical Association; 2019. p. 3–4.

Download PDF

Journal Publication

published 08 Feb, 2021

Read the published version in Implementation Research and Practice →

Version 1

posted

You are reading this older preprint version

Read the latest preprint version →

Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

Status:

Journal Publication

Version 1

Abstract

Contributions To The Literature

Background

Human-centered Design (hcd)

Current Aims

Method

Participants

Procedures

Measures

Analyses

Results

Item Descriptives and Correlations

Ius Factor Structure

Scale Correlations

Reliability/internal Consistency

Sensitivity

Distribution Of Ius Subscale Scores

Discussion

Limitations

Conclusions And Next Steps

Abbreviations

Declarations

References

Status:

Journal Publication

Version 1