All analyses were performed using R statistical software (version 4.2.3). We first compared demographic characteristics between recruitment platform samples. Then, we examined differences in mind wandering, attention task performance, and perceived risk as a function of risk condition, recruitment platform, and their interaction. Multitasking was only assessed as function of recruitment platform, however (see below). Next, we compared individual traits between recruitment platforms. Finally, we examined correlations between individual traits, mind wandering, task performance, and perceived risk within each recruitment platform sample.
Demographic information
Table 1 shows the demographic characteristics of the two platform samples. Demographic variables were compared between recruitment platform samples using t-tests for continuous variables and chi-square tests of independence for discrete variables. Results revealed that the samples did not significantly differ in age, sex, or gender, but did significantly differ in education and employment. Compared to the Prolific sample, the MTurk sample was more highly educated and more likely to be working full time versus studying, caregiving, or unemployed.
Table 1
Demographic characteristics of samples from each recruitment platform. Demographics did not significantly differ between the low-risk and high-risk conditions in either the Prolific sample or the MTurk sample. The categories, “some college,” “some undergraduate university,” and “some graduate university” also include completed programs. * p < .05
Variable
|
Prolific
n = 82
|
MTurk
n = 78
|
X2 (t)
|
Age, M (SD)
|
28.9 (4.41)
|
28.3 (4.02)
|
(0.99)
|
Sex, n (%)
|
|
|
2.82
|
Male
|
44 (53.7)
|
52 (66.7)
|
|
Female
|
38 (46.3)
|
26 (33.3)
|
|
Gender, n (%)
|
|
|
3.48
|
Man
|
44 (53.7)
|
50 (64.1)
|
|
Woman
|
38 (46.3)
|
26 (33.3)
|
|
Non-Binary
|
0 (00.0)
|
1 (2.40)
|
|
Education, n (%)
|
|
|
22.9*
|
High School or Less
|
12 (14.6)
|
6 (7.69)
|
|
Some College
|
15 (18.3)
|
9 (11.5)
|
|
Some Undergraduate University
|
34 (41.5)
|
18 (23.1)
|
|
Some Graduate University
|
18 (22.0)
|
45 (57.8)
|
|
Other
|
3 (3.6)
|
0 (0.00)
|
|
Employment, n (%)
|
|
|
16.1*
|
Full-Time Work
|
53 (64.6)
|
69 (88.5)
|
|
Work and Studies
|
14 (17.1)
|
8 (10.3)
|
|
Studies
|
8 (9.76)
|
1 (2.40)
|
|
Caregiver
|
4 (4.88)
|
0 (0.00)
|
|
Unemployed
|
3 (3.70)
|
0 (0.00)
|
|
Differences in attention task accuracy, mind wandering, and risk perception by risk condition and recruitment platform
Figure 2 displays means and confidence intervals for accuracy, mind wandering, and risk perception in the attention task by risk condition and recruitment platform. General linear mixed models (GLMMs), generated with the lme4 (version 1.1–32) and afex (version 1.2-1) packages, were used for analyses of accuracy and mind wandering. Since these variables are binary (e.g., correct vs. incorrect, focused vs. mind wandering) and rely on repeated measurement (i.e., multiple trials and thought probes), GLMMs used binomial distributions and included random intercepts by participant. Weights in models predicting mind wandering controlled for variability in number of thought probes delivered to each participant.
An ANOVA with chi-square tests assessed the contributions of between-subject factors, risk condition and platform, to accuracy in the attention task. Results revealed no significant main effect of risk condition, X2(1) = 2.48, p = .115. There was, however, a significant main effect of platform, X2(1) = 80.1, p < .001. A post-hoc z-test revealed significantly lower accuracy in the MTurk sample compared to the Prolific sample, OR = 0.05, 95% CI [0.03, 0.08]. There was also a significant interaction between risk condition and platform, X2(1) = 4.50, p < .05. Post-hoc z-tests revealed significantly higher accuracy in the high-risk group compared to the low-risk group in the MTurk sample, OR = 3.10, 95% CI [1.30, 7.38], but not in the Prolific sample, OR = 0.84, 95% CI [0.37, 1.91]. The interaction remained significant after removing participants with accuracy scores equal to zero from both samples.
An ANOVA with chi-square tests assessed the contributions of between-subjects factors, risk condition and platform, to mind wandering in the attention task. Results revealed no significant main effect of risk condition, X2(1) = 0.20, p = .655. There was, however, a significant main effect of platform, X2(1) = 14.0, p < .001. A post-hoc z-test revealed significantly more mind wandering in the MTurk sample compared to the Prolific sample, OR = 4.36, 95% CI [2.03, 9.37]. There was no significant interaction between risk condition and platform, X2(1) = 1.99, p = .158. Results remained the same after removing participants with accuracy scores equal to zero.
An ANOVA with F-tests examined the contributions of between-subjects factors, risk condition and platform, to risk perception in the attention task. Results revealed no significant effects of risk condition, F(1, 156) = 2.24, p = .136, platform, F(1, 156) = 0.29, p = .591, or their interaction, F(1, 156) = 0.43, p = .512.
Differences in multitasking between recruitment platforms
Figure 3 displays percentages of participants from each platform that reported engaging in media multitasking, non-media multitasking, and no multitasking. It also displays percentages of Likert responses to questions concerning how much participants from each platform perceived their multitasking to impact their attention and performance in the attention task. A chi-square test of independence and post-hoc z-tests explored between-platform differences in self-reports of multitasking. Since few participants in the Prolific sample reported multitasking, we did not test for differences in multitasking between risk conditions, nor did we test for between-platform differences in the self-reported impact of multitasking on attention and performance.
Results of the chi-square test revealed a significant difference in multitasking between platform samples, X2 = 35.0, p < .001. A post-hoc z-test revealed that, compared to the Prolific sample, a significantly greater proportion of MTurk participants reported engaging in media-related multitasking, OR = 19.8, 95% CI [5.73, 68.1]. There was no significant difference between platforms in reports of media-unrelated multitasking, however, OR = 0.35, 95% CI [0.04, 3.40].
Differences in Individual Traits Between Recruitment Platforms
Table 2 shows scores reflecting individual traits pertaining to distraction-related cognitive failures, inattention, and sensation seeking by recruitment platform. Between-platform comparisons were performed using t-tests. The MTurk sample reported significantly more distraction-related cognitive failures in life and work, as indicated by scores on the ARCES and WCFS, respectively. Furthermore, the MTurk sample reported significantly greater inattention and risk taking, as indicated by scores on the MAAS-LO and BSSS, respectively.
Table 2
Individual trait cognitive failures, inattention, and sensation seeking by recruitment platform. ARCES = Attention-Related Cognitive Errors Scale, MAAS-LO = Mindful Awareness of Attention Scale, BSSS = Brief Sensation Seeking Scale, WCFS = Workplace Cognitive Failure Scale.* p < .05
Variable
|
Prolific
n = 82
|
MTurk
n = 78
|
t
|
M
|
SD
|
M
|
SD
|
ARCES
|
2.54
|
0.63
|
3.34
|
1.00
|
-6.08*
|
MAAS
|
2.77
|
0.83
|
3.80
|
1.39
|
-5.63*
|
BSSS
|
2.69
|
0.81
|
3.67
|
0.88
|
-7.29*
|
WCFS
|
1.74
|
0.49
|
3.13
|
1.11
|
-10.2*
|
Correlations between individual traits, task performance, and mind wandering
Table 3 shows correlations between individual trait measures and task-related dependent variables for each platform. We collapsed across risk conditions because we did not find strong disengagement differences in prior analyses and sample sizes per correlation would have been halved. The lower triangle includes correlations for the Prolific sample and the upper triangle includes correlations for the MTurk sample. Spearman’s rank order correlations were used since the variables had different scales. Common to both samples were positive correlations between individual trait measures of inattention (MAAS) and cognitive failures (ARCES, WCFS) as well as mind wandering in the attention task. Furthermore, in both samples, probe-caught mind wandering during the attention task (MW) negatively correlated with accuracy (Acc) in the attention task and positively correlated with perception of risk. Interestingly, the MTurk sample exhibited several significant correlations that were not present for the Prolific sample. Only in the MTurk sample did sensation seeking (BSS) positively correlate with individual trait measures of inattention (MAAS) and cognitive failures (ARCES, WCFS), as well as mind wandering in the attention task. Also unique to the MTurk sample were negative correlations between individual trait measures (ARCES, WCFS, MAAS, and BSSS) and accuracy in the attention task. Overall, results revealed less conservative relations among study variables in the MTurk sample compared to the Prolific sample.
Table 3
Correlations between individual traits, task performance, mind wandering, and risk perception for Prolific (lower triangle) and MTurk (upper triangle). ARCES = Attention-Related Cognitive Errors Scale, MAAS = Mindful Awareness of Attention Scale, WCFS = Workplace Cognitive Failures Scale, BSSS = Brief Sensation Seeking Scale, Acc = accuracy, MW = mind wandering. Acc refers to the percentage of targets to which participants responded correctly. MW refers to the percentage of thought probes to which participants indicated off-task thoughts. Risk refers to participant perceptions of the chance of losing points from task errors. Results are collapses across low-risk and high-risk conditions. The lower triangle includes correlations for the Prolific sample. The upper triangle includes correlations for the MTurk sample. Coefficients were generated with Spearman’s rank order method.* p < .05, ** p < .01, *** p < .001
|
-Age
|
-ARCES
|
-MAAS
|
-WCFS
|
-BSSS
|
-Acc
|
-MW
|
-Risk
|
Age
|
|
-0.08
|
-0.07
|
-0.06
|
-0.19
|
-0.09
|
-0.12
|
-0.03
|
ARCES
|
-0.10
|
|
-0.76***
|
-0.88***
|
-0.78***
|
-0.53***
|
-0.56***
|
-0.19
|
MAAS
|
-0.13
|
-0.70***
|
|
-0.82***
|
-0.62***
|
-0.61***
|
-0.50***
|
-0.09
|
WCFS
|
-0.02
|
-0.69***
|
-0.46***
|
|
-0.75***
|
-0.63***
|
-0.58***
|
-0.12
|
BSSS
|
-0.09
|
-0.06
|
-0.10
|
-0.05
|
|
-0.39***
|
-0.34**
|
-0.22*
|
Acc
|
-0.08
|
-0.06
|
-0.10
|
-0.17
|
-0.02
|
|
-0.54***
|
-0.09
|
MW
|
-0.17
|
-0.30**
|
-0.33**
|
-0.29**
|
-0.04
|
-0.30**
|
|
-0.23*
|
Risk
|
-0.16
|
-0.07
|
-0.10
|
-0.11
|
-0.09
|
-0.09
|
-0.21
|
|