Metacognition during fake news detection induces an ineffective demand for disambiguating information

doi:10.21203/rs.3.rs-3921235/v1

Download PDF

Article

Metacognition during fake news detection induces an ineffective demand for disambiguating information

https://doi.org/10.21203/rs.3.rs-3921235/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

The mechanisms by which individuals evaluate the veracity of uncertain news and subsequently decide whether to seek additional information to resolve uncertainty remain unclear. In a controlled experiment participants assessed non-partisan ambiguous news and made decisions about whether to acquire extra information. Interestingly, confidence in their judgments of news veracity did not reliably predict actual accuracy, indicating limited metacognitive ability in navigating ambiguous news. Nonetheless, the level of confidence, although uncalibrated, was the primary driver of the demand for additional information about the news, with lower confidence driving a greater demand, regardless of its veracity judgment. This demand for disambiguating information, driven by the uncalibrated metacognition, was increasingly ineffective as individuals became more enticed by the ambiguity of the news. Our findings highlight how metacognitive abilities shape decisions to seek or avoid additional information amidst ambiguity, suggesting that interventions targeting ambiguity and enhancing confidence calibration could effectively combat misinformation.

Main Text

Humanities/Cultural and media studies

Social science/Psychology/Human behaviour

Social science/Complex networks

The unprecedented growth of the internet and social media platforms has been accompanied both by an abundance of content and by the spread of misinformation^1–4. Misinformation, characterized by false, inaccurate or misleading information, yields devastating consequences at the societal level, fueling polarization and fostering resistance to crucial initiatives such as climate action and vaccination efforts^5–7. Contrary to disinformation, misinformation does not need to be created deliberately to mislead. The inherent imprecision of misinformation frequently blurs the perceived boundaries between true or false information. Moreover, its capacity to create an illusion of consensus can inadvertently undermine individuals' ability to discern information authenticity. The appearance of credibility stemming from unified perspectives often obstructs critical evaluation, leaving individuals susceptible to unwittingly compromising their assessment of information authenticity⁸. These characteristics increase ambiguity^9,10, amplifying the difficulty to discern between true and false information. Individuals not only face challenges when having to evaluate the veracity of the information they are exposed to, they also struggle to effectively search for extra information to verify social and political claims, sources, and evidence¹¹. A number of strategies have been proposed to prevent the spread of misinformation^12–14, including fact-checking, directing attention to accuracy^15–17, censorship, encouraging more selective sharing by individuals^18,19 or capping the number of others to whom messages can be forwarded²⁰. Yet, fact-checking at the speed and scale of today’s platforms is often impractical for private companies or government agencies²⁰. An alternative approach could involve targetting individuals themselves and focusing on enhancing their abilities to assess the veracity estimation²¹.

Here, we were interested in understanding the cognitive mechanisms and the relationships between individuals’ judgment about the veracity of ambiguous news they are exposed to, the confidence in such judgment, and the willingness to seek additional information to better assess news veracity.

The willingness to gather extra information rather than sticking to one’s current knowledge may depend critically on the subjective confidence in one’s assessment. This has been demonstrated in the domain of perceptual decision making^22,23 but remains to be studied regarding real news. In perceptual decision tasks, confidence in one’s judgment accuracy plays a direct and determinant role in one’s willingness to sample more evidence to update one’s beliefs^23–26. In those tasks, subjective confidence in one’s judgment accuracy correlates closely with objective accuracy^27–31. However, in real-world situations, such as assessing the veracity of media news, the role of confidence in information search remains unclear, as is the relationship between subjective confidence and objective accuracy in the assessment of news veracity. Because decision accuracy and confidence are typically highly correlated^23,32,33, it is difficult to identify whether confidence causally influences the demand for extra information. Crucially, to provide such evidence concerning real news, one needs to demonstrate that confidence in one’s judgment predicts the demand for extra information, while controlling for objective performance accuracy in judging news as true or false. To that purpose, we designed our experiment using ambiguous news contents, deliberately leading to performance accuracy at chance level when assessing the veracity of these news. Understanding these relationships requires experimentalists to meticulously select news and rigourously control for their level of ambiguity.

Our focus was on news reflecting uncertain information, meaning they could be either false or true, with varying levels of perceived ambiguity. Importantly, these news were sourced from agents with no intention to deceive, thereby ruling out a deliberate willingness to propagate fake news. Examples of naïve agents endorsing and spreading false inaccurate information with no intention to deceive abound³⁴. Recent research report that only a very small percentage of people purposely endorse sharing misinformation online³⁵. We designed an incentivized within-subject experiment in which non-ego relevant news varied in content imprecision and propensity to polarize opinions. Specifically, these news only concerned non-partisan news, that is, news unrelated to political parties. Participants were presented with a set of brief news about ecology, democracy and social justice taken from the press that could be either true or false. Participants had to evaluate the veracity of each brief news and report their confidence in their judgment on a continuous scale, using a probability elicitation incentivized method. Then, participants had to decide on whether acquiring or not additional information about this news (to be received after the task was performed), and report their willingness-to-pay to have their information-seeking choice implemented (Fig. 1). Importantly, this latter procedure ensured that the willingness to acquire or not acquire extra information is equally balanced. This is unlike previous procedures used in the perceptual decision making domain for which acquiring extra information was costly but for which not acquiring extra information was free (e.g., ²⁴).

We investigated the relationship between objective accuracy in judging news veracity and confidence in this judgment, controlling for how news ambiguity influences the judgments of news as true or false. Specifically, we first tested whether confidence in one’s judgment about news veracity predicts the success in judging news veracity. Under varying ambiguity, we anticipated uncalibrated metacognition, with participants’ confidence uncorrelated with actual success. We then tested whether such confidence in one’s judgments drives the demand for extra information about these news. In line with the perceptual decision making literature on confidence-based information-seeking^23,24,36, we predicted that the demand for extra information should increase when confidence in one’s veracity assessment is at the lowest. In our context, confidence rather than beliefs is anticipated to play a pivotal role. Finally, we performed a moderated mediation analysis to reveal the relationships between the mechanisms underlying judgments of news veracity and the mechanisms underlying the demand for extra information about the news.

------------------------------- Insert Fig. 1 about here ---------------------------------------

Judgment of news veracity and success rate

We first confirmed that performance accuracy was at chance level when judging the veracity of news. Participants' average success (i.e., correctly judging a true news as true and correctly judging a false news as false) rate was of 51.6 ± 6.7% (Supplementary I.1, Fig. S1). A comparison of the performances with a random distribution within a Bayesian framework confirmed that performances were at chance level. Modelling random responses with a logistic function λ~logistic (λ₀,rscale) with priors ${\lambda }_{0}$= 0.5 and rscale = 1.5, the Bayes Factor (BF) favored the null hypothesis of chance-level by a factor of about 0.3. This factor is considered the low boundary of a moderate evidence³⁷, however posteriors probabilities fell within the range of [49–53]% success, centered around 51.5%. (Supplementary I.1, Fig. S3 & Table S2).

Next, we examined in what conditions participants’ successes deviated from chance level. Although judgment of ambiguous news veracity was equivalent to chance, participants performed better with true news (64 ± 11.9%) than false news (39.1 ± 12%). Lowest accuracy was for democracy-related news (48.6 ± 11.5), with a slightly higher accuracy for news related to ecology (52.2 ± 11.3) and social justice (53.8 ± 11.8) (See Supplementary I.1, Table S3). Binomial Mixed Linear Models (MLMs) showed that participants predicted true news significantly more accurately than false news (odds-ratio = 2.77, p < 0.001), with the veracity of news interacting with its theme (p < 0.001). Democracy-related news had significantly lowest accuracy compared to ecology and social justice (odds-ratio respectively at 1.19 and 1.26, all p < 0.005) (see Fig. 2A). Effects remained highly significant (p < 0.005) after controlling for socio-demographics, veracity judgment, and confidence (Supplementary I.1, Table S4).

Such relatively higher ability to assess true news accurately can be explained by a general tendency to declare information as true (59.5 ± 10.6%), with slightly more true news declared as true (60.9 ± 12.7%) than false news (58.2 ± 13.1%) (See Supplementary I.1, Table S5). Modelling the success in estimating veracity confirmed that the veracity judgment was a highly significant explaining variable (p = 0.003), withstanding the inclusion of control variables (Supplementary I.1, Table S6). Interestingly, binomial MLMs of veracity judgment revealed that participants were especially more likely to judge as true ecology-related (prob. = 0.678 ± .02) and social justice-related news (prob. = 0.637 ± .02) than democracy-related news (odds-ratios 1.71 and 1.43, respectively; all p < 0.001) (see Fig. 2B).

------------------------------- Insert Fig. 2 about here ---------------------------------------

Our analysis of the relationship between accuracy in judging news veracity and confidence in this judgment showed that participants’ confidence did not significantly predict actual success nor veracity judgments (all p > .05).

Results instead demonstrated that individuals’ responses were primarily influenced by news ambiguity, specifically both the imprecision and the polarization of news content, potentially leading to a perception of falsity (Supplementary I.1, Fig. S4). We modelled success in veracity judgment with MLMs incorporating imprecision and polarization predictors in interaction with news veracity (Supplementary I.1, Table S7). Note that the news content imprecision and propensity to polarize (from 0 to 10) were obtained from ratings of a group of subjects (n = 55) independent from the actual participants in the experiment (see Methods). The interaction effect of each predictor had a highly significant effect on the success of veracity judgment likelihood (all p < 0.001, all odds-ratios < 0.76). Specifically, success in judging true news increased when their content imprecision and propensity to polarize were at their minimum (minimum/maximum, imprecision odds-ratio = 1.82; polarization odds-ratio = 2.17) Conversely, for false news, success increased with maximal imprecision (minimum/maximum odds-ratio = 0.53) and maximal propensity to polarize (minimum/maximum odds-ratio = 0.22) (all p < 0.001). Furthermore, MLMs of veracity judgments showed that the likelihood of judging news as true decreased with increased imprecision (p < 0.001, odds-ratio = 0.78) and the propensity to polarize (p < 0.001, odds-ratio = 0.64). The effects in all models withstood the inclusion of socio-demographics, veracity-theme interaction, and confidence (Supplementary I.1, Table S7).

Finally, we found that alignment of beliefs with news concerns, distrust in experts and socio-demographics had no significant effect on the accuracy of veracity judgments. Using MLMs (see Method; Supplementary I.1, Fig. S5, Table S8 & S9), response times showed a positive effect on judgment accuracy (p = .007, odds-ratio = 1.07), albeit not robust to the inclusion of other factors. We used Bayesian inference hypothesis testing to support these findings. Comparing Bayesian versions of the regression models (see Method, Supplementary I.2, Fig. S6-S9), the winning model featured interaction terms between news veracity and both news content imprecision and propensity to polarize (see Table 1). Overall, individuals’ accuracy deviated from chance level in reaction to variations in news ambiguity. Precision and apparent consensus about news content were interpreted as a signal of veracity, while imprecision and apparent polarization were seen as signals of falsity. Note that we found no significant difference between true and false news either in terms of imprecision (mean ± SD = 5.53 ± 1.24 vs. 5.17 ± 1.25, 𝑟𝑎𝑛𝑘𝑠𝑢𝑚, 𝑝=0.09), or in terms of polarization (mean ± SD = 6.61 ± 1.62 vs. 6.22 ± 1.62, 𝑟𝑎𝑛𝑘𝑠𝑢𝑚, 𝑝=0.3).

Uncalibrated metacognitive sense of confidence

To further investigate the relationship between confidence and accuracy in estimating veracity, we examined participants’ calibration, that is their ability to accurately estimate the chances that the news is true or false (see Supplementary I.3, Table S10). The confidence-accuracy calibration reflects, for given veracity judgments (the news is evaluated as true or false), the relationship between the continuous scale of confidence ([1,100]) and the binary outcome (true or false). This calibration indexes the extent to which confidence in one’s judgment predicts the accuracy of this judgment. A perfect calibration is characterized by a linear confidence-accuracy function with 100% accuracy for 100% confidence, 90% accuracy for 90% confidence, etc. We sorted the individual confidence-accuracy relationships into ten bins and represented an area of well-calibrated estimation that spanned 10% (see Fig. 3).

We expected that participants’ confidence would be non-calibrated and uncorrelated with actual success in estimating the veracity of uncertain news. As the plot shows, participants’ accuracy in estimating veracity was independent from their confidence in their estimation. Participants were neither well-calibrated, nor ill-calibrated for estimating probabilities. Values above the diagonal signal underconfidence (individuals have a higher proportion of correct guesses than their reported level of confidence) while values below the diagonal reveal overconfidence (individuals have a lower proportion of correct guesses than their reported level of confidence). Figure 3 shows that underconfidence dominates for degrees of confidence below 50% whereas overconfidence dominates for degrees of confidence above 50%. Underconfidence dominates for true news whereas overconfidence dominates for false news, while news veracity judgment did not affect the relationships between confidence and success levels (Supplementary I.3, Fig. S10).

------------------------------- Insert Fig. 3 about here ---------------------------------------

To understand the determinants of confidence during estimation of news veracity, we examined the sources of variability using MLMs of alignment of beliefs with concerns related to the news, socio-demographics and response times. Moreover, we examined models predicting effects of imprecision and polarization predictors on confidence in veracity judgments.

Alignment of beliefs with news concerns and socio-demographics showed no significant effects on confidence (Supplementary I.3, Table S11), while effects of response times were highly significant and negative (p < 0.001). Importantly, the interaction of both ambiguity predictors with the judgments of news as true decreased confidence (p < 0.001), even after including control variables (Supplementary I.3, Table S12). Sex also revealed higher confidence levels for males than females (p < 0.001).

Confidence ratings were reliably affected by news content ambiguity, reflecting higher confidence that a news is true under low ambiguity and higher confidence that a news is false under high ambiguity. Comparing confidence levels between judgments of true and false news across three different levels of content imprecision and propensity to polarize revealed a significant effect of these variables on confidence. Even after the inclusion of control variables, confidence was higher for judgments of the news as true when imprecision was lowest (t ratio = 3.85, p < .001) and median (t ratio = 2.60, p = .0092). In contrast, confidence was not significantly different for judgments of the news as false than for judgments of the news as true when imprecision was at its highest level (t ratio = -1.84, p = .065). Conversely, confidence was higher for judgments of the news as true when the news content propensity to polarize was at its lowest level (z ratio = 8.61, p < .001), but higher for judgment of the news as false when polarization was highest (z ratio = -8.34, p < .001). Finally, confidence was not significantly different between the two types of judgments for a median polarization level (p = .57). These findings support the use of imprecision and polarization as signals of falsity, influencing veracity estimation.

Demand and avoidance of extra information

Next, we analyzed the demand for or avoidance of extra information about news that might resolve uncertainty. We predicted that despite the lack of calibration (i.e., a low degree of fit between confidence in news veracity judgment and the actual accuracy), individuals would use their metacognitive sense of confidence to decide whether or not to demand extra information about the news. Hence, we expected confidence to primarily explain the demand for extra information, particularly when confidence was low. First, we present participants' reception choices and subsequent Willingness-To-Pay (WTP). Then, we explore linear relationships between confidence and reception choices/WTP. To test our hypothesis, we estimated separate MLMs with variables capturing main and interaction effects of participant confidence and news veracity judgment. The dependent variables were the binomial demand for more information or the continuous WTP. Post-hoc comparisons were conducted on estimated marginal means.

82.9% of participants demanded extra information at least once, with an average frequency of 42.29 ± 31.9%. Choice of extra information did not significantly differ between news themes (Kruskal Wallis Chi square = 4.39, p = .11, df = 2; democracy: 41.04 ± 33.16; ecology: 43.27 ± 33.67; social justice: 42.56 ± 33.09; see Supplementary I.4, Table S13). Participants chose to receive extra information 42.51 ± 32.44% of the time when news were judged as false and 42.07 ± 32.14% of the time when news were judged as true. Bayesian modeling of reception choices between judgments (Jeffreys priors: α = 0.5, β = 0.5) revealed a negligible difference (delta = 0.23, 95% Credible Interval [-0.008, 0.012]), indicating similar demand for extra information regardless of veracity judgments.

Participants exhibited a higher willingness-to-pay (WTP) for receiving extra information (mean: 7.07 ± 4.96 ECU) compared to not receiving it (mean: 5.75 ± 5.69 ECU) (see Fig. 4; see Supplementary I.4, Table S14). Bayesian models of WTP for receiving and not receiving extra information (Jeffreys priors: µ = 0, σ = 1 from half-Cauchy distribution) showed that participants were willing to pay more to receive it than to avoid it (delta = 1.327, 95% Credible Interval [-2.302, -0.344]).

As predicted, confidence explains the demand for extra information (p < 0.001, odds-ratio = 0.59), with a significant negative interaction with veracity judgment (p < 0.001). These effects remained significant even after incorporating controls such as the interaction of news veracity and theme and socio-demographics (Supplementary I.4, Table S15). The results show that the probability of demanding extra information is not affected by news content ambiguity (i.e., imprecision and propensity to polarize) (see Fig. 4A) while it decreases as confidence in one’s judgment increases. Specifically, the decrease is more pronounced when the news is judged as false (minimum/maximum confidence; judgment as false, odds-ratio = 6.41; judgment as true, odds-ratio = 2.59) (see Fig. 4B). A regression analysis of WTP further supported these findings, revealing a significant interaction between confidence and the demand for information (p < 0.001) (see Fig. 4C), holding up against the inclusion of control variables (Supplementary I.4, Table S16). According to this model, the effect size of confidence (minimum – maximum confidence levels) on the WTP when participants opted not to receive extra information was − 1.74 (p < 0.001), whereas the effect size for the WTP to receive extra information was only − 0.13 (p = 0.69) (see Fig. 4D).

The alignment of beliefs with news concerns from only two organizations predicted reception choices while we found no evidence for effects of sociodemographics, response times, distrust or ambiguity on decisions to seek information that might resolve uncertainty about our ambiguous news.

To sum up, there is a significant inverse relationship between the demand for extra information about the news and confidence in one’s judgment about news veracity. Moreover, this relationship is stronger for the news that participants judged as false. Supporting these findings, participants are also willing to pay more to not receive more information about what they think they already know.

------------------------------- Insert Fig. 4 about here ---------------------------------------

A moderated mediation analysis further extended the role of confidence in the estimation of ambiguous news veracity (Table 2, Fig. 5 ; Supplementary I.5, Fig. S11). Confidence had a unique direct effect on the outcome reception choice (standardized interaction β = -0.15, Z = -13.96, p < .001). Its effect was specifically a mediator effect, whereby the ambiguity of news, that is, news content imprecision and news content propensity to polarize, had an indirect effect on the reception choices through the confidence (imprecision: standardized interaction β = -0.06, Z = -3.93, p < .001; polarization: standardized interaction β = 0.11, Z = 6.32, p < .001). Veracity judgment played a role by moderating the effect on the news content imprecision to confidence path (standardized interaction β = 0.1, Z = 2.43, p = .015) as well as the effect on the news content propensity to polarize to confidence path (standardized interaction β = -0.36, Z = -8.34, p < .001). This analysis shows that the uncalibrated metacognition operating during the evaluation of true and false news induces a demand for disambiguating information that is increasingly ineffective as individuals are lured by the ambiguity of the news.

------------------------------- Insert Fig. 5 about here ---------------------------------------

Using a novel experimental design, we carefully selected non-partisan and non-ego relevant news that offer various levels of content imprecision and polarization. Participants’ accuracy in assessing news veracity hovering at chance level confirmed that we manipulated news with uncertain contents, thereby allowing us to disentangle the effects of confidence from the effects of objective performance accuracy. We focused on news about ecology, democracy and social justice whose utility was mainly cognitive³⁸. That is, we chose news that could help individuals to form more accurate beliefs about the state of the world, and that would neither threaten their identity nor affect their perception of how others would see them. A sentiment analysis confirmed the neutrality of the stimuli emotional valence (Supplementary VI.1, Fig. S12). The reason was to restrict as much as possible distortions in the demand for extra information that would result from motivated reasoning to protect one’s image or identity.

How do individuals judge the veracity of uncertain news? Participants’ confidence did not predict their actual accuracy, however they systematically overestimated the prevalence of true news in the task. This inclination could stem from the automatic acceptance of statements and the cognitive strain associated with reevaluating previously acknowledged information³⁹. It may also be that individuals are inclined to regard information as correct if it is deemed "good enough", avoiding a costly in-depth analysis^40,41. An alternative perspective suggests that evolution has shaped human communication towards truthfulness, with altruism and gullibility as norms to ensure cooperation⁴². For instance, children tend to initially trust social partners⁴³. Moreover, some defend that there is a prevailing inclination toward intuitive honesty among humans⁴⁴, leading individuals to anticipate a higher frequency of true statements in the information they encounter.

While truthful communication is essential, signals must also convey useful information in the presence of uncertainty. Epistemic vigilance⁴⁵ has been proposed as an evolutionary tool, encouraging individuals to critically assess the veracity of statements. Our study reveals that participants consider ambiguity dimensions like content imprecision and polarizing tendencies. Higher imprecision and propensity to polarize increased the likelihood of individuals mistakenly declaring news as false with confidence. This is consistent with previous research showing that individuals disproportionately prefer information that would provide a sense of certainty⁴⁶. The imprecision in information content may signal unreliability, as it provides less clarity in the verifiability of the assertion whereas in the face of conflicting information, content polarization may signal untrustworthiness. Ambiguous content could hinder coordination and impose cognitive strains, leading individuals to preferentially identify such content and avoid it as an epistemic strategy for truth-seeking. The prominence of these dimensions, especially in comparison to alignment with beliefs or distrust toward experts, is consistent with the fact that we manipulated news with a primary emphasis on cognitive utility.

Participants’ metacognitive abilities were uncorrelated with success in estimating news veracity and we observed that their confidence-accuracy calibration was flat (Fig. 4). Confidence usually strongly correlates with objective accuracy in perceptual decision tasks or adaptive behavior^23,27,28. However, the relationship between one’s accuracy of judgment and one’s confidence about judgment is known to vary greatly with task difficulty, whereby confidence is decreasingly predicting accuracy as difficulty increases^47–50. The dissociation that we observed between confidence and actual success rate suggests a pattern specific to uncertain news, in contrast with perceptual information, with individuals struggling to gauge their level of knowledge when confronted with potential misinformation.

Crucially, although individuals held an inaccurate perception of their own knowledge, this metacognitive sense of confidence was the most decisive dimension that guided information-seeking behavior in our experiment. Participants were willing to pay more to not receiving more information about news that they estimated they already knew to be false. These results suggest that the decision to seek additional information likely stems from the expected benefit of this additional information in terms of subsequent cognition and reduction of uncertainty about the state of the world. This key finding presumably reflects that individuals use uncertainty – reflected in their confidence in their judgment – to choose whether to gather more evidence^23–26,36.

The present study provides empirical evidence indicating the challenges individuals face in distinguishing true from false uncertain news, often confusing precise or consensual information with truth. Our novel findings underscore the prime role of metacognitive abilities in mediating the relationship between ambiguous information assessment and the demand or avoidance of extra information. Individuals misjudge what they know but they also seek to receive information according to what they know. As a consequence, they misidentify shortfalls in their knowledge, preventing them from filling the gaps. This demonstrates that individuals are not only at risk of receiving undetected false information but also inefficiently explore their environment, potentially spreading false information upon sharing it³⁴. While previous literature suggests that people share false information due to a lack of attention to accuracy^16,17, our study suggests that their search for information to reduce uncertainty is driven by misplaced confidence in their veracity judgment. This search is increasingly ineffective as individuals are lured by the ambiguity of news. This findings are all the more important as our societies are facing major challenges with the extremely fast technical development of generative AI and the spread of deepfakes that will make the identification of veracity more and more difficult in the immediate future.

Our results give ground to possible interventions and changes in social media features to address the major challenges posed by misinformation and the limited ability of humans to detect the truth. They call for the development of education and media literacy programs fostering self-improvement of veracity estimation ability and self-motivated extra information seeking. This could be done by encouraging individuals to rate their confidence in news content and test it against evidence in oder to increase awareness. The ability to evaluate information and to subsequently search for extra information to assess the veracity of the news can also be trained with specific heuristics^11,51–53. Gamified solutions of probability calibration exercises could be tested in school media literacy programs and in training apps to improve assessment of one’s knowledge and detection ability, and the need for information-seeking^52–54. These interventions complement news content moderation, signaling of trustworthiness, and changes in the incentive structure of media platforms,^12,13,55,56 aiming both to decrease motivations to share content that receives high social reward at the cost of accuracy and to increase accuracy motivation^17,35.

Participants

269 participants with no history of neurological or psychiatric disorders participated in this online experiment run on Testable.org. Data were collected in two waves. A first one took place with 80 participants in November 2020. A second one with 189 participants spanned from December 2021 to January 2022. Except for additional questions in the final questionnaire, there were no differences in the experimental design between the two waves. Participants, mainly students in engineering and business, were recruited from the regular GATE-Lab subject-pool, Lyon, France. They were paid on average $15.92, including a $9 show-up fee, for an experiment that lasted 46 minutes on average. In total, two participants were excluded from the analyses due to outlying response times (“RT”) during news evaluation (one subject: RT = 51.79 ± 26.35; one subject: RT = 1.93 ± 1.31) compared to the mean response time (14.41 ± 8.44). Nine participants were excluded because they did not complete the final questionnaire. In total, 258 participants were included in the statistical analyses (127 males, mean age ± SD = 21.9 ± 2.78).

The study was approved by an internal ethics review board and complied with the European data protection regulation (GDPR). Informed consent was obtained from all subjects prior to participation.

Task and Design

To select our stimuli, we set-up a pre-test of every stimulus with independent raters and kept the stimuli that best fitted our criteria (see Supplementary II). Overall, our procedure closely follows the practical guide of Pennycook and colleagues for behavioral research on fake news and misinformation⁵⁷. In addition, we ran a sentiment analysis on all stimuli, separating for true and false news. Out of the 96 stimuli, 93.75% of the news were predominantly categorized as emotionally neutral (see Supplementary VI.1, Fig. S12).

Individuals’ worldviews have been shown to explain what they believe to be true⁵⁸. To have a proxy of such prior beliefs we instructed participants in the first part of the experiment to rate various political organizations that were related to the different news domains. We selected 12 organizations active in the domains of ecology, democracy or social justice. Each organization was described by a 1000-character (± 20%) statement taken from the organization websites, with minimal manipulation of the original website content. Participants indicated with six responses their liking, familiarity and closeness of values concerning organizations in direct connection with the topics of the news., on a scale from 0 to 7 (Supplementary III.1). For each topic, we selected two organizations aligned with concerns related to the news, and two organizations misaligned with them (See Supplementary IV).

We computed the participants’ adhesion to each organization (as a proxy of the knowledge of the domain) by aggregating their six responses in a score that was normalized on a scale from 0 to 100. The higher the score, the more likely the participant was to adhere to the organization and be knowledgeable about its domain of activity. After rating the organizations, participants read the instructions on the task and filled in a comprehension questionnaire about these instructions.

The second part of the experiment consisted of two stages (Supplementary III.2). The first stage included the veracity judgment task. Participants were divided into two groups that received 48 different stimuli each. Each of the 48 trials started with a fixation cross on the screen (Fig. 1). Then, a brief news, either true or false, was displayed. Participants were asked to report what was, in their opinion, the number of chances out of 100 that this news was true or false. Their response revealed their degree of confidence in their judgment. To respond, participants moved a slider either to the left (False) or to the right (True). The slider started at -100 on the left side and ended at + 100 on the right side. Thus, each move in a direction incremented their degree of confidence by 1%. The elicitation of probabilities was incentivized, following the Karni procedure⁵⁹. Participants were informed that, after the experiment, we would randomly draw eight trials and reward correct veracity judgments in these trials. For each selected trial, one robot out of 100 robots was randomly drawn. To each robot was associated an accuracy level between 0 to 100, corresponding to the probability of this robot to provide the correct answer. Participants were aware that if the randomly drawn robot had an accuracy level higher than their own reported degree of confidence, we would take the robot’s answer into account; otherwise, we would take the participant’s answer into account. Each correct veracity judgment in these eight trials was paid 50 Experimental Currency Units (ECU), with 100 ECU worth $2.

The second stage corresponded to the elicitation of the demand to receive extra information. After validating their veracity judgment and while their screen was still displaying the brief news, participants were asked to choose between receiving or not additional information related to the same news after the completion of the experiment. Finally, they had to report how much they were willing to pay, between 0 and 25 ECU of their 200 ECU initial endowment, to have their decision implemented (i.e., to receive or not receive further information), using the Becker–DeGroot–Marschak (BDM) procedure⁶⁰. In the case participants opted for more information, regardless of whether the information was true or false, they were eligible for receiving a debunk article investigating the content of the brief news in details. Debunk articles were taken from the French fake news debunk platforms Les Décodeurs du Monde, AFP Factcheck and Libération Checknews from the period 2017–2020. The additional information was sent by email to the participants after the experiment. All these aspects were made common knowledge before participants made their choices. At the end of the experiment, we randomly selected eight trials among the 48. For each selected trial, if the participant’s willingness-to-pay (WTP) was equal or above a randomly selected price between 0 and 25 (each price had an equal probability to be drawn), the program deducted the randomly selected price from his or her 200 ECU endowment and his or her decision was implemented. If the WTP was lower than the price, no deduction was operated and the option the participant did not choose was implemented.

At the end of the experiment, participants had to fill in several questionnaires allowing us to measure notably their exposition to information and their degree of curiosity (see Supplementary V). Epistemic curiosity may respond to the desire to stimulate positive feelings of intellectual interest or the desire to reduce undesirable states of information deprivation⁶¹. To check the relationship between veracity assessment, the demand for further information and epistemic curiosity, we administered the Litman questionnaire of Epistemic Curiosity⁶¹. Participants in the second wave of data collection answered additional questions about their perceived share of fake news circulating on Internet and social media. The objective was to check for a potential relationship between distrust in channels of information and veracity estimations.

Data Analysis

We computed power for first-wave (N = 79) data and simulated power for sample sizes up to 250 participants. We employed Mixed Linear Models (MLMs) of the confidence hypothesis, controlled for the veracity judgment and the interaction of news veracity with news theme. With α = .05, the observed fixed effect of confidence on information-seeking choices (β = -0.15) replicates findings from the literature on confidence-based information-seeking²³ and yields a power = .99. For an estimated fixed effect twice lower ( β = -0.72), a simulated N = 150 approximates a power = .99. For an estimated fixed effect three times lower ( β = -0.48), a simulated N = 200 approximates a power = .99. Hence, our sample size of N = 250 adequately tests study hypotheses (Supplementary VI.2).

After collecting data from the second wave, Bayesian analyses were conducted, modeling responses using beta-binomial or normal distributions with non-informative Jeffreys priors. Participant behavior consistency across groups and sessions was confirmed, leading to data pooling (Supplementary VI.3 & VI.4).

To control for objective performance accuracy in veracity judgments, we compared the success proportion in estimating veracity against a random distribution using a logistic function within a Bayesian framework. Our null hypothesis assumes a distribution of behaviors equivalent to randomness. We tested the probability of success at p = .5 and computed a Bayes factor to compare p = .5 and not p = .5. We defined a logistic function with priors for lambda = 0.5 and rscale = 0.5, iterating 10,000 times. Although this rscale is considered a medium value, it represents a tight distribution around the mean in our case. We also computed a logistic function with rscale = 1.5 for a wider distribution.

We tested our hypotheses of participants’ behavior using repeated measures MLMs. We modelled success in estimating veracity (correct or incorrect), veracity judgment (true or false), confidence (level per trial), demand for more information (choice to receive or not), and Willingness-To-Pay (ECUs amount per trial). The random structure of our MLMs included random effects for participants. Registering to the experiment required respecting our inclusion criteria. However, we failed to make reporting age, sex and education mandatory when fulfilling the socio-demographics fields at the beginning of the experiment. In total, three participants did not report their age, four did not report their sex, and 28 did not report their education. When accounting for the socio-demographics, we excluded 30 participants from the models.

We also tested Bayesian hypotheses of success in estimating veracity through separate Bayesian multilevel linear models (Table 1), aligning with models formulated for null hypothesis significance testing. Each model included the variables of interest, a simplified random structure (subject random effects) to save computation time and weakly informative priors (see Supplementary I.2). Models were compared using information criteria, particularly the Widely Applicable Information Criterion (WAIC), which measures predictive accuracy for a new dataset and penalizes models based on their parameter count. Bayesian stacking was employed to average Bayesian predictive distributions, with model weights derived from their information criteria performance, indicating their probability of being the best in terms of out-of-sample prediction⁶².

Finally, we ran a multiple moderated mediation model (see Fig. 5). We used a single model using bootstrapping to evaluate the significance of indirect effects across varying levels of the mediator and moderators. News content imprecision and propensity to polarize were the predictor variables, with veracity judgment moderating and confidence mediating their effects. Reception choice was the outcome variable. Confirmatory factor analysis ensured measurement adequacy and all factor loadings except news content propensity to polarize exceeded 0.6, while composite reliability and average variance extracted surpassed recommended thresholds (0.7 and 0.5, respectively)⁶³.

Acknowledgments

This research has benefited from the financial support of IDEXLYON from Université de Lyon (project INDEPTH) within the Programme Investissements d’Avenir (ANR-16-IDEX-0005) and of the LABEX CORTEX (ANR-11-LABX-0042) of Université de Lyon, within the program Investissements d’Avenir (ANR-11-IDEX-007) operated by the French National Research Agency. This work was also supported by grants from the Agence Nationale pour la Recherche to JCD (ANR-21-CE37-0032), and by MITI 2020 CNRS to JCD and MCV. We thank Pr Edmund Derrington for critically reading and correcting English in the draft of the manuscript.

Author information

Authors and Affiliations

CNRS, Neuroeconomics lab, ISCMJ and Université Claude Bernard Lyon 1, Lyon, France.

V. Guigon, J.-C. Dreher

Univ Lyon, CNRS, GATE UMR 5824, 35 Rue Raulin, 69007, Lyon, France.

V. Guigon, M. C. Villeval

IZA, Bonn, Germany.

M. C. Villeval

Contributions

CRediT author statement: Valentin Guigon: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing – original draft, Writing – review & editing, Visualization. Marie-Claire Villeval: Conceptualization, Methodology, Validation, Resources, Writing – original draft, Writing – review & editing, Supervision, Project administration, Funding acquisition. Jean-Claude Dreher: Conceptualization, Methodology, Validation, Resources, Writing – original draft, Writing – review & editing, Supervision, Project administration, Funding acquisition.

Corresponding author

Correspondence to J.C. Dreher: [email protected]

Competing Interests

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Research Transparency Statement

Funding: This research has benefited from the financial support of IDEXLYON from Université de Lyon (project INDEPTH) within the Programme Investissements d’Avenir (ANR-16-IDEX-0005) and of the LABEX CORTEX (ANR-11-LABX-0042) of Université de Lyon, within the program Investissements d’Avenir (ANR-11-IDEX-007) operated by the French National Research Agency. This work was also supported by grants from the Agence Nationale pour la Recherche to JCD (ANR-21-CE37-0032), and by MITI 2020 CNRS (MITI-2020-247719). Artificial intelligence: No artificial intelligence assisted technologies were used in this research or the creation of this article. Ethics: This research complies with the Declaration of Helsinki (2013), aside from the requirement to preregister human subjects research, and received approval from an internal ethics review board. This research complied with the European data protection regulation (GDPR). Informed consent was obtained from all subjects prior to participation. Preregistration: The study was not preregistered. Materials: All study materials are publicly available (https://osf.io/436pq/?view_only=1292b9f54f7d41a08f6e7274876ff6ae). Data: All primary data are publicly available (https://osf.io/436pq/?view_only=1292b9f54f7d41a08f6e7274876ff6ae). Analysis scripts: All analysis scripts are publicly available (https://osf.io/436pq/?view_only=1292b9f54f7d41a08f6e7274876ff6ae).

Vosoughi, S., Roy, D. & Aral, S. The spread of true and false news online. Science (80-. ). 359, 1146–1151 (2018).
van der Linden, S. Misinformation: susceptibility, spread, and interventions to immunize the public. Nat. Med. 28, 460–467 (2022).
Cinelli, M. et al. The echo chamber effect on social media. Proc. Natl. Acad. Sci. U. S. A. 118, (2021).
Allen, J., Howland, B., Mobius, M., Rothschild, D. & Watts, D. J. Evaluating the fake news problem at the scale of the information ecosystem. Sci. Adv. 6, 1–7 (2020).
Rapp, D. N. & Salovich, N. A. Can’t We Just Disregard Fake News? The Consequences of Exposure to Inaccurate Information. Policy Insights from Behav. Brain Sci. 5, 232–239 (2018).
Tsfati, Y. et al. Causes and consequences of mainstream media dissemination of fake news: literature review and synthesis. Ann. Int. Commun. Assoc. 44, 157–173 (2020).
Van Bavel, J. J., Rathje, S., Harris, E., Robertson, C. & Sternisko, A. How social media shapes polarization. Trends Cogn. Sci. 25, 913–916 (2021).
Brady, W. J., Crockett, M. J. & Van Bavel, J. J. The MAD Model of Moral Contagion: The Role of Motivation, Attention, and Design in the Spread of Moralized Content Online. Perspect. Psychol. Sci. 15, 978–1010 (2020).
Baillon, A., Cabantous, L. & Wakker, P. P. Aggregating imprecise or conflicting beliefs: An experimental investigation using modern ambiguity theories. J. Risk Uncertain. 44, 115–147 (2012).
Pushkarskaya, H., Smithson, M., Joseph, J. E., Corbly, C. & Levy, I. Neural Correlates of Decision-Making Under Ambiguity and Conflict. Front. Behav. Neurosci. 9, 1–15 (2015).
McGrew, S., Breakstone, J., Ortega, T., Smith, M. & Wineburg, S. Can Students Evaluate Online Sources? Learning From Assessments of Civic Online Reasoning. Theory Res. Soc. Educ. 46, 165–193 (2018).
Bak-Coleman, J. B. et al. Combining interventions to reduce the spread of viral misinformation. Nat. Hum. Behav. (2022). doi:10.1038/s41562-022-01388-6
Globig, L. K., Holtz, N. & Sharot, T. Changing the incentive structure of social media platforms to halt the spread of misinformation. Elife 12, 1–23 (2023).
Pennycook, G. & Rand, D. G. Lazy, not biased: Susceptibility to partisan fake news is better explained by lack of reasoning than by motivated reasoning. Cognition 188, 39–50 (2019).
Kozyreva, A., Lewandowsky, S. & Hertwig, R. Citizens Versus the Internet: Confronting Digital Challenges With Cognitive Tools. Psychol. Sci. Public Interes. 21, 103–156 (2020).
Pennycook, G. & Rand, D. G. Nudging social media sharing towards accuracy. Ann. Am. Acad. Pol. Soc. Sci. 700, 152–164 (2022).
Capraro, V. & Celadin, T. “I Think This News Is Accurate”: Endorsing Accuracy Decreases the Sharing of Fake News and Increases the Sharing of Real News. Personal. Soc. Psychol. Bull. 49, 1635–1645 (2023).
Guess, A. M. & Munger, K. Digital Literacy and Online Political Behavior. Polit. Sci. Res. Methods 11, 110–128 (2023).
Traberg, C. S., Roozenbeek, J. & van der Linden, S. Psychological Inoculation against Misinformation: Current Evidence and Future Directions. Ann. Am. Acad. Pol. Soc. Sci. 700, 136–151 (2022).
Jackson, M. O., Malladi, S. & McAdams, D. Learning through the grapevine and the impact of the breadth and depth of social networks. Proc. Natl. Acad. Sci. U. S. A. 119, (2022).
List, J. A., Ramirez, L. M., Seither, J., Unda, J. & Vallejo, B. Toward An Understanding of the Economics of Apologies: Evidence from a Large-Scale Natural Field Experiment. National Bureau of Economic Research (2024). doi:10.1093/ej/ueab062
Pouget, A., Drugowitsch, J. & Kepecs, A. Confidence and certainty: Distinct probabilistic quantities for different goals. Nat. Neurosci. 19, 366–374 (2016).
Desender, K., Boldt, A. & Yeung, N. Subjective Confidence Predicts Information Seeking in Decision Making. Psychol. Sci. 29, 761–778 (2018).
Schulz, L., Fleming, S. M. & Dayan, P. Metacognitive computations for information search: Confidence in control. Psychol. Rev. 130, 604–639 (2023).
Desender, K., Murphy, P., Boldt, A., Verguts, T. & Yeung, N. A postdecisional neural marker of confidence predicts information-seeking in decision-making. J. Neurosci. 39, 3309–3319 (2019).
Balsdon, T., Wyart, V. & Mamassian, P. Confidence controls perceptual evidence accumulation. Nat. Commun. 11, (2020).
Boldt, A. & Yeung, N. Shared neural markers of decision confidence and error detection. J. Neurosci. 35, 3478–3484 (2015).
Fleming, S. M., Weil, R. S., Nagy, Z., Dolan, R. R. & Rees, G. Relating introspective accuracy to individual differences in brain structure. Science (80-. ). 336, 670 (2010).
De Martino, B., Fleming, S. M., Garrett, N. & Dolan, R. J. Confidence in value-based choice. Nat. Neurosci. 16, 105–110 (2013).
Maniscalco, B., Peters, M. A. K. & Lau, H. Heuristic use of perceptual evidence leads to dissociation between performance and metacognitive sensitivity. Attention, Perception, Psychophys. 78, 923–937 (2016).
Pasquali, A., Timmermans, B. & Cleeremans, A. Know thyself: Metacognitive networks and measures of consciousness. Cognition 117, 182–190 (2010).
Fleming, S. M. & Lau, H. C. How to measure metacognition. Front. Hum. Neurosci. 8, 1–9 (2014).
Meyniel, F., Schlunegger, D. & Dehaene, S. The Sense of Confidence during Probabilistic Learning: A Normative Account. PLoS Comput. Biol. 11, 1–25 (2015).
Serra-Garcia, M. & Gneezy, U. Mistakes, Overconfidence, and the Effect of Sharing on Detecting Lies. Am. Econ. Rev. 111, 3160–3183 (2021).
Pennycook, G. et al. Shifting attention to accuracy can reduce misinformation online. Nature 592, 590–595 (2021).
Boldt, A., Blundell, C. & De Martino, B. Confidence modulates exploration and exploitation in value-based learning. Neurosci. Conscious. 2019, 1–12 (2019).
Lee, M. D. & Wagenmakers, E.-J. Bayesian Cognitive Modeling : A Practical Course. (2014).
Sharot, T. & Sunstein, C. R. How people decide what they want to know. Nat. Hum. Behav. 4, 14–19 (2020).
Gilbert, D. T., Krull, D. S. & Malone, P. S. Unbelieving the Unbelievable: Some Problems in the Rejection of False Information. J. Pers. Soc. Psychol. 59, 601–613 (1990).
Reder, L. M. & Kusbit, G. W. Locus of the Moses illusion: Imperfect encoding, retrieval, or match? J. Mem. Lang. 30, 385–406 (1991).
Ferreira, F., Bailey, K. G. D. & Ferraro, V. Good-enough representations in language comprehension. Curr. Dir. Psychol. Sci. 11, 11–15 (2002).
Tomasello, M. The Ontogenetic Foundations of Epistemic Norms. Episteme 17, 301–315 (2020).
Stengelin, R., Grueneisen, S. & Tomasello, M. Why should I trust you? Investigating young children’s spontaneous mistrust in potential deceivers. Cogn. Dev. 48, 146–154 (2018).
Capraro, V., Schulz, J. & Rand, D. G. Time pressure and honesty in a deception game. J. Behav. Exp. Econ. 79, 93–99 (2019).
Sperber, D. et al. Epistemic vigilance. Mind Lang. 25, 359–393 (2010).
Ambuehl, S. & Li, S. Belief updating and the demand for information. Games Econ. Behav. 109, 21–39 (2018).
Moore, D. A. & Healy, P. J. The Trouble With Overconfidence. Psychol. Rev. 115, 502–517 (2008).
Boldt, A., Gardelle, V. De & Yeung, N. The Impact of Evidence Reliability on Sensitivity and Bias in Decision Confidence. J. Exp. Psychol. Hum. Percept. Perform. 43, 1520–1531 (2017).
Weber, N. & Brewer, N. Confidence – Accuracy Calibration in Absolute and Relative Face Recognition Judgments. 10, 156–172 (2004).
Moore, D. A. & Schatz, D. The three faces of overconfidence. Soc. Personal. Psychol. Compass 1–12 (2017). doi:10.1111/spc3.12331
Atanasov, P., Witkowski, J., Ungar, L., Mellers, B. & Tetlock, P. Small steps to accuracy: Incremental belief updaters are better forecasters. Organ. Behav. Hum. Decis. Process. 160, 19–35 (2020).
Chang, W., Chen, E., Mellers, B. & Tetlock, P. Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments. Judgm. Decis. Mak. 11, 509–526 (2016).
Mellers, B. et al. Psychological Strategies for Winning a Geopolitical Forecasting Tournament. Psychol. Sci. 25, 1106–1115 (2014).
Moore, D. A. et al. Confidence calibration in a multiyear geopolitical forecasting competition. Manage. Sci. 63, 3552–3565 (2017).
Roozenbeek, J., van der Linden, S., Goldberg, B., Rathje, S. & Lewandowsky, S. Psychological inoculation improves resilience against misinformation on social media. Sci. Adv. 8, 1–12 (2022).
Celadin, T., Capraro, V., Pennycook, G. & Rand, D. G. Displaying News Source Trustworthiness Ratings Reduces Sharing Intentions for False News Posts. J. Online Trust Saf. 1, 1–20 (2023).
Pennycook, G., Binnendyk, J., Newton, C. & Rand, D. G. A practical guide to doing behavioral research on fake news and misinformation. Collabra Psychol. 7, 1–13 (2021).
Tappin, B. M., Pennycook, G. & Rand, D. G. Bayesian or biased? Analytic thinking and political belief updating. Cognition 204, 104375 (2020).
Karni, E. A Mechanism for Eliciting Probabilities. Econometrica 77, 603–606 (2009).
Becker, G. M., DeGroot, M. H. & Marschak, J. Measuring utility by a single-response sequential method. Behav. Sci. 226–232 (1964). doi:10.1002/bs.3830090304
Litman, J. A. Interest and deprivation factors of epistemic curiosity. Pers. Individ. Dif. 44, 1585–1595 (2008).
Anderson, D. R. & Burnham, K. P. Avoiding Pitfalls When Using Information-Theoretic Methods. J. Wildl. Manage. 66, 912–918 (2002).
Fornell, C. & Larcker, D. F. Evaluating Structural Equation Models with Unobservable Variables and Measurement Error. J. Mark. Res. 18, 39 (1981).

Tables 1 and 2 are available in the Supplementary Files section.

There is NO Competing Interest.

Download PDF

Version 1

posted

You are reading this latest preprint version

Metacognition during fake news detection induces an ineffective demand for disambiguating information

Status:

Version 1

Abstract

Figures

Introduction

Results

Judgment of news veracity and success rate

Uncalibrated metacognitive sense of confidence

Demand and avoidance of extra information

Discussion

Materials and Methods

Participants

Task and Design

Data Analysis

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Version 1