Neural signatures of emotional inference and experience align during social consensus

doi:10.21203/rs.3.rs-3487248/v1

Download PDF

Article

Neural signatures of emotional inference and experience align during social consensus

https://doi.org/10.21203/rs.3.rs-3487248/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Humans seamlessly transform dynamic social signals into inferences about the internal states of the people around them. To understand the neural processes that sustain this transformation, we collected fMRI data from participants (N = 100) while they rated the emotional intensity of people (targets) describing significant life events. Targets rated themselves on the same scale to indicate the intended “ground truth” emotional intensity of their videos. Next, we developed two multivariate models of observer brain activity– the first predicted the “ground truth” (r = 0.50, p < 0.0001) and the second predicted observer inferences (r = 0.53, p < 0.0001). When individuals make more accurate inferences, there is greater moment-by-moment concordance between these two models, suggesting that an observer's brain activity contains latent representations of other people’s emotional states. Using naturalistic socioemotional stimuli and machine learning, we developed reliable brain signatures that predict what an observer thinks about a target, what the target thinks about themselves, and the correspondence between them. These signatures can be applied in clinical data to better our understanding of socioemotional dysfunction.

Biological sciences/Psychology

Biological sciences/Neuroscience/Social behaviour

The “ground truth” emotional state of another person can be predicted from an observer’s brain activity.

An observer’s inferences about another person’s emotional state can also be predicted from the observer’s brain activity.
Neural patterns predictive of “ground truth” and inference are completely dissociable.
When our brain-based patterns of socioemotional “ground truth” and inference align, participants yield more accurate inferences.

Healthy social functioning increases life expectancy (Holt-Lunstad et al., 2010; Yang et al., 2016), buffers cognitive decline (Cacioppo & Cacioppo, 2014; Fakoya et al., 2020; Landeiro et al., 2017; Read et al., 2020; Ren et al., 2023), improves mental health conditions like depression (Luanaigh & Lawlor, 2008), and enriches a person’s daily life (Goodwin et al., 2001). The quality of a social interaction is shaped by two essential behaviors–how we signal our emotions and how we infer the emotions of others (Mehu & Scherer, 2012). People who make clear signals and accurate inferences tend to have healthy adolescent adjustment, stable relationships, and high subjective well-being (Gilbert et al., 2020; Hall et al., 2009). Conversely, people who make ambiguous signals and inaccurate inferences are more likely to experience social isolation, or meet criteria for clinical conditions like Autism Spectrum Disorder (ASD; Cassidy et al., 2014), schizophrenia (Green et al., 2008; Takano et al., 2017; Vucurovic et al., 2020), social anxiety (Button et al., 2012; Hirsch & Mathews, 2000), and borderline personality disorder (Baez et al., 2015; Dziobek et al., 2011). Therefore, a developed understanding of how the brain forms socioemotional inferences is clinically advantageous and has the potential to reduce loneliness and improve public health.

The neural processes that lead to conscious inference are computationally complex and require the integration of multiple sources of information such as an observer’s internal homeostatic state, their past experiences, expectations, and social schemas (Chang et al., 2021; Pugh et al., 2022). Neuroimaging studies have implicated the amygdala, medial prefrontal cortex (mPFC), temporoparietal junction (TPJ), and precuneus (Skerry & Saxe, 2014; Spunt & Adolphs, 2019; Zaki et al., 2009) in socioemotional inference; however, integrated models of how multiple brain regions interact with one another to form an inference are lacking. Prior research also fails to disentangle signal intent from inference. This is partly because the “ground truth” of the signaler’s intent is not available when stimuli are derived from normative data sets or actors, instead of real people. Previous work from our lab compared observer inferences to storytellers’ self-reported “ground truth” ratings and found that accuracy is associated with higher activation in the inferior parietal lobule (IPL), premotor cortex, superior temporal sulcus (STS), and mPFC (Zaki et al., 2009). Accuracy, however, is not a neural process, but an outcome of social signal processing. To better characterize how the brain processes social signals and forms a socioemotional inference, it is necessary to disentangle a target’s “ground truth” intent from an observer’s inference and develop multivariate models of brain activity that can be combined to explain an individual’s inference accuracy.

Naturalistic stimuli are necessary for the study of social signal processing (Zaki & Ochsner, 2009). Though static and unimodal stimuli are useful for isolating specific components of emotion perception and induction, low dimensional stimuli forgo the complexity of the dynamic environments we live in and limit our ability to study social behavior (Eickhoff et al., 2020; Jääskeläinen et al., 2021; Zaki & Ochsner, 2009). Naturalistic audiovisual recordings allow us to capture the subtleties of nonverbal communication, such as nuanced facial expressions, tone of voice, and body language. Moreover, naturalistic stimuli reduce demand characteristics (Grimm, 2010) and allow us to examine the context in which social interactions occur. For instance, studying how people recognize distress in real-life situations provides a better understanding of empathic inference than asking people to rate emotions in a hypothetical situation. First-person narratives, such as storytelling, are particularly opportune stimuli for the study of real-world social interactions. Such stimuli can be sufficiently constrained in the laboratory by limiting spurious background activity while still allowing for the natural unfolding of dynamic socioemotional signaling (Ong et al., 2021). Overall, studying naturalistic stimuli can lead to a more accurate understanding of real-world social interactions, which can inform the development of treatments for social and emotional disorders.

To improve our understanding of how an observer’s brain perceives signals and forms inferences in naturalistic contexts, we collected videos of people (targets) describing emotional events in their lives. After being recorded, targets provided moment-by-moment ratings of how they felt as they spoke. These self-reports constitute the “ground truth” of their social signals. We then scanned a second set of participants (observers) with fMRI while they viewed targets and rated what they thought the target was feeling at each moment on the same scale as targets themselves (see Fig. 1). These ratings constitute the observer’s dynamic inferences. Correlations between observer inference and target self-report served as a measure of “empathic accuracy,” or agreement between two people about what one of them feels. We then used machine learning to develop two neural signatures of socioemotional processing– one indicative of the signaler’s intent (i.e., the “ground truth”) and one indicative of the observer’s inference–and tested if their correspondence is related to empathic accuracy. This work extends our understanding of how the human brain forms socioemotional inferences in three novel ways. First, we use high quality and well-curated naturalistic stimuli that reflect people’s real-world experiences. Second, we use multivariate methods to develop integrated brain models of socioemotional processing. Finally, we investigate the “ground truth” intentions of social targets instead of relying solely on observer inferences to understand an observer’s behavior. This allows us to analyze how a target’s intended signals are perceived, and how that initial perception is related to the observer’s conscious inference. Together, this work advances our scientific understanding of the brain processes that support complex, real-world emotion inference.

The “ground truth” intended emotional intensity of another person’s social signals can be predicted from an observer’s brain activity

We developed a novel whole-brain fMRI pattern that can predict, from an observer’s brain activity (N = 100), the “ground truth” intended emotional intensity of the target (Fig. 1B). The model’s average within-subject correlation between predicted and actual ratings was r = 0.50 ± 0.02 (standard error), p < 0.0001 (Fig. 2A). It was developed using least absolute shrinkage and selection operator-regularized principal components regression (LASSO-PCR; see STAR methods) with leave-one-subject-out cross validation (LOO-CV). The accuracy of our model is on par with other emotion models in the literature (see Figure S4) and our model’s validity was verified in a held-out validation set (average r = 0.19 ± 0.002; t(99) = 9.65, p < 0.0001; see Fig. 2B). To test its specificity, we repeated this analysis in the inference validation set. The “ground truth” model fit the “ground truth” validation set better, but the difference was not significant (average r = 0.18 ± 0.002; t(99) = 0.48, p = 0.63) in a two-tailed paired t-test. Note the stimuli in the validation set are unimodal dynamic stimuli (audio-only and visual-only), while the training data are multimodal (audiovisual). Further testing of this model on audiovisual and other socioemotional stimuli is necessary to determine its out-of-sample sensitivity and specificity.

Our model’s ability to decode the target’s self-reported emotional state indicates that there is a latent representation of a social signal’s “ground truth” in the brains of observers. This latent representation is comprised of a distributed pattern of brain activity that relies most significantly on activity in the right visual and anterior insular cortices as well as the right angular gyrus, left posterior cingulate (PCC), bilateral precuneus, and bilateral superior and inferior frontal gyri (5,000 bootstrap samples; FDR corrected q < 0.05; Fig. 2C; see Table S1 for complete list).

We compared the unthresholded brain pattern to other brain maps in the NeuroSynth database (Yarkoni et al., 2011) to characterize functions associated with this network. The “ground truth” pattern is largely unique from previously published beta maps but is most similar to those characterizing brain activity related to ‘resting state,’ ‘theory of mind,’ ‘person,’ ‘social, ‘autobiographical,’ beliefs,’ ‘spatial,’ ‘[scene] construction,’ ‘speech,’ and ‘self-referential processing,’ respectively (Fig. 2D). The unthresholded brain pattern was also compared to well-validated brain-based models of emotion, empathy, and interoception to further gauge its sensitivity and specificity the “ground truth” intensity of a target’s social signal (see STAR Methods and Figure S5). The “ground truth” model was dissimilar from emotion models that predict how an observer feels in response to an image or event (PINES cosine similarity = -0.03; Chang et al., 2015; NPS cosine similarity = -0.03; Wager et al., 2013; social rejection cosine similarity = -0.04; Woo et al, 2014). However, it was weakly similar to naturalistic models of empathic care (cosine similarity = 0.06) and distress (cosine similarity = 0.10; Ashar et al., 2016). Overall, this suggests that the “ground truth” brain pattern is not reducible to the general processing of affect but is capturing information related to the comprehension of dynamic socioemotional schemas.

An observer’s socioemotional inference can be predicted from their brain activity

We identified a pattern of brain activity that correlated with the level of intensity observers (N = 100) ascribed to storytellers on a moment-to-moment basis (Fig. 1C). The model’s average within-subject correlation between predicted and actual ratings was r = 0.53 ± 0.02, p < 0.0001 (Fig. 3A). This model was trained using LOO-CV LASSO-PCR to predict observer’s inferences (see STAR Methods). The inference model was verified in its held-out validation set (average r = 0. 32 ± 0.002; t(99) = 12.48, p < 0.0001). The model was specific to observer inferences– it’s performance on its own validation set (r = 0.23 ± 0.002) was significantly higher than its performance on the “ground truth” validation set in a two-tailed paired t-test (t(99) = 2.77, p = 0.007, CI = [0.03–0.16], Cohen’s d = 0.33; Fig. 3B).

The brain regions that most significantly contributed to the formation of socioemotional inferences include the cerebellum (bilateral crus), left precuneus, right primary somatosensory cortex, right inferior frontal gyrus, bilateral superior medial frontal gyrus, bilateral lingual gyrus, bilateral temporal pole, and bilateral anterior insular cortex (5,000 bootstrap samples; FDR corrected q < 0.05; see Table S2 for complete list; see Fig. 3C thresholded at p < 0.01 uncorrected).

Again, we compared the unthresholded brain pattern to other published brain maps to characterize functions associated with this network. The networks and functions in the NeuroSynth database which were most similar to the inference model were: ‘resting state,’ ‘somatosensory,’ ‘person,’ ‘theory of mind,’ ‘social,’ ‘spatial,’ ‘foot,’ ‘moral’, ‘self-referential,’ and ‘beliefs,’ respectively (Fig. 3D). The inference model was unique from the “ground truth” model in that it was associated with maps related to somatosensory simulation and bodily action. When we compared the inference model to the brain-based models of emotion, empathy, and interoception in Figure S5, we found that like the “ground truth” model, the inference model is unique from existing signatures but has a small positive cosine similarity with models of dynamic human-to-human empathic processing. Overall, these comparisons indicate that our model of socioemotional inference draws largely from mentalizing networks.

Neural patterns underlying “ground truth” signal intent and inference are dissociable

Both the “ground truth” and inference models were developed on observer brain activity, however, they reveal dissociable components of socioemotional processing (cosine similarity = 0.29). Cosine similarity ranges from − 1 to 1, where values closer to 0 indicate the two vectors are orthogonal to each other. When the value is closer to 1, images are more similar. When the value is closer to -1, images are more opposite. To further investigate their dissociability, we assessed general overlap of voxels that significantly contribute to each model’s performance. After thresholding, the only voxel-overlap was in the right anterior insula (Figure S6), however, voxel-wise patterns in the bilateral insular cortex (mask from Harvard-Oxford Atlas) were unique (cosine similarity = 0.22). Together, these metrics indicate that the two models capture unique components of socioemotional brain activity.

Next, we tested if bran activity patterns related to “ground truth” and inference are separable at each level of stimulus intensity. To do this, we trained a binary linear SVM (see STAR Methods) to separate subject-level “ground truth” and inference maps at each level of rating intensity. Each classification accuracy was significantly greater than chance (level 1: Acc. = 70.00% ±3.3%, p < 0.0001, Area Under the Curve (AUC) = 0.69; level 2: Acc. = 69.00% ±3.3%, p < 0.0001, AUC = 0.75; level 3: Acc. = 70.00% ±3.3%, p < 0.0001, AUC = 0.73; level 4: Acc. = 69.00% ± 3.3%, p < 0.0001, AUC = 0.68; level 5: Acc. = 70.00% ±3.3%, p < 0.0001, AUC = 0.75) indicating that the “ground truth” and inference activity patterns are linearly separable, and therefore, represent unique neural processes (Fig. 4A). To better understand the brain regions that distinguish between “ground truth” and inference, the predictive voxel weights for the classifier trained at the highest level of intensity (level 5) was bootstrapped (5,000 samples) and thresholded (FDR q < 0.05) so that the brain regions where “ground truth” and inference maximally diverge could be compared (Fig. 4B). Patterns of activity in the dorsal anterior cingulate (dACC), PCC, anterior insula, pallidum, and precuneus maximally separate “ground truth” and inference at the highest level of socioemotional intensity. These regions correspond with those that diverge at the lowest level of intensity (see Figure S7). Together, these analyses indicate that the multivariate patterns which comprise the “ground truth” and inference models are dissociable from one another and therefore reflect unique, but simultaneous, components of socioemotional processing.

When the brain-based models of a social signal’s “ground truth” and an observer’s inference align, observers are more empathically accurate

After we identified these two unique components that underlie socioemotional processing, we sought to test how they interact in relation to an individual person’s empathic accuracy. To do this, we applied the “ground truth” and inference models to participant-level brain activity when participants made inaccurate inferences (low empathic accuracy) and accurate inferences (high empathic accuracy; see Fig. 5A and STAR Methods for details). Then we correlated the predictions of the two models across all participants. When participants are inaccurate, there is more variance across the predictions of the “ground truth” and inference models, and, therefore, they are weakly correlated (r = 0.28, p < 0.01; Fig. 5A). However, when participants are highly accurate, there is less variance between the model predictions, and they are highly correlated (r = 0.64, p < 0.001). The alignment between the two models was significantly greater during high accuracy performance than low accuracy performance (z = -2.71, p = 0.003; Fig. 5B). We verified this effect in the validation trials (low empathic accuracy alignment r = 0.58, p < 0.0001; high empathic accuracy alignment r = 0.79, p < 0.0001; two-tailed z-test of the correlation difference z = 2.83, p = 0.005; Fig. 5C). This analysis indicates that when an observer is making accurate inferences, both the neural patterns underlying the “ground truth” of the socioemotional signal and the observer’s inference are highly similar to the observer’s brain activity at that moment. That is, greater concordance between the latent representation of the target’s intended signal intensity and the observer’s inference indicates greater empathic accuracy.

Brain regions associated with the divergence from the “ground truth” during inaccurate inference

In a post-hoc exploratory univariate analysis, brain activity during low accuracy trials (see schematic in Fig. 5A) was used to predict the pattern expression of the inference model while controlling for the pattern expression of the “ground truth” model. ‘Pattern expression’ is the dot-product between two multivariate patterns (see STAR Methods for details). Low accuracy trials were selected so we could isolate regions where inference-related activity maximally diverged from latent “ground truth”-related activity. The resulting model weights were corrected for multiple comparisons (FDR q < 0.05, k = 25). Positive clusters were revealed in the right primary somatosensory cortex (S1) and right parahippocampal gyrus (Fig. 6). Negative clusters were revealed in the left insula and left primary motor cortex (M1). This activity is dissociable from maps of finger tapping in the NeuroSynth database (cosine similarity = -0.07). These results suggest that activity in S1, M1, and parahippocampal gyrus contribute to the formation of a dynamic socioemotional inference, above and beyond the latent “ground truth” representation; albeit, here, activity in these regions contribute to the formation of an incorrect inference.

The effectiveness of social signaling depends upon both the intended content of the signal (i.e., the socioemotional “ground truth”) and the interpretation of the signal by an observer (i.e., socioemotional inference; Mehu & Scherer, 2012). To better understand how social signaling is processed in the brain, we sought to dissociate the neural patterns underlying “ground truth” and inference in a dynamic naturalistic storytelling paradigm and test how these patterns relate to empathic accuracy. We found that both “ground truth” (i.e., the target’s self-reported emotional intensity) and an observer’s inferences about the target’s emotional intensity could be predicted from observers’ brain activity. The multivariate brain patterns derived from these predictions are dissociable; however, when the models’ predictions align, observers make more accurate inferences. These findings suggest that there is some latent representation of a target’s “ground truth” emotional intensity that observers transform into conscious inference. Moreover, these findings suggest that participants quickly and accurately represent another’s mental state even when they make an incorrect inference.

How can it be possible that participants “represent” the “ground truth” of another person’s social signals and that this “representation” can be dissociable from the participants’ conscious inferences? Human adults have highly adept schemas of social information that are activated when they perceive prototypical socioemotional expressions (Izard, 2007). Prior work has found that rich, category-specific visual features can be readily mapped to distinct emotions that are coded across many brain regions including primary visual cortex (Kragel et al., 2019). Unless a target is intentionally trying to deceive an observer, or has a disorder that impacts socioemotional communication, a target will convey information in a manner that will activate the correct schema in an observer (Chang et al., 2021). That is the function of social signaling– to convey information in a manner that will be quickly and accurately understood by the individual an organism wishes to convey it to (Mehu & Scherer, 2012). We suspect that the “ground truth” pattern revealed in this investigation is capturing this process of schema-activation in the observers. Schema-activation in this investigation is (a) specific to the intensity of a signal and (b) highly dynamic and multimodal.

The “ground truth” pattern relies significantly on brain areas implicated in speech comprehension (right angular gyrus; Seghier, 2013) and scene construction (PCC and calcarine sulcus; Irish et al., 2015), as well as mentalizing and social cognition (bilateral superior frontal gyrus, precuneus, and anterior insula; Van Overwalle & Baetens, 2009). It does not, however, rely on the amygdala– a brain region heavily implicated in emotion perception (Spunt & Adolphs, 2019). The whole brain “ground truth” pattern is also unique from PINES, which is a multivariate pattern that can predict the emotional intensity of negative static images with high accuracy (Chang et al., 2015). PINES was trained on emotional images, only some of which were social. Therefore, it is possible that our “ground truth” pattern is specific to the processing of socioemotional schemas. This pattern may be something the brain constructs in dynamic social interactions that helps it to create a stable representation of the social target. This interpretation is inspired by constructionist theories of emotion, which state that emotions are not fixed entities that have distinct and specific brain circuits, but flexible processes that are inseparable from the context in which they emerge (Barrett, 2017; Russell, 2003).

It is important to note that none of the subcortical regions commonly implicated in emotion perception (i.e., the amygdala, striatum, and periaqueductal gray) were significantly weighted in the “ground truth” or inference models. Instead, the distributed patterns primarily included cortical regions that perform complex multisensory integration (i.e., angular gyrus, frontal gyrus, PCC, temporal pole, and the precuneus; Scheliga et al., 2022). From a constructionist viewpoint, the socioemotional “ground truth” pattern may reflect neural processing that abstracts the literal input (i.e., facial expressions, body movements, speech and vocal intonations) into socioemotional schemas related to intensity. Similarly, the inference model may represent a subsequent stage of processing, where the information the target signaled is related to the observer’s past and current experiences and their expectations for the future. Indeed, the inference pattern relied both on mentalizing networks and brain areas implicated in social abstraction and somatosensory processing– the temporal pole and S1, respectively (Quandt et al., 2017). Being that the “ground truth” and inference models were (1) dissociable, (2) verifiable in their own held-out validation sets, and (3) unique from other published models of emotion induction, like PINES, we suspect that they capture unique components of social signal processing– schema activation and deliberate inference formation– that can only be experimentally-evoked by dynamic, naturalistic stimuli.

When the “ground truth” and inference models’ predictions aligned in the brains of the observers, observers made more accurate inferences. Furthermore, the two unique models could be combined to predict the empathic accuracy of observers in held-out validation trials. Interestingly, when individuals made inaccurate inferences, activity in right S1 and PHG increased with the intensity of their inferences when controlling for the intensity of the “ground truth” representation. This suggests that somatosensory simulations may support the transformation of socioemotional “ground truth” into a conscious, reportable inference. Part of the transformation from “ground truth” to inference in this paradigm requires a motor response: Observer’s must update their ratings with a button press. Therefore, it is difficult to disentangle this brain pattern from button pressing entirely; however, this pattern of brain activity was dissimilar from patterns associated with finger tapping in the NeuroSynth database. Furthermore, though the left lateralization of activation in M1 was consistent with the (right) hand making the ratings, the positive clusters in S1 and the PHG were ipsilateral to the ratings-hand, suggesting this activity is not reducible to task-induced motion. This is consistent with prior research that shows S1 activates during motor imagery and empathy (Blakemore et al., 2005; Hooker et al., 2010; Jafari et al., 2020; Schaefer et al., 2020). Furthermore, prior studies of imagination have found right lateralization of imagined stimuli (Reddan et al., 2018). Indeed, a growing thrust of empathy research purports a role for somatosensory simulations in the understanding of social interactions (Genzer et al., 2022; Jospe et al., 2020, 2022). As for the PHG, activity in this region is implicated in episodic memory and the processing of scenes (for a review see Aminoff et al., 2013). Taken together, these results suggest that inference involves an internalizing of the events described by the target, and that people simulate the actions described and relate them to their own prior experiences and expectations.

This study has several limitations. First, the “ground truth” and inference ratings occur spontaneously and sometimes simultaneously in this paradigm, therefore, we are unable to directly model the transformation of “ground truth” into inference in the brain. Second, though the stimuli themselves are highly dynamic and complex, the models are trained to predict only a single dimension of socioemotional information: intensity. This was done because naturalistic stories often signal positive and negative information at a faster rate than we can sample the brain data (see Polimeni & Lewis, 2021). For example, a participant may be describing both the sadness and the love they felt after the death of a family member. These are intense, complex emotions, therefore, removing valence from individual ratings allowed us to better model dynamic shifts in emotion signaling and to isolate signatures of signal intent from observer inference. Further validation of our models on other naturalistic and social audiovisual data is necessary to determine their sensitivity and specificity to both a target’s self-reported internal emotional state and an observer’s conscious inference of that state.

Various neuroimaging studies have attempted to predict aspects of socioemotional processing from human brain activity (see Figure S4 for a summary), however, this is one of the first investigations to situate socioemotional processing within the ethological framework of social signaling and inference. Most existing models were developed to predict an observer’s internal emotional state after it was experimentally influenced by an image or story, and the stimuli are often exaggerated social signals (e.g., an image of an angry actor pointing a gun). This is the first investigation to delineate brain activity related to the “ground truth” of social signals from brain activity related to the inferences observers make when they perceive these signals. The dissociability of these processes provides a new foundation for emotion research and delineates new promising targets for clinical intervention.

Sharing information is essential to the well-being of individuals and the communities they are a part of because individuals must interact with each other to achieve personal needs that cannot be achieved alone. Effective signaling can engender social bonds, mutual aid, and collaboration (Mauss et al., 2011). Ineffective signaling, however, can be costly. A missed alarm call can result in death, while misunderstanding social signals can result in ostracization or rejection. People with ASD experience this from both ends: They have difficulty understanding the intentions, thoughts, and feelings of others and have difficulty being understood. As a result, people with ASD experience high levels of loneliness and social isolation relative to other disability groups (Causton-Theoharis et al., 2009; Mazurek, 2014). However, little research emphasis has been placed on people’s ability to decode the social signals of people with ASD. The current study creates a new avenue for such investigations because it allows researchers, for the first time, to disentangle a signal’s intent from an observer’s inference, in dynamic real-world situations.

In summary, our study delves into the intricate processes of how humans interpret dynamic social signals and make inferences about the internal states of others. Using naturalistic stimuli, fMRI data, and machine learning, we established two distinct neural signatures: one predicting the "ground truth" intended emotional intensity of a target and another predicting an observer's inferences. These neural patterns are dissociable, indicating that they are separate components of socioemotional processing. Notably, when these brain-based models align, individuals demonstrate higher empathic accuracy. This work offers valuable insights into the brain processes that underlie the interpretation of social signals and has the potential to inform treatments for social and emotional disorders.

CONTACT FOR RESOURCE SHARING

Further information and requests for resources and code should be directed to and will be fulfilled by the Lead Contact, Marianne Reddan ([email protected]).

EXPERIMENTAL MODEL AND SUBJECT DETAILS

One hundred (59 Women, 37 Men, 4 No Response, average age = 25.23 STD = 9.96) adult healthy members of the Stanford University community participated in this study. Forty-two participants identified as White, 32 as Asian American, 14 as Hispanic/Latinx, 7 as Black, and 5 did not report their race (see Supplementary Fig. 1A). Participants were asked to provide their gender in binary terms (male or female). Participants were asked to complete the MacArthur Scale of Subjective Social Status (Adler et al., 2000). Average perceived socioeconomic status (SES) on the 10-point ladder scale was 6.71 (STD = 1.57). SES was skewed towards the upper and upper middle classes (Mode = 7; Median = 7; Supplementary Fig. 1B). Eighty-nine participants reported having at least some college education (this includes those currently enrolled as Stanford undergraduates). Eighty-three reported their parents had at least some college education. Participants were all right-handed.

Participants were recruited through internal and surrounding communities near Stanford University in Stanford, CA. All participants gave informed consent and were paid for their participation. This study was approved by the Stanford University Institutional Review Board.

METHOD DETAILS

Experimental procedures

Eligibility Criteria. Participants were required to be between the ages of 18 to 65. Participants who had contraindications for the MRI environment were excluded from this study. Sample size was determined via previously published studies of empathic accuracy and narrative storytelling (Zaki et al., 2009).

Stimuli. We selected a subset of 24 videos (19 unique targets or storytellers) from a curated high-quality video dataset of 193 video clips of 49 volunteers describing emotional life events known as the Stanford Emotional Narratives Dataset (SENDv1; Ong et al., 2021). The targets used in this selection were balanced for gender (11 Women and 8 Men) and racial representation (9 White, 5 Black, 4 Asian, and 1 Latinx/Hispanic; see Supplementary Fig. 1A). Many of the targets were current students. "Ground truth" self-ratings of emotional valence were obtained from targets on a moment-by-moment bipolar scale (very negative to very positive). Twelve of the 24 videos featured negative life events, while the other 12 featured positive life events. Videos were cut down in length so that they ranged from 1–3 minutes. Five targets contributed two separate videos (one positive, one negative) to this dataset, however, trials with duplicate targets were not used in the main analysis (see Experimental Session for more information). Stimuli were displayed using Psychtoolbox v3 via MATLAB R2017B.

Randomization. This is a within-subjects design, where stimulus presentation was pseudorandomized in three a priori orders. Each subject was randomly assigned to an order when they signed up for the experiment. Video presentation and intertrial interval (ITI) lengths were shuffled and fixed to an order number.

Practice Session. All participants completed a “practice session” before the experiment where they learned how to make their ratings on the slider provided. Participants viewed 10-s clips of stimuli from the SENDv1 dataset that are unique from those used in the study.

Experimental Session. All 100 participants were shown short movies of first-person emotional narratives while inside the fMRI environment. While viewing, participants rated the emotional-narrative stimuli moment-by-moment on a bivalent scale by way of button presses that moved a slider on screen. Participants pressed the buttons to move it right or left with their index and middle finger, respectively, of their right hand. There were three sensory conditions in this experiment: a visual-only condition, where participants could see but not hear the videos (8 trials); an auditory-only condition where they could hear but not see the videos (8 trials); and an audiovisual condition where they could both see and hear the videos (8 trials). There were 24 trials total, divided into two runs of 12 trials each (See Supplementary Fig. 2A). The audiovisual condition (8 trials) comprises the model training data and is the modality of interest in the present study. The other two conditions were combined and used as a held-out validation set. Due to a coding error, only 23 videos were spread across the audiovisual condition during counterbalancing (the missing video was negatively valenced; see Supplementary Fig. 3 for a depiction of the valence of all the audiovisual stimuli). Five targets contributed two separate videos (one positive, one negative) to this dataset, however, the sensory channel of presentation was not repeated within targets. Videos were separated by an ITI (fixation cross) that ranged from 4–12 s.

Post-Experiment Session. Immediately after the fMRI experiment, participants were shown 10 s clips of each video they watched in the scanner, in the same sensory modality (auditory-only, visual-only, and audiovisual) and order that was presented to them in the scanner. First, participants answered True-False questions about the story to assess comprehension and attention. Next, participants were asked if the target was familiar to them. The targets were recruited on Stanford campus; therefore, it was important to confirm that our participants were not friends or acquaintances with the targets. Participants were also asked to rate the overall emotional valence of the stimulus on the same bipolar scale they used during the experiment. This session was displayed via MATLAB R2017B on an Apple laptop computer in an experiment room in the lab.

DATA COLLECTION

Ratings acquisition

Participants were asked to “rate how positive or negative you think the target is feeling moment to moment” on a bipolar visual analog scale (VAS) scale with anchor points ‘very negative’ to ‘very positive.’ Participants did not see any number responses, but their ratings were recorded on a scale from 0 to 100. Ratings were collected via a 2x4 bimanual button box through the fORP 932 modular response box system. Ratings were collected through Psychtoolbox v3 (http://psychtoolbox.org/) in MATLAB R2017B, and were sampled by Psychtoolbox every 0.5 s. One button press moved the slider by 12 pts. The slider could be moved to the right or left via buttons pressed by the index and middle finger on the participant’s right hand.

Neuroimaging acquisition

The study was conducted at the Stanford University Center for Cognitive and Neurobiological Imaging (CNI) using a 3T GE Discovery MR750 scanner (running ESE version 23.1 v02: the GE operating system) and a 32 channel Nova Medical head coil. The scan began with a 3-plane localizer or ‘scout’ scan to find the participant’s head. Next, we collected a 3D T1 weighted anatomical scan at 0.9 mm isotropic resolution (Flip Angle = 12; FOV = 23). This scan uses GE’s “BRAVO” sequence. This was followed by a higher-order shim and a field map was collected before each functional scan. Functional scans used a gradient echo sequence (Full brain BOLD EPI, TR = 2 s, TE = 25 ms, FOV = 23.2, Flip Angle = 77, 46 slices at a 2.9 mm slice thickness, Acquisition Order = Interleaved, voxel size = 2.9mm^3). A 3x in-plane acceleration was used to reduce EPI distortion. Functional image acquisition was divided into two runs. Between runs there was a break of approximately 15–30 s where the experimenter checked the participant to make sure they were comfortable and alert.

QUANTIFICATION AND STATISTICAL ANALYSIS

Outlier removal

No outlier subjects were found nor removed from this dataset. One participant experienced a technical error where they were shown duplicate trials. Duplicate trials were removed from this participant’s data prior to analysis.

Preprocessing of the “ground truth” and inference ratings

“Ground truth” and inference ratings were down sampled from (0.5 s) to the rate of the TR (2 s). Next, both sets of ratings were range-normalized within participants (for observers, range normalization was done across all video stimuli). Range-normalization involved subtracting the minimum of that participant's ratings then dividing by the [max - min] of that person’s ratings, from a rating at any given time point. This is also known as feature scaling, and it yields values that are shifted and rescaled so that all values range between 0 and 1. Next, those ratings were adjusted into 5 levels of valence-independent emotional intensity. That is, ratings between 41–50 and 51–60 were set to level 1; ratings between 31–40 and 61–70 to level 2; ratings between 21–30 and 71–80 to level 3; ratings between 11–20 and 81–90 to level 4; and ratings between 0–10 and 91–100 to level 5. Level 5 is the highest intensity rating. This was done because naturalistic stories often combine positive and negative emotions and can signal positive and negative information at a faster rate than we can sample the brain data. For example, a participant may be describing both the sadness they felt and the love they felt after the death of a family member. These are both intense, complex emotions, so removing valence from individual ratings allows us to better model dynamic shifts emotion signaling and to isolate signatures of signal intent from an observer’s inference.

Derivation of empathic accuracy ratings

Empathic accuracy ratings were constructed by first subtracting the normalized “ground truth” ratings from the normalized observer inference ratings, across the whole time series. These ratings were then converted into 5 levels where the highest level 5 reflects times where there was no difference between the observer’s ratings and the “ground truth;” that is, when observers were most accurate. The lowest level 1 reflects when there was a maximal difference between the observer’s ratings and the “ground truth” in either direction; that is, when observers were least accurate. This measure is not valence independent.

Imaging preprocessing

Imaging data were first converted into BIDS via in-house scripts. All data were then preprocessed using FMRIPREP version stable (Esteban et al., 2019; Esteban, Oscar et al., 2023; RRID:SCR_016216), a Nipype (Gorgolewski et al., 2011; Gorgolewski, et al., 2017; RRID:SCR_002502) based tool. Each T1w (T1-weighted) volume was corrected for INU (intensity non-uniformity) using N4BiasFieldCorrection v2.1.0 (Tustison et al., 2010) and skull-stripped using antsBrainExtraction.sh v2.1.0 (using the OASIS template). Brain surfaces were reconstructed using recon-all from FreeSurfer v6.0.1 (Dale et al., 1999; RRID:SCR_001847), and the brain mask estimated previously was refined with a custom variation of the method to reconcile ANTs-derived and FreeSurfer-derived segmentations of the cortical gray-matter of Mindboggle (Klein et al., 2017; RRID:SCR_002438). Spatial normalization to the ICBM 152 Nonlinear Asymmetrical template version 2009c (Fonov et al., 2009; RRID:SCR_008796) was performed through nonlinear registration with the antsRegistration tool of ANTs v2.1.0 (Avants et al., 2008; RRID:SCR_004757), using brain-extracted versions of both T1w volume and template. Brain tissue segmentation of cerebrospinal fluid (CSF), white-matter (WM) and gray-matter (GM) was performed on the brain-extracted T1w using fast (Zhang et al., 2001; FSL v5.0.9, RRID:SCR_002823).

Functional data was slice time corrected using 3dTshift from AFNI v16.2.07 (Cox, 1996; RRID:SCR_005927) and motion corrected using mcflirt (FSL v5.0.9; Jenkinson, 2003). This was followed by co-registration to the corresponding T1w using boundary-based registration (Greve & Fischl, 2009) with six degrees of freedom, using bbregister (FreeSurfer v6.0.1). Motion correcting transformations, BOLD-to-T1w transformation and T1w-to-template (MNI) warp were concatenated and applied in a single step using antsApplyTransforms (ANTs v2.1.0) using Lanczos interpolation.

Physiological noise regressors were extracted applying CompCor (Behzadi et al., 2007). Principal components were estimated for the two CompCor variants: temporal (tCompCor) and anatomical (aCompCor). A mask to exclude signal with cortical origin was obtained by eroding the brain mask, ensuring it only contained subcortical structures. Six tCompCor components were then calculated including only the top 5% variable voxels within that subcortical mask. For aCompCor, six components were calculated within the intersection of the subcortical mask and the union of CSF and WM masks calculated in T1w space, after their projection to the native space of each functional run. Framewise displacement (Power et al., 2014) was calculated for each functional run using the implementation of Nipype.

Many internal operations of FMRIPREP use Nilearn (Abraham et al., 2014; RRID:SCR_001362], principally within the BOLD-processing workflow. For more details of the pipeline see https://fmriprep.readthedocs.io/en/stable/workflows.html.

Univariate analysis

Preprocessed BOLD runs were concatenated for each participant. Next a participant-specific design matrix was created. It included 36 nuisance regressors extracted via FMRIPREP. These nuisance regressors included CSF and white matter regressors and their derivatives as well as 3D motion regressors and their derivatives and spikes or motion outliers. Next, the regressors of interest were added to the design matrix and three unique “single trial” models were fit to the observers’ brains.

Single Trial Observer Rating Models

Regressors were constructed for each rating level for each video stimulus and input into the participant’s design matrix (see Supplementary Fig. 2B). Because participant’s made individual ratings for each video, the number of regressors for each participant varied. Ratings regressors were convolved using the canonical HRF function from the SPM12 toolbox (https://www.fil.ion.ucl.ac.uk/spm/). The nuisance regressors and intercept were added to each design matrix and then an ordinary least squares (OLS) regression was run for each participant. This entire procedure was repeated three times, for the three unique rating types (“ground truth” target self-ratings, observer interference ratings, and empathic accuracy). Variance Inflation Factors (VIF) were calculated for all regressors of interest to perform data quality control. Event trials with a VIF greater than three standard deviations from an individual’s mean were removed as outliers. We tested for intercorrelation across the design matrix and found no issues of multicollinearity within regressors of interest.

Predictive analysis

Model Training

Brain activity for each intensity level within each participant was averaged into a single beta map and then a LASSO-PCR model (lasso number = 120; error type = Mean Squared Error (MSE)) was trained, via LOO-CV, to predict the intensity level (1–5). LASSO-PCR is a machine-learning–based regression technique. Due to its penalization method, LASSO regression simultaneously performs feature selection and model estimation (Tibshirani, 1996). We used the standard lasso hyperparameters built into the CANLab toolbox (https://github.com/canlab) for the development of the neurologic pain signature (NPS; Wager et al., 2013) and the picture-induced negative emotion signature (PINES; Chang et al., 2015), and did not tune these parameters during training. This procedure was performed twice. First, for the “ground truth” ratings and second for the observer inference ratings. The results were two brain-based models of “ground truth” signal intent and socioemotional inference, respectively. Our metric for assessing model accuracy was prediction-outcome correlation (OLS partial r). We used permutation tests to obtain unbiased estimates of accuracy and bootstrap tests (5,000 samples) to determine which brain areas made reliable contributions to prediction.

Model Validation

To validate the models, we applied it to analogous “ground truth” and inference brain activity during the auditory-only and visual-only trails. These validation maps also had five levels of ‘ground truth’ emotion intensity and five levels of inferred emotion intensity. Validation trials were “held out,” meaning that they were not included in training or tested on during training.

There were two validation sets-one for each model–a “ground truth” validation set, and an inference validation set. A prediction-outcome correlation (partial r) between the model and each validation image was calculated for each subject (N = 100). Then we averaged participant’s partial r values and compared them within and across the validation sets. The validity of the model was established, first, by applying it to its own validation set (i.e., “ground truth” model was tested on the “ground truth” validation set) and, next, by testing if the average performance was positively greater than zero in a two-tailed single sample t-test.

To further validate the models, we tested its specificity to its own validation set. We did this by applying the model to the opposite validation set (i.e., “ground truth” model was tested on the inference validation set). Then we tested if the model’s average performance was higher on its own validation set versus the opposite validation set via a two-tailed paired t-test. We expected the “ground truth” model to have higher prediction-outcome correlation on the “ground truth” validation set than on the inference validation set. Likewise, we expected the inference model to have higher prediction-outcome correlation on the inference validation set than the “ground truth” validation set.

Characterization of the model pattern weights

The unthresholded predictive weight map (see Fig. 1) was input into the NeuroSynth Image Decoder (https://neurosynth.org/decode/) to quantitatively compare it to images in the Neurosynth database. This allows us to assess what behaviors and functions are most associated with our patterns of brain activity, across published literature. The top 50 terms that loaded onto the map were pruned down to remove single brain regions and redundant concepts (like theory of mind and ‘tom’). Then a word cloud was constructed of the top 20 terms. The networks and functions in the NeuroSynth database which were most similar to the model are depicted in the word cloud (Figs. 2 & 3), and the words are scaled by strength of similarity.

To verify that the “ground truth” and inference models are not reducible to general processing of affect, we compared our models’ patterns with previously published and publicly available models of affect and emotion. We used cosine similarity, a multivariate method of assessing the similarity between two vectors, to make these comparisons between our models’ pattern weights and eight published models of various components of PINES (Chang et al., 2015), NPS (Wager et al., 2013), the Vicarious Pain Signature (VPS; Krishnan et al., 2016), Empathic Care & Empathic Distress (Ashar et al., 2016), Social Rejection (Woo et al., 2014), Galvanic Skin Response (GSR; Eisenbarth, Chang, & Wager, 2016), and Heart Rate (Heart; Eisenbarth, Chang, & Wager, 2016; see Supplementary Fig. 5). These models are available freely online (https://github.com/canlab). Cosine similarity ranges from − 1 to 1, where values closer to 0 mean the two vectors are orthogonal or perpendicular to each other. When the value is closer to 1, it means the angle is smaller and images/brain patterns are more similar. When the value is closer to -1, it means the images are more opposite.

Comparison of the “ground truth” and inference maps

We tested the separability of the “ground truth” and inference subject-level intensity beta maps with a linear support vector machine (SVM; C = 1, optimizer = Adam (adaptive moment estimation), LOO-CV). This classification was performed across each level of intensity, and a receiver operating characteristic (ROC) curve was calculated across all the CV folds (within subject; Fig. 4). In this analysis, a classification accuracy significantly greater than chance indicates that the activity patterns are linearly separable, and therefore, represent unique neural processes (see Reddan, Wager, & Schiller, 2018). To better understand the brain regions that distinguish between “ground truth” and inference, the predictive voxel weights for the classifier trained at the highest level of intensity (level 5) was bootstrapped (5,000 samples) and thresholded (FDR q < 0.05) so that the brain regions where “ground truth” and inference maximally diverge could be compared (Fig. 4B). This thresholding was repeated also for the classifier trained at the lowest level of intensity (level 1) as a sanity check (Supplementary Fig. 7).

Model Alignment and Empathic Accuracy

To test how our models relate to an observer’s empathic accuracy, we took the dot-product between each model and individual subject’s (N = 100) maps of empathic accuracy. Then we correlated the resulting dot-products from low empathic accuracy trials and high empathic accuracy trials across all participants. Finally, the alignment analysis was validated on empathic accuracy maps from the visual-only and auditory-only trials, which are independent from the “ground truth” and inference model training data, to ensure that results were not biased. Two-tailed z-tests of the correlation difference were used to compare alignment scores because these scores are correlations.

During first-level modeling, the empathic accuracy regressors may overlap in time with the “ground truth” and inference regressors because empathic accuracy is partially derived from target and observer ratings. Therefore, first we confirmed the separability of the “ground truth” and inference subject-level beta maps from the empathic accuracy subject-level beta maps before testing how the alignment of the “ground truth” and inference models are related to empathic accuracy. The separability of the “ground truth” and inference beta maps from the empathic accuracy beta maps at two levels of empathic accuracy (low accuracy and high accuracy) were confirmed using a linear SVM (C = 1, optimizer = andre, LOO-CV). This classification was performed across each level of intensity vs the low and high empathic accuracy maps. The “ground truth” and inference maps were separable from the low and high empathic accuracy maps (accuracy range = 57% − 67%; SE = 1.5–1.6%; p < 0.0001).

COI

The all of the authors of this manuscript certify that they have no affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers’ bureaus; membership, employment, consultancies, stock ownership or other equity interest; and expert testimony or patent-licensing arrangements) or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.

INCLUSION AND DIVERSITY

We worked to ensure gender balance in the recruitment of human subjects. We worked to ensure ethnic or other types of diversity in the recruitment of human subjects. We worked to ensure that the study questionnaires were prepared in an inclusive way. One or more of the authors of this paper self-identifies as an underrepresented ethnic minority in their field of research or within their geographical location. One or more of the authors of this paper self-identifies as a gender minority in their field of research.

Funding

This work was supported by NIH grant #R01MH112560

Author Contributions

J.Z. and T.D.W. designed research; D.C.O, S.M., I.K. performed research; M.R. analyzed data; M.R. and J.Z. wrote the paper.

Abraham, A., Pedregosa, F., Eickenberg, M., Gervais, P., Mueller, A., Kossaifi, J., Gramfort, A., Thirion, B., & Varoquaux, G. (2014). Machine learning for neuroimaging with scikit-learn. Frontiers in Neuroinformatics, 8. https://doi.org/10.3389/fninf.2014.00014
Adler, N. E., Epel, E. S., Castellazzo, G., & Ickovics, J. R. (2000). Relationship of subjective and objective social status with psychological and physiological functioning: Preliminary data in healthy, White women. Health Psychology, 19(6), 586–592. https://doi.org/10.1037/0278-6133.19.6.586
Aminoff, E. M., Kveraga, K., & Bar, M. (2013). The role of the parahippocampal cortex in cognition. Trends in Cognitive Sciences, 17(8), 379–390. https://doi.org/10.1016/j.tics.2013.06.009
Ashar, Y. K., Andrews-Hanna, J. R., Dimidjian, S., & Wager, T. D. (2017). Empathic Care and Distress: Predictive Brain Markers and Dissociable Brain Systems. Neuron, 94(6), 1263-1273.e4. https://doi.org/10.1016/j.neuron.2017.05.014
Avants, B., Epstein, C., Grossman, M., & Gee, J. (2008). Symmetric diffeomorphic image registration with cross-correlation: Evaluating automated labeling of elderly and neurodegenerative brain. Medical Image Analysis, 12(1), 26–41. https://doi.org/10.1016/j.media.2007.06.004
Baez, S., Marengo, J., Perez, A., Huepe, D., Font, F. G., Rial, V., Gonzalez‐Gadea, M. L., Manes, F., & Ibanez, A. (2015). Theory of mind and its relationship with executive functions and emotion recognition in borderline personality disorder. Journal of Neuropsychology, 9(2), 203–218. https://doi.org/10.1111/jnp.12046
Barrett, L. F. (2017). The theory of constructed emotion: An active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, 12(1), 1–23. https://doi.org/10.1093/scan/nsw154
Behzadi, Y., Restom, K., Liau, J., & Liu, T. T. (2007). A component based noise correction method (CompCor) for BOLD and perfusion based fMRI. NeuroImage, 37(1), 90–101. https://doi.org/10.1016/j.neuroimage.2007.04.042
Blakemore, S.-J., Bristow, D., Bird, G., Frith, C., & Ward, J. (2005). Somatosensory activations during the observation of touch and a case of vision–touch synaesthesia. Brain, 128(7), 1571–1583. https://doi.org/10.1093/brain/awh500
Bush, K. A., Gardner, J., Privratsky, A., Chung, M.-H., James, G. A., & Kilts, C. D. (2018). Brain States That Encode Perceived Emotion Are Reproducible but Their Classification Accuracy Is Stimulus-Dependent. Frontiers in Human Neuroscience, 12, 262. https://doi.org/10.3389/fnhum.2018.00262
Button, K. S., Browning, M., Munafò, M. R., & Lewis, G. (2012). Social inference and social anxiety: Evidence of a fear-congruent self-referential learning bias. Journal of Behavior Therapy and Experimental Psychiatry, 43(4), 1082–1087. https://doi.org/10.1016/j.jbtep.2012.05.004
Cacioppo, J. T., & Cacioppo, S. (2014). Older adults reporting social isolation or loneliness show poorer cognitive function 4 years later. Evidence Based Nursing, 17(2), 59–60. https://doi.org/10.1136/eb-2013-101379
Cassidy, S., Ropar, D., Mitchell, P., & Chapman, P. (2014). Can Adults With Autism Spectrum Disorders Infer What Happened to Someone From Their Emotional Response?: Inferences from reactions. Autism Research, 7(1), 112–123. https://doi.org/10.1002/aur.1351
Causton-Theoharis, J., Ashby, C., & Cosier, M. (2009). Islands of Loneliness: Exploring Social Interaction Through the Autobiographies of Individuals With Autism. Intellectual and Developmental Disabilities, 47(2), 84–96. https://doi.org/10.1352/1934-9556-47.2.84
Chang, L. J., Gianaros, P. J., Manuck, S. B., Krishnan, A., & Wager, T. D. (2015). A Sensitive and Specific Neural Signature for Picture-Induced Negative Affect. PLOS Biology, 13(6), e1002180. https://doi.org/10.1371/journal.pbio.1002180
Chang, L. J., Jolly, E., Cheong, J. H., Rapuano, K. M., Greenstein, N., Chen, P.-H. A., & Manning, J. R. (2021). Endogenous variation in ventromedial prefrontal cortex state dynamics during naturalistic viewing reflects affective experience. Science Advances, 7(17), eabf7129. https://doi.org/10.1126/sciadv.abf7129
Cox, R. W. (1996). AFNI: Software for Analysis and Visualization of Functional Magnetic Resonance Neuroimages. Computers and Biomedical Research, 29(3), 162–173. https://doi.org/10.1006/cbmr.1996.0014
Dale, A. M., Fischl, B., & Sereno, M. I. (1999). Cortical Surface-Based Analysis. NeuroImage, 9(2), 179–194. https://doi.org/10.1006/nimg.1998.0395
Dziobek, I., Preißler, S., Grozdanovic, Z., Heuser, I., Heekeren, H. R., & Roepke, S. (2011). Neuronal correlates of altered empathy and social cognition in borderline personality disorder. NeuroImage, 57(2), 539–548. https://doi.org/10.1016/j.neuroimage.2011.05.005
Eickhoff, S. B., Milham, M., & Vanderwal, T. (2020). Towards clinical applications of movie fMRI. NeuroImage, 217, 116860. https://doi.org/10.1016/j.neuroimage.2020.116860
Eisenbarth, H., Chang, L. J., & Wager, T. D. (2016). Multivariate Brain Prediction of Heart Rate and Skin Conductance Responses to Social Threat. The Journal of Neuroscience, 36(47), 11987–11998. https://doi.org/10.1523/JNEUROSCI.3672-15.2016
Esteban, O., Markiewicz, C. J., Blair, R. W., Moodie, C. A., Isik, A. I., Erramuzpe, A., Kent, J. D., Goncalves, M., DuPre, E., Snyder, M., Oya, H., Ghosh, S. S., Wright, J., Durnez, J., Poldrack, R. A., & Gorgolewski, K. J. (2019). fMRIPrep: A robust preprocessing pipeline for functional MRI. Nature Methods, 16(1), 111–116. https://doi.org/10.1038/s41592-018-0235-4
Esteban, Oscar, Markiewicz, Christopher J., Goncalves, Mathias, Provins, Céline, Kent, James D., DuPre, Elizabeth, Salo, Taylor, Ciric, Rastko, Pinsard, Basile, Blair, Ross W., Poldrack, Russell A., & Gorgolewski, Krzysztof J. (2023). fMRIPrep: A robust preprocessing pipeline for functional MRI (23.0.2) [Computer software]. Zenodo. https://doi.org/10.5281/ZENODO.852659
Fakoya, O. A., McCorry, N. K., & Donnelly, M. (2020). Loneliness and social isolation interventions for older adults: A scoping review of reviews. BMC Public Health, 20(1), 129. https://doi.org/10.1186/s12889-020-8251-6
Fonov, V., Evans, A., McKinstry, R., Almli, C., & Collins, D. (2009). Unbiased nonlinear average age-appropriate brain templates from birth to adulthood. NeuroImage, 47, S102. https://doi.org/10.1016/S1053-8119(09)70884-5
Genzer, S., Ong, D. C., Zaki, J., & Perry, A. (2022). Mu rhythm suppression over sensorimotor regions is associated with greater empathic accuracy. Social Cognitive and Affective Neuroscience, 17(9), 788–801. https://doi.org/10.1093/scan/nsac011
Gilbert, K., Hall, K., & Codd, R. T. (2020). Radically Open Dialectical Behavior Therapy: Social Signaling, Transdiagnostic Utility and Current Evidence. Psychology Research and Behavior Management, Volume 13, 19–28. https://doi.org/10.2147/PRBM.S201848
Goodwin, R., Cook, O., & Yung, Y. (2001). Loneliness and life satisfaction among three cultural groups. Personal Relationships, 8(2), 225–230. https://doi.org/10.1111/j.1475-6811.2001.tb00037.x
Gorgolewski, K., Burns, C. D., Madison, C., Clark, D., Halchenko, Y. O., Waskom, M. L., & Ghosh, S. S. (2011). Nipype: A Flexible, Lightweight and Extensible Neuroimaging Data Processing Framework in Python. Frontiers in Neuroinformatics, 5. https://doi.org/10.3389/fninf.2011.00013
Gorgolewski, Krzysztof J., Esteban, Oscar, Ellis, David Gage, Notter, Michael Philipp, Ziegler, Erik, Johnson, Hans, Hamalainen, Carlo, Yvernault, Benjamin, Burns, Christopher, Manhães-Savio, Alexandre, Jarecka, Dorota, Markiewicz, Christopher J., Salo, Taylor, Clark, Daniel, Waskom, Michael, Wong, Jason, Modat, Marc, Dewey, Blake E, Clark, Michael G., … Ghosh, Satrajit. (2017). Nipype: A flexible, lightweight and extensible neuroimaging data processing framework in Python. 0.13.1 (0.13.1) [Computer software]. Zenodo. https://doi.org/10.5281/ZENODO.581704
Greve, D. N., & Fischl, B. (2009). Accurate and robust brain image alignment using boundary-based registration. NeuroImage, 48(1), 63–72. https://doi.org/10.1016/j.neuroimage.2009.06.060
Grimm, P. (2010). Social Desirability Bias. In J. Sheth & N. Malhotra (Eds.), Wiley International Encyclopedia of Marketing (1st ed.). Wiley. https://doi.org/10.1002/9781444316568.wiem02057
Hall, J. A., Andrzejewski, S. A., & Yopchick, J. E. (2009). Psychosocial Correlates of Interpersonal Sensitivity: A Meta-Analysis. Journal of Nonverbal Behavior, 33(3), 149–180. https://doi.org/10.1007/s10919-009-0070-5
Hirsch, C. R., & Mathews, A. (2000). Impaired positive inferential bias in social phobia. Journal of Abnormal Psychology, 109(4), 705–712. https://doi.org/10.1037/0021-843X.109.4.705
Holt-Lunstad, J., Smith, T. B., & Layton, J. B. (2010). Social Relationships and Mortality Risk: A Meta-analytic Review. PLoS Medicine, 7(7), e1000316. https://doi.org/10.1371/journal.pmed.1000316
Hooker, C. I., Verosky, S. C., Germine, L. T., Knight, R. T., & D’Esposito, M. (2010). Neural activity during social signal perception correlates with self-reported empathy. Brain Research, 1308, 100–113. https://doi.org/10.1016/j.brainres.2009.10.006
Irish, M., Halena, S., Kamminga, J., Tu, S., Hornberger, M., & Hodges, J. R. (2015). Scene construction impairments in Alzheimer’s disease – A unique role for the posterior cingulate cortex. Cortex, 73, 10–23. https://doi.org/10.1016/j.cortex.2015.08.004
Izard, C. E. (2007). Basic Emotions, Natural Kinds, Emotion Schemas, and a New Paradigm. Perspectives on Psychological Science, 2(3), 260–280. https://doi.org/10.1111/j.1745-6916.2007.00044.x
Jääskeläinen, I. P., Sams, M., Glerean, E., & Ahveninen, J. (2021). Movies and narratives as naturalistic stimuli in neuroimaging. NeuroImage, 224, 117445. https://doi.org/10.1016/j.neuroimage.2020.117445
Jafari, M., Aflalo, T., Chivukula, S., Kellis, S. S., Salas, M. A., Norman, S. L., Pejsa, K., Liu, C. Y., & Andersen, R. A. (2020). The human primary somatosensory cortex encodes imagined movement in the absence of sensory information. Communications Biology, 3(1), 757. https://doi.org/10.1038/s42003-020-01484-1
Jenkinson, M. (2003). Fast, automated,N-dimensional phase-unwrapping algorithm. Magnetic Resonance in Medicine, 49(1), 193–197. https://doi.org/10.1002/mrm.10354
Jospe, K., Genzer, S., Klein Selle, N., Ong, D., Zaki, J., & Perry, A. (2020). The contribution of linguistic and visual cues to physiological synchrony and empathic accuracy. Cortex, 132, 296–308. https://doi.org/10.1016/j.cortex.2020.09.001
Jospe, K., Genzer, S., Mansano, L., Ong, D., Zaki, J., Soroker, N., & Perry, A. (2022). Impaired empathic accuracy following damage to the left hemisphere. Biological Psychology, 172, 108380. https://doi.org/10.1016/j.biopsycho.2022.108380
Klein, A., Ghosh, S. S., Bao, F. S., Giard, J., Häme, Y., Stavsky, E., Lee, N., Rossa, B., Reuter, M., Chaibub Neto, E., & Keshavan, A. (2017). Mindboggling morphometry of human brains. PLOS Computational Biology, 13(2), e1005350. https://doi.org/10.1371/journal.pcbi.1005350
Kragel, P. A., & LaBar, K. S. (2015). Multivariate neural biomarkers of emotional states are categorically distinct. Social Cognitive and Affective Neuroscience, 10(11), 1437–1448. https://doi.org/10.1093/scan/nsv032
Kragel, P. A., Reddan, M. C., LaBar, K. S., & Wager, T. D. (2019). Emotion schemas are embedded in the human visual system. Science Advances, 5(7), eaaw4358. https://doi.org/10.1126/sciadv.aaw4358
Landeiro, F., Barrows, P., Nuttall Musson, E., Gray, A. M., & Leal, J. (2017). Reducing social isolation and loneliness in older people: A systematic review protocol. BMJ Open, 7(5), e013778. https://doi.org/10.1136/bmjopen-2016-013778
Loos, E., Egli, T., Coynel, D., Fastenrath, M., Freytag, V., Papassotiropoulos, A., de Quervain, D. J.-F., & Milnik, A. (2019). Predicting emotional arousal and emotional memory performance from an identical brain network. NeuroImage, 189, 459–467. https://doi.org/10.1016/j.neuroimage.2019.01.028
Luanaigh, C. O., & Lawlor, B. A. (2008). Loneliness and the health of older people. International Journal of Geriatric Psychiatry, 23(12), 1213–1221. https://doi.org/10.1002/gps.2054
Mauss, I. B., Shallcross, A. J., Troy, A. S., John, O. P., Ferrer, E., Wilhelm, F. H., & Gross, J. J. (2011). Don’t hide your happiness! Positive emotion dissociation, social connectedness, and psychological functioning. Journal of Personality and Social Psychology, 100(4), 738–748. https://doi.org/10.1037/a0022410
Mazurek, M. O. (2014). Loneliness, friendship, and well-being in adults with autism spectrum disorders. Autism, 18(3), 223–232. https://doi.org/10.1177/1362361312474121
Mehu, M., & Scherer, K. R. (2012). A psycho-ethological approach to social signal processing. Cognitive Processing, 13(S2), 397–414. https://doi.org/10.1007/s10339-012-0435-2
Melissa J. Green, Jennifer H. Waldron, Ian Simpson, & Max Coltheart. (2008). Visual processing of social context during mental state perception in schizophrenia. Journal of Psychiatry and Neuroscience, 33(1), 34.
Polimeni, J. R., & Lewis, L. D. (2021). Imaging faster neural dynamics with fast fMRI: A need for updated models of the hemodynamic response. Progress in Neurobiology, 207, 102174. https://doi.org/10.1016/j.pneurobio.2021.102174
Power, J. D., Mitra, A., Laumann, T. O., Snyder, A. Z., Schlaggar, B. L., & Petersen, S. E. (2014). Methods to detect, characterize, and remove motion artifact in resting state fMRI. NeuroImage, 84, 320–341. https://doi.org/10.1016/j.neuroimage.2013.08.048
Pugh, Z. H., Choo, S., Leshin, J. C., Lindquist, K. A., & Nam, C. S. (2022). Emotion depends on context, culture and their interaction: Evidence from effective connectivity. Social Cognitive and Affective Neuroscience, 17(2), 206–217. https://doi.org/10.1093/scan/nsab092
Quandt, L. C., Lee, Y.-S., & Chatterjee, A. (2017). Neural bases of action abstraction. Biological Psychology, 129, 314–323. https://doi.org/10.1016/j.biopsycho.2017.09.015
Read, S., Comas-Herrera, A., & Grundy, E. (2020). Social Isolation and Memory Decline in Later-life. The Journals of Gerontology: Series B, 75(2), 367–376. https://doi.org/10.1093/geronb/gbz152
Reddan, M. C., Wager, T. D., & Schiller, D. (2018). Attenuating Neural Threat Expression with Imagination. Neuron, 100(4), 994-1005.e4. https://doi.org/10.1016/j.neuron.2018.10.047
Ren, Y., Savadlou, A., Park, S., Siska, P., Epp, J. R., & Sargin, D. (2023). The impact of loneliness and social isolation on the development of cognitive decline and Alzheimer’s Disease. Frontiers in Neuroendocrinology, 69, 101061. https://doi.org/10.1016/j.yfrne.2023.101061
Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145–172. https://doi.org/10.1037/0033-295X.110.1.145
Schaefer, M., Cherkasskiy, L., Denke, C., Spies, C., Song, H., Malahy, S., Heinz, A., Ströhle, A., Schäfer, M., Mianroudi, N., & Bargh, J. A. (2020). Empathy-Related Brain Activity in Somatosensory Cortex Protects From Tactile Priming Effects: A Pilot Study. Frontiers in Human Neuroscience, 14, 142. https://doi.org/10.3389/fnhum.2020.00142
Seghier, M. L. (2013). The Angular Gyrus: Multiple Functions and Multiple Subdivisions. The Neuroscientist, 19(1), 43–61. https://doi.org/10.1177/1073858412440596
Skerry, A. E., & Saxe, R. (2014). A Common Neural Code for Perceived and Inferred Emotion. The Journal of Neuroscience, 34(48), 15997–16008. https://doi.org/10.1523/JNEUROSCI.1676-14.2014
Spunt, R. P., & Adolphs, R. (2019). The neuroscience of understanding the emotions of others. Neuroscience Letters, 693, 44–48. https://doi.org/10.1016/j.neulet.2017.06.018
Takano, Y., Aoki, Y., Yahata, N., Kawakubo, Y., Inoue, H., Iwashiro, N., Natsubori, T., Koike, S., Gonoi, W., Sasaki, H., Takao, H., Kasai, K., & Yamasue, H. (2017). Neural basis for inferring false beliefs and social emotions in others among individuals with schizophrenia and those at ultra-high risk for psychosis. Psychiatry Research: Neuroimaging, 259, 34–41. https://doi.org/10.1016/j.pscychresns.2016.11.003
Tibshirani, R. (1996). Regression Shrinkage and Selection Via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1), 267–288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
Tustison, N. J., Avants, B. B., Cook, P. A., Yuanjie Zheng, Egan, A., Yushkevich, P. A., & Gee, J. C. (2010). N4ITK: Improved N3 Bias Correction. IEEE Transactions on Medical Imaging, 29(6), 1310–1320. https://doi.org/10.1109/TMI.2010.2046908
Van Overwalle, F., & Baetens, K. (2009). Understanding others’ actions and goals by mirror and mentalizing systems: A meta-analysis. NeuroImage, 48(3), 564–584. https://doi.org/10.1016/j.neuroimage.2009.06.009
Vucurovic, K., Caillies, S., & Kaladjian, A. (2020). Neural correlates of theory of mind and empathy in schizophrenia: An activation likelihood estimation meta-analysis. Journal of Psychiatric Research, 120, 163–174. https://doi.org/10.1016/j.jpsychires.2019.10.018
Wager, T. D., Atlas, L. Y., Leotti, L. A., & Rilling, J. K. (2011). Predicting Individual Differences in Placebo Analgesia: Contributions of Brain Activity during Anticipation and Pain Experience. The Journal of Neuroscience, 31(2), 439–452. https://doi.org/10.1523/JNEUROSCI.3420-10.2011
Wager, T. D., Atlas, L. Y., Lindquist, M. A., Roy, M., Woo, C.-W., & Kross, E. (2013). An fMRI-Based Neurologic Signature of Physical Pain. New England Journal of Medicine, 368(15), 1388–1397. https://doi.org/10.1056/NEJMoa1204471
Yang, Y. C., Boen, C., Gerken, K., Li, T., Schorpp, K., & Harris, K. M. (2016). Social relationships and physiological determinants of longevity across the human life span. Proceedings of the National Academy of Sciences, 113(3), 578–583. https://doi.org/10.1073/pnas.1511085112
Zaki, J., & Ochsner, K. (2009). The Need for a Cognitive Neuroscience of Naturalistic Social Cognition. Annals of the New York Academy of Sciences, 1167(1), 16–30. https://doi.org/10.1111/j.1749-6632.2009.04601.x
Zaki, J., Weber, J., Bolger, N., & Ochsner, K. (2009). The neural bases of empathic accuracy. Proceedings of the National Academy of Sciences, 106(27), 11382–11387. https://doi.org/10.1073/pnas.0902666106
Zhang, Y., Brady, M., & Smith, S. (2001). Segmentation of brain MR images through a hidden Markov random field model and the expectation-maximization algorithm. IEEE Transactions on Medical Imaging, 20(1), 45–57. https://doi.org/10.1109/42.906424

There is NO Competing Interest.

SupplementaryMaterials.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Neural signatures of emotional inference and experience align during social consensus

Status:

Version 1

Abstract

Figures

Results Summary

Introduction

Results

An observer’s socioemotional inference can be predicted from their brain activity

Neural patterns underlying “ground truth” signal intent and inference are dissociable

Brain regions associated with the divergence from the “ground truth” during inaccurate inference

Discussion

Methods

CONTACT FOR RESOURCE SHARING

EXPERIMENTAL MODEL AND SUBJECT DETAILS

METHOD DETAILS

Experimental procedures

DATA COLLECTION

Ratings acquisition

Neuroimaging acquisition

QUANTIFICATION AND STATISTICAL ANALYSIS

Outlier removal

Preprocessing of the “ground truth” and inference ratings

Derivation of empathic accuracy ratings

Imaging preprocessing

Univariate analysis

Single Trial Observer Rating Models

Predictive analysis

Model Training

Model Validation

Characterization of the model pattern weights

Comparison of the “ground truth” and inference maps

Model Alignment and Empathic Accuracy

COI

INCLUSION AND DIVERSITY

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1