Multi-disciplinary Characterization of Embarrassment: Behavioral and Acoustic Modeling

doi:10.21203/rs.3.rs-5002955/v1

Download PDF

Article

Multi-disciplinary Characterization of Embarrassment: Behavioral and Acoustic Modeling

https://doi.org/10.21203/rs.3.rs-5002955/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Introduction: Embarrassment is a social emotion that shares many characteristics with social anxiety (SA). Most people experience embarrassment in their daily lives, but it is quite unattended in research. We characterized embarrassment in an interdisciplinary approach, introducing a behavioral paradigm and applying machine learning approaches, including acoustic analyses.

Methods:33 participants had to write about an embarrassing experience and had then, without knowing it prior, to read it out loud to the conductor. Embarrassment was then examined with two different approaches: Firstly, from a subjective view, with self-report measures from the participants. Secondly, from an objective, machine-learning approach, where trained models tested the robustness of our embarrassment data set (i.e., prediction accuracy), and then described embarrassment in a dimensional (i.e., dimension: valence, activation, dominance; VAD) and categorical (i.e., comparing embarrassment to other emotional states) way.

Results: The subjective rating of embarrassment was increased after participants read their stories out loud, and participants with higher SA scores experienced higher embarrassment than participants with lower SA scores. The state of embarrassment was predicted at 86.4% at the best of the unweighted average recall rate. While the simple VAD dimensional analyses did not differentiate between the state of embarrassment and the references, the complex emotional category analyses characterized embarrassment as closer to boredom, a neutral state, and less of sadness.

Conclusion: Combining an effective behavioral paradigm and advanced acoustic modeling, we characterized the emotional state of embarrassment, and the identified characteristics could be used as a biomarker to assess SA.

Biological sciences/Psychology

Biological sciences/Psychology/Human behaviour

Embarrassment is a social emotion that most of us experience through daily life, and it occurs when a desired social image of oneself is threatened (Miller & Tangney, 1994; Tangney et al., 1996). It can be seen as a self-conscious and, simultaneously, other-conscious emotion: While one is more self-aware in an embarrassing situation, one is also concerned about other people's judgment (Tangney et al., 1996). Embarrassment is experienced mostly in public with other people. However, it can still occur privately when the audience is imagined and is more likely to occur around strangers than loved ones. Embarrassment is usually accompanied by typical physiological changes, such as blushing (Hofmann et al., 2006; Keltner, 2005; Miller, 2012), or changes in the voice and non-verbal behavior, such as avoiding eye contact or lowering one’s head. Tangney et al. (1996) categorized embarrassment as a “negatively valenced emotion” (pp. 1264), but there are also positive consequences that can arise for the person affected as well as the audience: According to Miller (2012), people often react helpfully in embarrassing situations, and showing embarrassment in the form of blushing, for example, elicits a favorable impression of the affected person. Stocks et al. (2011) additionally differed between personal and empathic embarrassment. Whilst the first is experienced for oneself, the latter is experienced for another person while, for example, observing an embarrassing task. This study, however, focused solely on personal embarrassment.

Core aspects of embarrassment, such as the fear of negative evaluation, fear of being rejected by others, and heightened self-consciousness, are also important aspects of social anxiety (SA) as well as social anxiety disorder (SAD). According to Rozen and Aderka (2023) embarrassment, SA, and SAD were consistently associated with each other in physiological measures, neural activities, and self-reports of emotions for clinical and non-clinical samples. Socially anxious people are generally more easily embarrassed and respond with more intense embarrassment than less socially anxious people (Leary & Hoyle, 2013; Rozen & Aderka, 2023). As with embarrassment, SA occurs mostly in public; if in private, an imagined audience or an imagined reaction from others is necessary. While embarrassment appears sudden, brief, and due to an actual misstep, SA appears gradually and over a longer period of time and can occur without having done anything wrong (Miller, 2009).

Embarrassment is generally neglected in research on basic emotions (Simon-Thomas et al., 2009). Especially when papers report on vocal cues in different emotions, they often examine either the classical basic emotions according to Ekman & Friesen (1971) or other emotions than embarrassment (e.g., Devillers & Vidrascu, 2007; Juslin et al., 2018; Patel et al., 2011; Sauter et al., 2010). Due to the consistent association of embarrassment and SA and the fact that embarrassment is still a neglected emotion, it is important to look more into the emotion itself, its relationship to other emotions, and how it is associated with SA from a basic research and clinical point of view.

Therefore, this paper's main goal was to exploratorily examine embarrassment and capture the emotion from different points of view. On the one hand, embarrassment can be compared categorically to other emotions; it can be described as how it relates to and shares information with them. There are different data sets in different languages, consisting of emotional speech, where the data has already been labelled and tested accordingly (see e.g., Burkhardt et al., 2005). On the other hand, embarrassment itself can be described dimensionally in more depth. Grimm et al. (2007) proposed a three-dimensional emotion space consisting of the axis valence, activation, and dominance (VAD), which was also used in this study to describe embarrassment dimensionally.

Previous studies investigating fundamental emotional research used either classical subjective clinical psychology approaches, relying mainly on the participants' self-reports and ratings of a few individual experts, or more objective measures such as neuroimaging methods and psychophysiological measures (Bastin et al., 2016). If subjective and objective measures are combined in embarrassment studies, they often focus on somatic or neuronal features (Hofmann et al., 2006; Müller-Pinzler et al., 2012) but seldom on voice parameters. The same goes for physiological indicators of SA or embarrassment: Research most often investigated body, hand, and head movements or gaze activity (Keltner et al., 2019). According to Weeks et al. (2012), there are several advantageous characteristics of voice parameters as physiological indicators of, for example, SAD, such as being less biased to subjects’ responses and more objective than through self-questionnaires. So, even though participants do not say the same, one can objectively compare the paralinguistic information of their answers (Burkhardt et al., 2005).

Human speech can be divided into verbal (linguistic) and non-verbal (paralinguistic) sounds. While it is obvious that verbal sounds play an important role in communication, non-verbal aspects, such as paralinguistics, carry a lot of additional information in conversations, such as the emotional and mental state of the person speaking (Kadali & Mittal, 2020). Conversely, changes in the voice may indicate changes in a person's mental and emotional state. This study, therefore, combined subjective measures (see section 2.3.) and objective engineering and machine learning approaches (see section 2.4.) to examine embarrassment.

To describe and examine embarrassment from different points of view, this paper had the following four goals: The first goal was to induce embarrassment in participants and test whether the induction was successful and, if so, how embarrassment was related to SA. We hypothesized that embarrassment would indeed be induced, with the participants being significantly more embarrassed during the embarrassment induction task compared to the pre- and post-induction periods. Verifying that embarrassment was induced was the only goal with a hypothesis. This was the fundament for the other three following exploratory goals, where we used our acoustic data to gain further insights using automatic speech processing techniques. The second goal was to test how well our trained model could predict our sample data in pre-induction, embarrassment, and post-induction and show the robustness of our embarrassment data. The third goal was to adopt a dimensional approach and map embarrassment onto the VAD dimension. The fourth and last goal was comparing embarrassment to other emotions, thus following a categorical approach. For the third and fourth goals, publicly available emotional speech corpora (i.e., acoustic samples with emotional labels) were used to train our models.

2.1. Participants

Undergraduate psychology students from the University of Berne, Switzerland, were recruited for this study. The average age of the sample (N = 33) was 23.73 years (SD = 4.68), with 78.8% female participants (n = 26). The exclusion criteria were a.) any current or past neurological or psychological disorders, b.) regular medication intake, and c.) regular substance abuse. For their participation, they received 1.5 out of 12 mandatory experimental credits. All participants gave written informed consent to participate in the study. The ethics committee of the faculty of human sciences at the University of Bern approved the study protocol (2020-06-00004). The experiment was performed in accordance with the relevant guidelines and regulations according to the Declaration of Helsinki.

2.2. Procedure

See Fig. 1 for an overview of the study procedure. If the participants did not fulfil any exclusion criteria, they advanced to fill out the questionnaires Short Form Social Phobia Scale (SPS-6), Short Form Social Interaction Anxiety Scale (SIAS-6), Community Assessment of Psychic Experiences (CAPE), and demographic data online via Qualtrics not later than 12 hours before the assessment appointment.

The experimenter conductor was instructed to respond neutrally respectively to not give either a positive reaction (e.g., laugh along with the participant) or a negative reaction (e.g., look dismissive) during the procedure and had all instructions written on a script to maximize standardization. This study was embedded in a larger study based on Mota et al. (2017) and represents part 2 of the procedure. Since part 1 is irrelevant to this study, it was not further explained here (see Mota et al., 2017, for the study procedure of part 1). At t0, participants were shown a neutral picture from the picture set of Mota et al. (2017) and were instructed to tell a story about the picture for at least 30 seconds. If they did not meet the 30-second mark, they were asked to talk more about the picture. At t1, they received a short explanation of the characteristics of embarrassment and how it differed from shame (see e.g., Miller & Tangney, 1994; Rozen & Aderka, 2023; Tangney et al., 1996). The assignment then was to write about a situation where they felt embarrassed, which happened no longer than one year ago. The participants were instructed to write only a few sentences about the setting of the embarrassing situation and then to mainly focus on the following aspects of their experience in the situation: emotions (i.e., “What did you feel in this situation?”), cognitions, physiological reactions, their own behavior, and the behavior of the bystanders. Therewith, participants were more likely to be immersed in the whole experience and not only recollect mere cold facts of the situation (Tangney et al., 1996). After being informed orally, they also received all instructions in written form. Participants had a maximum of 4 minutes to come up with a situation and then a maximum of 14 minutes to write about the situation. At t2, they were asked to read the story aloud to the instructor and were instructed to talk for at least 30 seconds. If they did not meet the 30-second mark, they were asked to talk more about the situation. At t3, they had the same task as at t0 with the same instructions. Between each time point (t0 – t3), they had to indicate their level of embarrassment on a visual analog scale (VAS; see section 2.3.3.).

At the end of the study (after t3), participants were asked to guess the study's purpose. They were then informed about the actual purpose of the study. For ethical reasons, they were asked to state their current well-being, and the instructor intervened if a participant stated particularly low well-being.

Before starting the main assessment, participants answered online questionnaires. On the day of the assessment, participants were told to tell a story based on a neutral picture before (t0) and after (t3) embarrassment induction (t2). The first ten seconds of speech in t0, t2, t3 were used as pre-embarrassment induction (pre), embarrassment induction, (emb), and post-embarrassment induction (post) for the acoustic analysis. VAS = visual analog scale.

2.3. Measures

2.3.1. Short Form Social Phobia Scale (SPS-6) and Short Form Social Interaction Anxiety Scale (SIAS-6)

The Social Phobia Scale (SPS; Mattick & Clarke, 1998) and the Social Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998) are two self-report questionnaires to measure different aspects of SA. The SPS measures SA in performance-related situations (e.g., fear of attracting attention while queueing), while SIAS measures SA in interactional situations (e.g., difficulty in talking to other people). For this study, the short form of the SPS (SPS-6; Peters et al., 2012) and the SIAS (SIAS-6, Peters et al., 2012) were used, which were directly translated into German for this study. The two questionnaires are presented together with 6 SPS-6 items and 6 SIAS-6 items rated on a Likert-Scale from 0, “not at all”, to 4, “extremely”, with a total sum score ranging from 0 to 48. According to Peters et al. (2012), the cut-off scores for clinically relevant SA symptoms are SPS-6 ≥ 2 and SIAS-6 ≥ 7.

2.3.2. Visual Analog Scale (VAS)

For the measurement of the strength of embarrassment participants felt before, during, and after the intervention, a paper-pencil visual analog scale (VAS) was used at each time point (i.e., after t0, t1, and t2). Participants had to assess on a 10cm long line how embarrassed they felt at the moment from 0, “not embarrassed at all”, on the left end to 10, “extremely embarrassed”, on the right end. The measured length from the zero point to the participants’ mark on each line indicated the strength of embarrassment for each time point (cf. Delgado et al., 2018). The interrater variability of two independent raters who measured the distance between all lines was r(97) = 0.998.

2.3.3. Recording procedure and device

Each participant's assessment was audio-recorded from t0 to t3. The recording device was always placed on the table in front of the participant with approx. 1 meter from the edge of the table opposite the participant. The recording device SONY ICD-UX570 was used. The sample rate was at 44.1 kHz with a bit depth of 16-bit. The audio data was stored in “Waveform Audio File Format” (.wav).

2.4. Statistical analysis – behavioral data

The statistical analysis of the behavioral data was performed with R Studio (Version: 2024.04.2 + 764; Posit team, 2024). Descriptive statistics were reported, and to examine differences and relationships, non-parametric tests were used due to a lack of normal distribution in the behavioral data (see section 3.1.).

2.5. Acoustic Data Analysis

The acoustic data from each participant’s first 10 seconds at the beginning of speech in t0, t2, and t3 phases was used for pre-embarrassment induction (pre), embarrassment induction (emb), and post-embarrassment induction (post), respectively.

2.5.1. Auxiliary emotional corpora

To train regression and classification models for the conceptual and dimensional description of embarrassment, two well-known emotional datasets in German were selected: the Berlin Emotional Speech Database (EMO-DB) (Burkhardt et al., 2005) and the “Vera am Mittag” (VAM) dataset (Grimm et al., 2008).

The VAM corpora consist of 947 emotional German speech samples collected from 47 speakers (36 female). Speech segments were selected from 12 broadcasts of the TV talk show “Vera am Mittag” (in Engl.: “Vera at noon”). The weighted average values with evaluator weighted estimator (EWE) techniques of VAD emotional dimensions were used to train our regression models for the dimensional VAD representation of embarrassment (Grimm et al., 2007). Each speech sample in the database has EWE smoothed VAD dimensional labels in the range of [-1,+1]: valence (negative: -1 & positive: 1), activation (calm: -1 & excited: 1), and dominance (weak: -1 & strong: 1) (Kehrein, 2003).

EMO-DB covers 7 emotions (i.e., anger, joy, neutral, sadness, disgust, fear, and boredom). The corpus consists of 10 professional actors (5 female) speaking out 10 predefined phonetically balanced and emotionally neutral sentences. We utilized a subset of 493 utterances, which achieved naturalness and recognizability rates of 60% and 80%, respectively, as obtained during a perception test involving 20 subjects. This subset was used to train our emotion categories-based classification models.

2.5.2. Acoustic feature representations

Considering a comparable sparse amount of training samples for acoustic modeling, we decided to use knowledge-based handcrafted features and pre-trained data-driven feature representations. Also, the handcrafter features are also interpretable. Thus, it can help in gaining insight into the acoustic variations related to speech production in the present study’s context.

For the knowledge-based handcrafted feature representations (FRs), we use the (a) ComPaRE 2016 feature set provided by the openSMILE extraction tool (see Eyben et al. (2010) for more information), which has been studied for several paralinguistic speech processing tasks (Schuller et al., 2013). The ComPaRE set contains 6373 static turn-level features resulting from the computation of functionals (statistics) over low-level descriptor contours.

Considering top performance positions on challenge leaderboards and state-of-the-art neural embeddings for the Speech Emotion Recognition (SER) task from the Speech processing Universal PERformance Benchmark (SUPERB) challenge (Yang et al., 2021), we employed (b) Wav2Vec2 (WV2 EM) (Wagner et al., 2023), fine-tuned for dimensional SER on the MSP-Podcast (Lotfian & Busso, 2019) as emotion data-driven feature representation, and (c) WAVLM (large) (Chen et al., 2022) as general data-driven feature representation.

2.5.3. Machine learning experimental setup

To address our second goal (i.e., test how well our classification model could predict speaker state and distinguish between non-embarrassment and endorsement data samples), we used a speaker-independent experimental protocol, more precisely leave-one-speaker-out (LOSO) protocol. This protocol simulates realistic conditions and maintains a good balance between training and testing subsets. In the LOSO protocol, we used speech samples from 32 speakers to train predictive models, and speech samples from the one remaining speaker were used for testing. This procedure was repeated 33 times, and predictions were aggregated into one file to estimate recognition rates. To measure recognition performance, we used unweighted average recall (UAR). During the experimental study, we used both three-class and two-class experimental settings. The three-class setting included pre-embarrassment, embarrassment, and post-embarrassment labels. In the two-class setting, we provided a selection of different combinations or grouped pre- and post-embarrassment classes into one class (i.e., non-embarrassment). Random Forest (RF) and Support Vector Machines (SVM) classifiers were used to train predictive models that take ComPaRE FR as input. The results obtained from the speaker-independent evaluation study for the second goal is presented in Section 3.2.

For the third goal (i.e., adopting a dimensional approach and mapping embarrassment onto the VAD dimension), we employed a cross-corpora approach, where first three independent Random Forest-based regressors that take as input ComPaRE FR and predict valence, arousal, and dominance values, respectively, were trained on the emotional speech samples of the VAM dataset. As mentioned earlier in Section 2.5.1, the prediction values for each of the VAD emotional dimensions are lying in the range [-1,1]. After training the regressors on the VAM database, we used the models to predict VAD labels and estimate a possible location of the embarrassment state in the circumplex model of affect introduced by Russell (1980). For this purpose, we used pre- and post-embarrassment samples as a reference and modeled possible shifts of numerical VAD labels in the context of these reference samples. The results of this experimental study are presented in Section 3.3.

For the fourth goal (i.e., comparing embarrassment to other emotions, following a categorical approach), as there are no categorical emotion labels associated with the data collected in our study, we again employed a cross-corpora approach. In this approach, a categorical emotion classifier was first built on EMO-DB corpus (see Section 2.5.1) and then the classifier outputs for our speech data were analyzed. Considering the different recording conditions and the phonetic differences between High German (in EMO-DB corpus) and Swiss German (in our data), for robust analysis, we took a multiple classifier/expert approach. As illustrated in Fig. 2, we trained three different categorical emotion classifier systems based on (a) handcrafted FR, (b) data-driven feature representation tuned for emotional analysis task (emotion data-driven FR), and (c) general data-driven FR on EMO-DB corpus. Each of the emotion classification systems were trained to predict class conditional probabilities of seven emotion classes (including neutral class). For our analysis, we passed each of our speech data samples through those three trained classifiers and took the average of the emotion class conditional probabilities estimated by the three classifiers. The resulting average emotion class conditional probabilities were then analyzed. As in the previous cases, we used pre- and post-embarrassment samples as reference samples and treated them as separate classes. The results of this experimental study are presented in Section 3.4.

3.1. SPS-6 & SIAS-6 results and embarrassment induction

Table 1 SPS-6 & SIAS-6 descriptive statistics

Table 1 shows the results from the questionnaires SPS-6 and SIAS-6. 16 Participants were above the SIAS cut-off (i.e., sum SIAS ≥ 7), 17 participants were above the SPS cut-off (i.e., sum SPS ≥ 2), and 9 participants were above both cut-offs (i.e., sum SIAS ≥ 7 and sum SPS ≥ 2). Therefore, according to the cut-offs, 27.3% (9 / 33) showed clinically relevant SA symptoms in both questionnaires.

The first goal of the study was to verify the embarrassment induction. The Shapiro-tests for the differences of the 3 VAS scores (VAS 1 – VAS 2, VAS 1 – VAS 3, VAS 2 – VAS 3) (see Fig. 1) between t0 and t3 showed a significant result for the difference VAS 2 – VAS 3 (p = .023). Therefore, all further calculations were made with non-parametric tests. Multiple pairwise Wilcoxon tests showed a significant difference between the VAS 2 (M = 2.52, SD = 2.09) and VAS 3 (M = 4.40, SD = 2.38) score (V = 551, p < .001). This means that the participants got significantly more embarrassed after reading the story aloud to the conductor than after only writing the story. This can also be observed visually in Fig. 3 for 31 out of 33 participants. The other differences were not significant (VAS1 – VAS2, p = .13, VAS 1 – VAS 3, p = .20).

Spearman correlations were calculated between the SIAS-6 and SPS-6 total scores, the total sum of both questionnaires and the VAS 2 – VAS 3 difference score. There was a significant negative correlation between the difference score and the total sum score of the SPS-6 and SIAS-6 (r_s = – 0.36, p = .037). This means that the higher SA is (here defined by the overall sum of both questionnaires), the more embarrassed a person became after reading the story out loud.

3.2. Prediction of embarrassment based on acoustic features

The second goal of the study was to test the possible prediction performance of machine learning models applied to predict embarrassment. During our preliminary analysis of embarrassment prediction, we used three-class and two-class configurations for classification experimental setups. In the two-class settings, in addition to the non-embarrassment class concept (see section 2.5.3), we used a pre- vs. post-embarrassment configuration. Additionally, considering the scarcity of collected embarrassment data and the need for better interpretability, we used only knowledge-based acoustic features - the ComPaRE (see section 2.5.2.) feature set.

Table 2

*Prediction performance UAR of RF and SVM for various combinations of pre-, embarrassment- and post-sets*
	pre vs. emb	post vs.emb	pre vs. post	(pre + post) vs. emb	pre vs. emb vs. post
RF	0.848	0.833	0.621	0.803	0.636
SVM	0.864	0.818	0.591	0.818	0.596
Note. UAR = Unweighted Average Recall. RF = Random Forest. SVM = Support Vector Machine. emb = embarrassment.

The bold print indicates trends for a higher prediction accuracy for RF and SVM.

Table 2 shows the prediction performance (i.e., the probability of predicting the correct set) of RF and SVM. The best performance was found for predicting the pre-set vs. the embarrassment-set (pre vs. emb), where the right prediction for the respective set was made in 84.8% of the cases for RF and 86.4% for the SVM (since it was a two-class configuration, a performance by chance would be 0.5. The results here indicated a very good prediction performance). The second- and third-best performance was found for the prediction of the post vs. embarrassment (post vs. emb) and for the pre- and post-set combined vs. the embarrassment-set ((pre + post) vs. emb). The pre- vs. post-comparison was slightly above chance and could not be discriminated satisfactorily. In the case of the three-class configuration (pre vs. emb. vs. post), the main misclassification confusion was observed between the pre- and post-embarrassment classes: The pre- and post-state showed a tendency to have similar characteristics since their predictive performance was slightly above chance (see section 3.4. for quantitative proof of similarity).

Supplementary Table 2 shows the top-ranked features for each combination of prediction from Table 2. Feature ranking was conducted based on RF feature importance rates. The glossary in Supplementary Table 1 can be consulted for more information on the features. A more detailed description of acoustic features and their corresponding mathematical models can also be found in the paper of Eyben (2016). Since the best prediction performances were found for pre vs. embarrassment and (pre + post) vs. embarrassment, only their top-ranking features were examined in more detail. For selected set configurations (pre vs. embarrassment and (pre + post) vs. embarrassment), mainly vocal tract-related dynamic features are presented in the top-ranking list (i.e., the most discriminative features).

3.3. Dimensional modeling of embarrassment using the dimensions valence, arousal, and dominance (VAD) from VAM

In order to find the possible location of the embarrassment state in the circumplex model of affect introduced by Russell (1980), we decided to process the embarrassment data using pre-trained VAD regressors. Figure 4a – 4c shows a density map for the predicted valence, arousal, and dominance levels for pre vs. embarrassment and post vs. embarrassment states for all 33 participants (represented as dots). Considering the sparsity of the collected data, we used knowledge-based feature representations (i.e., ComParE) for acoustic modeling in our regression models.

Predictive models were trained on the VAM database speech samples. The regressors trained on VAD emotional dimensionalities were used to predict VAD levels for pre-, post-, and embarrassment samples. To evaluate changes in VAD levels for embarrassment phenomena, we mapped the obtained predictions in 2D plots presented in Fig. 4a – 4c.

Arousal (Fig. 4b) & dominance (Fig. 4c) seem to indicate embarrassment phenomena (more points are on the embarrassment side from the diagonal), while valence (Fig. 4a) does not show any predictive value (dots are quite evenly distributed around the diagonal) and should not be used without further linguistic post-hoc tests.

Nevertheless, none of the VAD dimensions could differentiate enough between pre- vs. embarrassment and between post- vs. embarrassment states. The obtained results showed that embarrassment phenomena could be characterized by comparably complex changes in VAD levels compared to pre vs. embarrassment and post vs. embarrassment states.

Supplementary Table 3, the mutual information (MI) between the regression task for the VAD emotional dimensions on VAM samples and the embarrassment detection task can be found. Selected acoustic features have high MI rates for both tasks (embarrassment detection and VAM-based emotion modeling). As in our previous feature ranking analysis based on RF feature importance, vocal tract-related modulation features are influential in the selected features with high shared MI. Additionally, Supplementary Table 4 shows MI between EMO-DB and embarrassment.

3.4. Categorical modeling of embarrassment using the 7 emotions from EMO-DB

For the classification task, the database EMO-DB was used to train the classifiers for acoustic emotion categories. The database contains emotional speech data from 5 female and 5 male actors speaking in German. They portrayed the following 7 emotions: happiness, sadness, disgust, fear, boredom, anger, and neutral state, to which our embarrassment data set was compared.

Figure 5a shows the probability density functions (PDF) for posterior probability of all EMO-BD emotions over the 3 audio time points (pre, embarrassment, post). In other words, it shows the probability of how much each emotion is represented in our embarrassment sample for each time point. There is a skewed distribution for each time point and a clear distribution change in posterior probabilities of the emotions boredom, sadness, and neutral state over the time points. The most indicative changes in these 3 emotions are plotted in Fig. 5b. The histogram plots show average posteriors for sadness, boredom, and neutral state.

For quantitative proof of the significance of emotional posterior modeling, we used a t-test to evaluate the average posteriors of each emotional state. The above-mentioned indicative changes were also observed in Table 3 with the mean posterior values of each emotion: The obtained p-values for sadness, boredom, and neutral state posteriors are highly significant for both pre vs. embarrassment comparisons and the post vs. embarrassment comparisons. At the same time, the p-values for pre vs. post comparisons for all emotional states are not significant. With that, it could be shown that pre- and post-states show indeed similar characteristics, which was already assumed in Table 2.

In conclusion, being embarrassed seems to shift the voice to less sadness but more boredom and neutral state-related characteristics. But this shift is only temporary and disappears after a while (at post).

Table 3

*EMODB’s emotional representations in mean posterior values over all subjects (N = 33) and the p-value for the t-test for different pairs of mean*
emotion	mean			p-value
	pre	emb.	post	pre vs. emb.	post vs. emb.	pre vs. post
happiness	0.06	0.06	0.06	.503	.652	.818
sadness	0.34	0.25	0.33	< .001	.001	.506
disgust	0.15	0.15	0.15	.699	.397	.697
fear	0.08	0.09	0.09	.079	.146	.685
boredom	0.24	0.28	0.25	< .001	< .001	.402
anger	0.05	0.05	0.05	.574	.582	.317
neutral	0.08	0.12	0.08	< .001	< .001	.200
Note. p-values were retrieved from t-tests.

Supplementary Tables 3 and 4 shows the MI for categorical emotion classification on EMO-DB and the dimensional modeling. We used the complete emotion set presented in the EMO-DB database and binary classification mapping configurations: high/low and arousal/valence. Results in Supplementary Tables 3 and 4 show that the embarrassment phenomenon could cause changes in both emotional dimensions, such as arousal and valence. Hence, the highest shared MI in the emotion classification task with the 7 emotional classes experimental setup can be observed.

This study aimed to systematically examine embarrassment categorically and dimensionally with an interdisciplinary self-report and machine-learning approach.

In the first step, the induction was tested with subjective self-reported measures (i.e., SPS-6, SIAS-6, VAS), and it was shown that participants got significantly more embarrassed after reading the story aloud than only writing it and that the embarrassment induction was successful. This was particularly the case for people who reported higher SA symptoms before the experiment.

In the second step, the robustness of the embarrassment data was tested based on acoustic features. The robustness of the data set was shown with high prediction performance trends for the comparisons pre vs. embarrassment and pre- and post-set combined vs. the embarrassment-set, with SVM showing slightly better predictions than RF. The pre- vs. post-prediction performance was slightly above chance; one hypothesis is that people return to the initial state so that pre- and post cannot be distinguished from each other.

In the third step, embarrassment was examined dimensionally in the VAD dimensions. It was shown that there was a shift in the dimensions of arousal and dominance, while there was none for valence. In order to obtain an applicable prediction with valence, one would have to adapt linguistic-based techniques since embarrassment is a complex emotion, and valence is dependent on different factors (as is natural language processing in general domain-dependent). A participant could, for example, try to mask their embarrassment by changing the voice so that valence would be perceived as positive, even though the content of the audio data is negative. These results suggest either that embarrassment might not be described with simply VAD dimensions since there were no differences found (i.e., distribution graphs were comparable and comparable number of points at each side of the diagonal), or that the model itself is too simple to capture a complex emotion like embarrassment.

In the fourth step, embarrassment was examined categorically by comparing it to seven other emotional states. It was shown that the state of embarrassment had characteristics of less sadness and more boredom and a neutral state in comparison to the pre- and post-state. On the other hand, the pre- and post-states did not show any significant differences, implying that the participants returned, after feeling embarrassed, to a pre-similar state. It additionally shows that our chosen method of embarrassment induction only leads to a temporary heightened embarrassment state and might, therefore, be safe to use with subclinical or clinical samples. This generally suggests that embarrassment is indeed a complex emotion since we observed a shift in multiple emotions (see Fig. 5a). There might also be cultural differences that might explain why there is a higher posterior probability of sadness at pre (one would rather expect a neutral state) or due to the emotions being acted and therefore over-exaggerated.

The behavioral data has shown that reading the story out loud induced embarrassment (in comparison to writing the embarrassing story down). The advantage of this individualized embarrassment induction compared to other induction methods, like singing or holding a presentation, is that writing a personal story and reading it out loud makes sure that it is embarrassing for each person while singing or holding a presentation might even be pleasant for certain people. Indeed, 31 out of 33 participants reported increased embarrassment after speaking about their embarrassing experience. In addition, the machine learning models found shifts for embarrassment in acoustic features, arousal and dominance on the VAD dimensionality, and emotional characteristics of sadness, boredom and neutral state. One explanation is that the articulation changes when people get embarrassed. They might try to become more in control of their voice and to articulate more precisely, hence the change in dominance and the heightened boredom and neutral state (which could also explain the decrease in sadness). Another explanation could be that participants emotionally distanced themselves from the situation due to embarrassment and, therefore, the characteristics of the voice changes. It would have been interesting to have an additional VAS at the end of t3 to test whether the subjective feeling of embarrassment would coincide with the return of the voice to the pre-similar state.

In general, embarrassment can be predicted with high accuracy from the voice only, even though we had a small sample and fixed durations of voice samples (i.e., 10 seconds, respectively, the same amount of information for each participant), which shows that our fundamental study set-up is valid. It also shows the overall good robustness of the data set, that the classification models trained on EMO-DB can be used to understand the embarrassment phenomenon, and that the t-tests provide an additional source of information for the classification models. Another advantage of our setting is the high degree of naturalness, respectively, the ecological validity since emotion-related studies often work with acted respectively simulated emotions (Drahota et al., 2008; El Ayadi et al., 2011).

Our study shows the following limitations: Firstly, no control respectively comparable group existed. Due to the within-study design, every participant is a control for themselves for intra-individual differences, but there is no comparison for inter-individual differences for time-effects. One could, for example, add another group that writes about a neutral, instead of an embarrassing, story. Secondly, even though we were able to find robust effects, the number of participants (N = 33) in the study was rather small. Thirdly, the majority of the participants were female (n = 26). Therefore, we were not able to control for gender differences. Fourthly, we did not control for a linguistic bias. This was already acknowledged in the results part, where valence was not further considered. Fifthly, as no data set exists which could be used to train our models in Swiss German, they were trained in High German. Lastly, carry-over effects due to part 1 of the study cannot be ruled out. It could be, for example, that participants were more sad or bored due to part 1.

By examining paralinguistic voice parameters associated with embarrassment, this paper contributes to multiple research gaps at once. Due to the especially big gap in voice analyses for embarrassment, this paper focused mostly on the fundamental description and classification of embarrassment with voice parameters rather than the association between SA and embarrassment. A future paper will address this association between a healthy sample (control group) and a SAD sample. The SAD sample would be particularly interesting to investigate since we found a correlation between SA and the height of embarrassment, assuming at least more robust and bigger effects. It would also be interesting to compare acoustic indicators with additional (neuro)physiological indicators.

In conclusion, we were able to describe the characteristic way of speaking for people in a state of embarrassment. With a multi-disciplinary approach, we have established an effective behavioral paradigm to induce embarrassment and characterize participants’ response to the embarrassment induction in acoustic characteristics. Those identified characteristics could be used for the assessment of SA and SAD.

Acknowledgments

We thank Lara Dinner for helping us with the data acquisition.

Competing Interests:

none.

Funding:

UPD university-based fund.

Author’s Contributions

The conceptualization of this project was done by D.S. and Y.M.

The data acquisition for the sample was done by D.S., Y.M., and M.S.

The calculations of the behavioral data were done by D.S. with the support of Y.M.

The calculations of the machine learning data were done by B.V. with the support of M.MD.

The manuscript was written by D.S. with the support of Y.M. and B.V.

All authors read and approved the final manuscript.

Data Availability Statement

The behavioral and machine learning data used to support the findings of this study are available upon request. Contact D.S. for the behavioral data and B.V. for the machine learning data. However, voice data is available only after the approval of the local ethics committee due to the privacy protection of the Human Research Act in Switzerland.

Ethics Declarations

The ethics committee of the faculty of human sciences at the University of Bern approved the study protocol (2020-06-00004). All participants gave written informed consent to participate in the study. We hereby confirm that the experiment was performed in accordance with the relevant guidelines and regulations according to the Declaration of Helsinki.

Bastin, C., Harrison, B. J., Davey, C. G., Moll, J., & Whittle, S. (2016). Feelings of shame, embarrassment and guilt and their neural correlates: A systematic review. Neuroscience & Biobehavioral Reviews, 71, 455–471. https://doi.org/10.1016/j.neubiorev.2016.09.019
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., & Weiss, B. (2005). A database of German emotional speech. Interspeech 2005, 1517–1520. https://doi.org/10.21437/Interspeech.2005-446
Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., & Wei, F. (2022). WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. IEEE Journal of Selected Topics in Signal Processing, 16(6), 1505–1518. https://doi.org/10.1109/JSTSP.2022.3188113
Delgado, D. A., Lambert, B. S., Boutris, N., McCulloch, P. C., Robbins, A. B., Moreno, M. R., & Harris, J. D. (2018). Validation of Digital Visual Analog Scale Pain Scoring With a Traditional Paper-based Visual Analog Scale in Adults. JAAOS: Global Research and Reviews, 2(3), e088. https://doi.org/10.5435/JAAOSGlobal-D-17-00088
Devillers, L., & Vidrascu, L. (2007). Real-Life Emotion Recognition in Speech. In C. Müller (Ed.), Speaker Classification II: Selected Projects (pp. 34–42). Springer. https://doi.org/10.1007/978-3-540-74122-0_4
Drahota, A., Costall, A., & Reddy, V. (2008). The vocal communication of different kinds of smile. Speech Communication, 50(4), 278–287. https://doi.org/10.1016/j.specom.2007.10.001
El Ayadi, M., Kamel, M. S., & Karray, F. (2011). Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition, 44(3), 572–587. https://doi.org/10.1016/j.patcog.2010.09.020
Eyben, F. (2016). Real-time Speech and Music Classification by Large Audio Feature Space Extraction. Springer International Publishing. https://doi.org/10.1007/978-3-319-27299-3
Eyben, F., Wöllmer, M., & Schuller, B. (2010). Opensmile: The munich versatile and fast open-source audio feature extractor. Proceedings of the 18th ACM International Conference on Multimedia, 1459–1462. https://doi.org/10.1145/1873951.1874246
Grimm, M., Kroschel, K., Mower, E., & Narayanan, S. (2007). Primitives-based evaluation and estimation of emotions in speech. Speech Communication, 49(10), 787–800. https://doi.org/10.1016/j.specom.2007.01.010
Grimm, M., Kroschel, K., & Narayanan, S. (2008). The Vera am Mittag German audio-visual emotional speech database. 2008 IEEE International Conference on Multimedia and Expo, 865–868. https://doi.org/10.1109/ICME.2008.4607572
Hofmann, S. G., Moscovitch, D. A., & Kim, H.-J. (2006). Autonomic correlates of social anxiety and embarrassment in shy and non-shy individuals. International Journal of Psychophysiology, 61(2), 134–142. https://doi.org/10.1016/j.ijpsycho.2005.09.003
Juslin, P. N. (2018). The Mirror to Our Soul? Comparisons of Spontaneous and Posed Vocal Expression of Emotion. J Nonverbal Behav, 40.
Kadali, D. B., & Mittal, V. K. (2020). Studies on Paralinguistic Sounds, Emotional Speech and Expressive Voices. Workshop on Speech, Music and Mind (SMM 2020), 11–15. https://doi.org/10.21437/SMM.2020-3
Kehrein, R. (2003). Die Prosodie authentischer Emotionen. Sprache · Stimme · Gehör, 27(02), 55–61. https://doi.org/10.1055/s-2003-40251
Keltner, D. (2005). Signs of Appeasement: Evidence for the Distinct Displays of Embarrassment, Amusement, and Shame. In P. Ekman & E. L. Rosenberg (Eds.), What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS) (p. 0). Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195179644.003.0007
Keltner, D., Sauter, D., Tracy, J., & Cowen, A. (2019). Emotional Expression: Advances in Basic Emotion Theory. Journal of Nonverbal Behavior, 43(2), 133–160. https://doi.org/10.1007/s10919-019-00293-3
Leary, M. R., & Hoyle, R. H. (2013). Handbook of Individual Differences in Social Behavior. Guilford Publications.
Lotfian, R., & Busso, C. (2019). Building Naturalistic Emotionally Balanced Speech Corpus by Retrieving Emotional Speech from Existing Podcast Recordings. IEEE Transactions on Affective Computing, 10(4), 471–483. IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2017.2736999
Mattick, R. P., & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 16.
Miller, R. S. (2009). Social anxiousness, shyness, and embarrassability. In Handbook of individual differences in social behavior (pp. 176–191). The Guilford Press.
Miller, R. S. (2012). The interactive origins and outcomes of embarrassment. In W. R. Crozier & P. J. de Jong (Eds.), The Psychological Significance of the Blush (1st ed., pp. 185–202). Cambridge University Press. https://doi.org/10.1017/CBO9781139012850.013
Miller, R. S., & Tangney, J. P. (1994). Differentiating Embarrassment and Shame. Journal of Social and Clinical Psychology, 13(3). https://www.proquest.com/docview/1292186743/citation/9F6BBC5A7B70447BPQ/1
Mota, N. B., Copelli, M., & Ribeiro, S. (2017). Thought disorder measured as random speech structure classifies negative symptoms and schizophrenia diagnosis 6 months in advance. Npj Schizophrenia, 3(1), 18. https://doi.org/10.1038/s41537-017-0019-3
Müller-Pinzler, L., Paulus, F. M., Stemmler, G., & Krach, S. (2012). Increased autonomic activation in vicarious embarrassment. International Journal of Psychophysiology, 86(1), 74–82. https://doi.org/10.1016/j.ijpsycho.2012.07.183
Patel, S., Scherer, K. R., Björkner, E., & Sundberg, J. (2011). Mapping emotions into acoustic space: The role of voice production. Biological Psychology, 87(1), 93–98. https://doi.org/10.1016/j.biopsycho.2011.02.010
Peters, L., Sunderland, M., Andrews, G., Rapee, R. M., & Mattick, R. P. (2012). Development of a short form Social Interaction Anxiety (SIAS) and Social Phobia Scale (SPS) using nonparametric item response theory: The SIAS-6 and the SPS-6. Psychological Assessment, 24(1), 66–76. https://doi.org/10.1037/a0024544
Posit team. (2024). RStudio: Integrated Development Environment for R. Posit Software, PBC, Boston, MA. (2023.12.1.402) [Computer software]. http://www.posit.co/.
Rozen, N., & Aderka, I. M. (2023). Emotions in social anxiety disorder: A review. Journal of Anxiety Disorders, 95, 102696. https://doi.org/10.1016/j.janxdis.2023.102696
Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178. https://doi.org/10.1037/h0077714
Sauter, D. A., Eisner, F., Calder, A. J., & Scott, S. K. (2010). Perceptual Cues in Nonverbal Vocal Expressions of Emotion. Quarterly Journal of Experimental Psychology, 63(11), 2251–2272. https://doi.org/10.1080/17470211003721642
Schuller, B., Steidl, S., Batliner, A., Vinciarelli, A., Scherer, K., Ringeval, F., Chetouani, M., Weninger, F., Eyben, F., Marchi, E., Mortillaro, M., Salamin, H., Polychroniou, A., Valente, F., & Kim, S. (2013). The INTERSPEECH 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism. 148–152. https://doi.org/10.21437/Interspeech.2013-56
Simon-Thomas, E. R., Keltner, D. J., Sauter, D., Sinicropi-Yao, L., & Abramson, A. (2009). The voice conveys specific emotions: Evidence from vocal burst displays. Emotion, 9(6), 838–846. https://doi.org/10.1037/a0017810
Stocks, E. L., Lishner, D. A., Waits, B. L., & Downum, E. M. (2011). I’m Embarrassed for You: The Effect of Valuing and Perspective Taking on Empathic Embarrassment and Empathic Concern: EMPATHIC EMBARRASSMENT. Journal of Applied Social Psychology, 41(1), 1–26. https://doi.org/10.1111/j.1559-1816.2010.00699.x
Tangney, J. P., Miller, R. S., Flicker, L., & Barlow, D. H. (1996). Are shame, guilt, and embarrassment distinct emotions? Journal of Personality and Social Psychology, 70, 1256–1269. https://doi.org/10.1037/0022-3514.70.6.1256
Vlasenko, B., Vyas, S., & Magimai.-Doss, M. (2024). Comparing data-Driven and Handcrafted Features for Dimensional Emotion Recognition. ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 11841–11845. https://doi.org/10.1109/ICASSP48485.2024.10446134
Wagner, J., Triantafyllopoulos, A., Wierstorf, H., Schmitt, M., Burkhardt, F., Eyben, F., & Schuller, B. W. (2023). Dawn of the Transformer Era in Speech Emotion Recognition: Closing the Valence Gap. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 10745–10759. https://doi.org/10.1109/TPAMI.2023.3263585
Weeks, J. W., Lee, C.-Y., Reilly, A. R., Howell, A. N., France, C., Kowalsky, J. M., & Bush, A. (2012). “The Sound of Fear”: Assessing vocal fundamental frequency as a physiological indicator of social anxiety disorder. Journal of Anxiety Disorders, 26(8), 811–822. https://doi.org/10.1016/j.janxdis.2012.07.005
Yang, S., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., Huang, T.-H., Tseng, W.-C., Lee, K., Liu, D.-R., Huang, Z., Dong, S., Li, S.-W., Watanabe, S., Mohamed, A., & Lee, H. (2021). SUPERB: Speech Processing Universal PERformance Benchmark. Interspeech 2021, 1194–1198. https://doi.org/10.21437/Interspeech.2021-1775

No competing interests reported.

SupplementaryInformation.docx

Download PDF

Reviewers agreed at journal
09 Nov, 2024
Reviews received at journal
23 Sep, 2024
Reviewers agreed at journal
13 Sep, 2024
Reviewers invited by journal
12 Sep, 2024
Editor assigned by journal
10 Sep, 2024
Editor invited by journal
09 Sep, 2024
Submission checks completed at journal
06 Sep, 2024
First submitted to journal
30 Aug, 2024

You are reading this latest preprint version

Multi-disciplinary Characterization of Embarrassment: Behavioral and Acoustic Modeling

Status:

Version 1

Abstract

Figures

1. Introduction

2. Materials & Methods

2.1. Participants

2.2. Procedure

2.3. Measures

2.3.1. Short Form Social Phobia Scale (SPS-6) and Short Form Social Interaction Anxiety Scale (SIAS-6)

2.3.2. Visual Analog Scale (VAS)

2.3.3. Recording procedure and device

2.4. Statistical analysis – behavioral data

2.5. Acoustic Data Analysis

2.5.1. Auxiliary emotional corpora

2.5.2. Acoustic feature representations

2.5.3. Machine learning experimental setup

3. Results

3.1. SPS-6 & SIAS-6 results and embarrassment induction

3.2. Prediction of embarrassment based on acoustic features

3.3. Dimensional modeling of embarrassment using the dimensions valence, arousal, and dominance (VAD) from VAM

3.4. Categorical modeling of embarrassment using the 7 emotions from EMO-DB

4. Discussion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1