The (not-so) valid and reliable linguistic markers of depression and anxiety in symptomatic adults: A randomised cross over trial

doi:10.21203/rs.3.rs-5041714/v1

Psycho-linguistic theory postulates that linguistic features expressed in individuals’ communications may be valid indicators of their mental health. This trial aimed to examine the validity and reliability of linguistic markers of depressive and anxiety symptoms in adults. Using a randomised cross over trial design, 218 adults provided eight different types of text data of varying frequencies and emotional valance including SMS data, social media posts, personal expressive essays, and letters to a friend. Linguistic features were extracted from each task using LIWC22 software and correlated with participants’ self-report symptom scores. Machine learning models were used to determine which linguistic features had the strongest associations with symptoms. There were no linguistic features consistently associated with depressive or anxiety symptoms within tasks or across all tasks. Features found to be associated with depressive symptoms were different for each task and there was only some degree of reliability of these features within the repeated tasks. In all the machine learning models, predicted values were weakly associated with actual values for both depressive and anxiety symptoms. Some of the text tasks were found to have lower levels of engagement and negative impacts on participants’ mood. Overall, these findings indicate that there may be few valid and reliable group-level linguistic markers of depression and anxiety when examining several types of individuals’ text data.

Biological sciences/Psychology

Health sciences/Diseases/Psychiatric disorders/Anxiety

Health sciences/Diseases/Psychiatric disorders/Depression

depression

anxiety

linguistics

LIWC

artificial intelligence

digital phenotype

mental health

psychiatry

psychology

Depression and anxiety are common mental illnesses that negatively affect the health and wellbeing of millions worldwide, eroding individuals’ quality of life and productivity. Identifying depressive and anxiety symptoms among individuals and populations in an automated and wide scale manner may allow for timely intervention at a population level, triaging risk and preventing potential escalation into severe disorders and suicidality. Digital phenotyping is the emerging field of science that mines the digital data generated by individuals in the daily course of their lives to uncover “objective, quantifiable characteristics of illness that can be measured accurately and reproducibly” ¹. There has been a rapid increase in the number of research studies attempting to use this approach to understand various mental health states and the need for intervention among populations and individuals ^2,3. Automated linguistic analysis of individuals’ text data may be an advantageous method of digital phenotyping because individuals generate large volumes of text data daily, and these can be readily collected and analysed at scale using low-cost real time software ^4,5. By segmenting individuals’ text data into linguistic features that can be observed and quantified, linguistic analysis has enabled researchers to rapidly explore the relationships between linguistic expression and mental illness.

1.1. Linguistic markers of depression and anxiety

To date, more than fifty studies have suggested that linguistic features expressed in different forms of individuals’ text data may provide markers of the cognitive, emotional, and social deficits of depression and anxiety. Guided by psycholinguistic theory, it has been hypothesised that mental illness significantly alters one’s linguistic style whereby the words used by individuals may indicate the reduced positive emotion, ruminative self-preoccupation, ⁶, low levels of motivation and pleasure, and difficulty foreseeing positive future events that characterise depressive and anxious states ^7–10. Indeed, a meta-analysis of 21 studies of individuals’ written and spoken communications including personal essays, social media posts and clinician interviews, found that higher depressive symptoms were marked by the increased use of first-person singular pronouns, irrespective of gender or other demographics ¹¹. A second meta-analysis of 26 studies of individuals’ linguistic expression in varied writing tasks found that individuals with depressive symptoms were distinguished from non-depressed individuals by small to moderate differences in first-person singular pronouns use, negative emotion word use and positive emotion word use ¹². Another systematic review of 25 studies on the linguistic features of depression also found that first-person singular pronouns and negative emotion words were significantly associated with depressive symptoms, although the strength and magnitude of the results varied ¹³. However, several studies attempting to identify linguistic markers of depression and anxiety have had conflicting findings ^14–16, likely influenced by differences in the text forms analysed and variations in methodology such as clinical versus community samples. Furthermore, the strength of the associations between linguistic features and symptoms are consistently weak suggesting limited utility for prediction. While collectively the studies suggest that there may be linguistic markers of mental illness, the validity, reliability and overall utility of this approach remains in its infancy.

There is also increasing evidence of a task effect whereby contextual factors (i.e. the purposes and conditions under which text data is created) may influence the strength and direction of the relationships between linguistic features and mental health symptoms. For example, Havigerova ¹⁷ found that linguistic features expressed by individuals in a fictional letter to a friend inviting them on a holiday had greater predictive utility for depression than the linguistic features expressed in fictional formal cover letters, complaints, or apology letters. Similarly, Minori ¹⁸ and Cuteri ¹⁹ found that written personal reflections of a complex picture was the most useful text task for generating valid linguistic markers of depression when compared to written self-reflection tasks about one’s self or one’s friends. In contrast, Kazmierczak et al. ²⁰ found that depressed individuals consistently used the same types of language and word choices regardless of what they were describing, suggesting that some aspects of linguistic expression may be stable. However, these past studies have not accounted for the potential intervention effects of expressive writing tasks on participants’ mood ²¹ or their levels of engagement. This is pertinent to the study of depression, given that depression is characterised by low levels of motivation, biased processing of emotional stimulus, enhanced recall of negative emotional experiences, and difficulty foreseeing positive future events ^7–10. As this has not been controlled for in many prior studies, it remains unclear whether it is individuals’ mental health symptoms driving variations in their linguistic expression, the task itself, or an interaction effect.

With the growth of smartphone and digital-mediated communications, researchers have examined the utility of organic text data such as SMS and social media posts for generating markers of mental illness. When examining SMS text data, Stamatis et al., ²² found that individuals with higher depressive symptoms used significantly fewer words related to anticipation, trust, social processes, and affiliation whereas the inverse use of words was found for high anxiety symptoms. Also in SMS data, Meyerhoff ²³ found that individuals with higher depressive symptoms used significantly more first-person singular pronouns, filler words, sexual words, anger and negative emotion words, but only in their SMS with close contacts. Several studies have revealed potential linguistic markers of depression in individuals’ social media posts ^24–29 with several supervised and unsupervised machine learning models showing promise for detecting depressive symptoms ³⁰. Notably however, Waterloo ³¹ found that different digital communication platforms (e.g. WhatsApp, Facebook, Twitter and Instagram) elicited significant variations in an individual’s linguistic expression. This is consistent with Tlachac ³², who found that that individuals’ SMS messages were more useful for detecting depression than their Twitter posts. Similarly, Liu ³³ also found significant cross-platform differences in linguistic expression, such that a Facebook-derived model of linguistic predictors of depression performed poorly when applied to SMS data from the same individuals. Smirnov ³⁴ also found that the number and significance of linguistics markers in social media messages and personal essays about oneself and their relations varied according to depression status (i.e., clinical depression diagnosis versus high depressive symptomatology) and text type. Based on their results, Smirnov and colleagues also posited that the linguistic markers of depression were most likely influenced by the text topic and the conditions under which the text was written, rather than mental health status alone.

Overall, variations in the types of text data, the number of linguistic features examined, and the analytical approaches used have made it difficult to assess the validity and reliability of linguistic markers of mental illness. Past studies have been largely observational, collecting text data from a single timepoint and source. In studies collecting and combining several types of text data, there has been little attempt to control for order effects, whereby the completion of certain study activities such as mental health surveys or negatively-anchored writing tasks may prime individuals’ to respond in particular ways. Many studies have not accounted for the varied contextual factors such as the individual ¹⁷, the communication modality ³⁵, the audience domain, the task stimuli, or the study conditions, that may influence the emergence of linguistic markers of mental illness. Furthermore, the rapid changes in digital communication patterns among individuals as evidenced by generational shifts to short-form abbreviated messages and image-based content ^36,37 combined with the expansion of internet security measures to protect individuals’ data privacy, has made the automated extraction and mining of digital data by third parties increasingly difficult ³⁸. As digital phenotyping gains popularity and data collection becomes more invasive, it is essential that researchers have a strong scientific and ethical rationale for extracting and analysing of large volumes of personal text data. A clearer understanding of the potential utility of text data for providing meaningful insight into the health and wellbeing of individuals will help to ensure this.

1.2. Objectives of current study

Using a randomised cross over experimental design to control for potential order and task effects, this trial aimed to examine the validity and reliability of linguistic features of symptoms of depression and anxiety in symptomatic adults by comparing several text types previously found to generate significant markers. This study controlled for potential reverse-causation and interaction effects by separating out the text tasks, randomising delivery and collecting multiple samples over time. Furthermore, this study explored whether markers found at the task-level remained significant when all text data was combined into one corpus. Using machine learning models, this study aimed to determine which linguistic features were the most strongly associated with depressive and anxiety symptoms. This study also examined the acceptability of the text tasks and the effects of these on participants’ mood and engagement in this type of digital phenotyping. Based on prior findings, it was hypothesised that several linguistic features would be significantly associated with symptoms of depression and anxiety but there would be variability in the significance of these markers depending on the task type. Consequently, it was hypothesised that markers found to be significant at the task level may not retain their significance when all text data was combined. It was also hypothesised that text tasks with negative sentiment would negatively impact participants’ mood and be less engaging to complete. The findings of this investigation will help to determine the validity and reliability of linguistic markers of depression and anxiety and the acceptability of collecting text data for this approach.

2.1. Design

This study utilised a 9-week randomised crossover design to examine the utility of eight different types of text data of varying frequencies and emotional valance, for generating linguistic markers of depression and anxiety. The study took place between October 2021 and December 2021 and was conducted online. Ethics approval was obtained from the University of New South Wales Human Research Ethics Committee (HC210397) and trial was prospectively registered with the Australian New Zealand Clinical Trials Registry (ACTRN12621001248853). This report concords with the CONSORT for randomised crossover design (ref).

Outlined in Table 1, this trial employed four unique sequences of text tasks to control for potential order effects. Participants were randomly allocated to one of the four sequences upon completion of baseline. Four of the text tasks (SMS, social media posts, emotion diaries, expressive essays) were repeated by participants up to 5 times throughout an intervention period (referred to from here on in as ‘repetitive tasks’, i.e. Tasks ABCD) to ensure the required volume of text was collected and to enable repeated analyses. The repetitive tasks were administered according to a set frequency within the allocated sequence, whereas the once-off tasks (i.e., Tasks VWXY) were completed in the same order for all participants. In total, participants were asked to complete 19 tasks. For more information on the text tasks please see Supplementary Material (Section 1, Table S1.).

Table 1

The text task allocations in each of the four sequences in the randomised cross-over trial design.
Sequence (Task order)	Day 1	Period 1	Day 14	Period 2	Day 28	Period 3	Day 42	Period 4	Day 56
1 (AVBWCXDY)	Nil	A	V	B	W	C	X	D	Y
2 (BVCWDXAY)	Nil	B	V	C	W	D	X	A	Y
3 (CVDWAXBY)	Nil	C	V	D	W	A	X	B	Y
4 (DVAWBXCY)	Nil	D	V	A	W	B	X	C	Y

Table notes: Task descriptions: A = SMS retrieval, B = Social media post, C = Emotion diary, D = Expressive writing, V = Personal description, W = Friends description, X = Reflection of a complex image, Y = Holidaying letter. Each intervention period was 14 days. The total study duration was 63 days including 7 days completion time for final survey.

After each text task was completed for the first time, participants then immediately completed an acceptability questionnaire. Participants were also asked to complete five fortnightly mental health symptom questionnaires on days 1 (baseline), 14, 28, 42, and 56 (study endpoint). The mental health symptom questionnaires were completed immediately after the once-off text tasks to control for any carry-over effects. A ‘wash out’ period (minimum of 3 days) was implemented between the fortnightly mental health symptom questionnaires and the repetitive text tasks to further reduce any carryover effects of the mental health questionnaires. Figure 1 provides the study timeline and schedule of activities for participants allocated to sequence 1.

2.2. Participants and sample size

As the study aimed to examine the relationship between linguistic features and depressive/anxiety symptoms, participants were required to report at least moderate symptoms of depression or anxiety (i.e., total score ≥ 10 on the Patient Health Questionnaire-9 or the Generalized Anxiety Disorder scale at screening), be at least 18 years of age, located in Australia, and have an active mobile phone number and email address. Participants who reported a suicide attempt in the past six months or who reported that they were currently experiencing extreme and unmanageable emotional distress at screening were excluded. The target sample size was set at a minimum of 140 participants, based on detecting moderate correlations (r = 0.3–0.5) between linguistic features and mental health symptoms, and a 40% attrition rate.

2.3. Recruitment and consent

This study utilised a paid social media campaign on Facebook and Instagram. Study adverts also featured in the Black Dog Institute’s e-newsletters, website, and social media channels (Instagram and Facebook). Individuals who clicked on the study adverts were redirected to the study website where they were able to complete the eligibility screener. Eligible participants provided their consent to participate via an online Participant Information Sheet and Consent Form.

2.4. Procedure

Participants received an initial email and SMS notification to complete each task, followed by two reminders. Text tasks and surveys were only open for completion within the allocated two-week intervention period. There was no word limit for any of the tasks (except for Task B so that it replicated a social media platform). Participants were reminded not to worry about spelling, sentence structure or grammar, to just write freely as instructed. Participants could opt out of any task at any time and had the option to report “I feel too distressed to complete this task today”. To reduce attrition, the reimbursement for the mental health surveys was 10AUD at baseline and increased by 5AUD every second fortnight. Participants also received an additional 5AUD for each of the attempted one-off tasks. The maximum reimbursement received was 95AUD.

2.5. Measures

Depressive symptoms

The Patient Health Questionnaire-9 (PHQ-9) is a widely used 9-item, validated self-report questionnaire that assesses the severity of depressive symptoms over the past two weeks. Each item is scored on a scale ranging from 0 (not at all) to 3 (nearly every day). Item scores are summed to obtain a total score, which can range from 0 to 27. Higher scores indicate higher levels of depressive symptoms. Total scores can be used to indicate ‘nil to mild’ symptoms (0–9), ‘moderate’ symptoms (10–14), and ‘moderately severe’ to ‘severe’ symptoms (15+). In the current study, the Cronbach’s alpha for the PHQ-9 was 0.72.

Anxiety symptoms

The Generalized Anxiety Disorder-7 (GAD-7) is a widely used, 7-item validated self-report questionnaire that assesses the presence of generalized anxiety symptoms in the past two weeks. Each item is scored on a scale ranging from 0 (not at all) to 3 (nearly every day). Item scores are summed to obtain a total score, which can range from 0 to 21. Higher scores indicate a higher levels of anxiety symptoms. Total scores can be used to indicate ‘nil to mild’ symptoms (0–9), ‘moderate’ symptoms (10–14), and ‘moderately severe’ to ‘severe’ symptoms (15+). In the current study, the Cronbach’s alpha for the GAD-7 was 0.83.

Demographics

At baseline, participants reported their age, gender identity, educational attainment, relationship status, paid employment, whether they were born in Australia, whether they identified as being First Nations peoples (Aboriginal and/or Torres Strait Islander), gender- and/or sexuality-diverse (Lesbian, Gay, Bisexual, Transgender, Queer, Intersex, and/or other sexuality and gender diversity; LGBTQI+), and the Internet device that they completed the study on. Participants were asked to report what language they usually spoke at home and how often they needed help from someone to complete forms or read written material (e.g., brochures, newspapers), answered on a 5-point Likert scale ranging from never (0) to always (4).

Clinical characteristics

At baseline, participants were asked whether they had ever received a formal diagnosis of depression, anxiety, or other mental illness from a medical practitioner or mental health professional, the duration of mental illness (age when formally diagnosed, age of onset), and whether they were currently taking any antidepressant or other pharmacological medication for their mental illness, as prescribed by a medical practitioner.

Mood monitoring

The 6-item short form of the Multidimensional Mood Questionnaire (MDMQ) ³⁹ was administered immediately before and after the negatively-anchored expressive tasks (i.e. Task C and D, total of 6 times) to explore the impact of these on participants’ mood. Using a 7-point Likert scale, participants were asked to indicate how they felt in this very moment, in relation to six dimensions of mood: fatigue, content, agitation, energy, unwell, and relaxation. The scale has demonstrated the ability to differentiate synchronized mood states and changes within individuals over time ³⁹. No reverse scoring was implemented in this exploratory analysis.

Acceptability

Participants were asked to complete a short 3-item acceptability questionnaire upon first completion of the tasks. The first item “How easy or difficult was this task?” was answered using a 7-point Likert scale from very difficult (1) to very easy (7). The second item “How interesting was this task?” was answered using a Likert scale not at all interesting (1) to very interesting (7). The final item “How willing are you to complete this task again?” was answered using a 7-point Likert scale from not at all willing (1) to very willing (7).

2.6. Data collection and analysis

Data was collected using the Black Dog Institute Research Engine. Participants’ text data was pre-processed in Microsoft Excel (see Supplementary Material Section 2, S.2.). Linguistic features were extracted using the validated Linguistic Inquiry and Word Count (LIWC-22) tool ⁴⁰ which calculated the percentage of words within a text that reflected a predefined dictionary of 117 linguistic features. The LIWC-22 summary variables (analytical thinking, clout, authenticity, and emotional tone) were excluded from our analyses due to the lack of transparency on the dimensions of these variables, resulting in 113 linguistic features. Linguistic features were first extracted within each task, then averaged across tasks, while mental health scores were averaged across task timepoints in the combined analyses.

All analyses were performed using R 4.2.1 ⁴¹. Correlation plots were created using the ‘ggplot2’ package in R ⁴². Nested cross-validation machine learning models were developed using the “nestedcv” package in R ⁴³ and the ‘caret' package in R ⁴⁴. Between-subject ANOVAs were used to compare differences in mental health outcomes across the four trial sequences at baseline. ANCOVAs were also used to control for the demographics age, gender, and education. Post-hoc tests were corrected for using Tukey’s HSD method. The four sequences were then combined into one sample for analysis given no significant differences in symptoms across sequences (see Supplementary Material Section 3, S.3.1). Pearson correlations were used to test associations between linguistic features and mental health symptoms. The false discovery rate of p-values were adjusted to q-values using the Benjamani and Hochberg approach⁴⁵. Models were run with and without outliers for sensitivity testing.

When testing the associations between participants’ linguistic features and mental health symptoms, common machine learning models including elastic net models, random forest, and support vector machines were used to test whether linguistic features predicted mental health symptoms. Models were evaluated using nested cross-validation (see Supplementary Materials Section 3, S.3 for rationale). Performance of the models was assessed by inspecting the RMSE, r², the mean absolute error (MAE), and by correlating the predicted and observed values. Models were also developed using demographics (age, gender, education) and clinical characteristics (diagnosis age, diagnosis onset, medication status) to determine whether the inclusion of these could improve model performance. One-hot encoding was used for categorical variables and those with more than two levels were dichotomized, including education (secondary schooling versus university education), and gender (male versus female; for a full list of variables, see Supplementary Material Section 4, Table S.4). Variable stability plots were generated to visualize how often predictors were selected in each outer fold and final model selection. Feature importance for final models fit to the whole dataset were assessed using Shapley Additive Explanation (SHAP) values, which quantify the contribution of each feature to the predictions of a machine learning model for each participant. A higher absolute SHAP value implies a stronger impact of that feature on the model's output. Bee swarm plots were also used to visualize SHAP values and inspect the distribution of the effect across participants.

Mixed linear models were used to test whether the expressive text tasks impacted mood and the acceptability scores. Models included mood scores (MDMQ) and acceptability scores as the dependent variables, subject as the random effect, and time, sequence group, and the interaction between time and sequence group as the fixed effects. For the impacts of the text tasks on mood, planned contrasts examined the changes in mood between completions 1 and 2, completions 3 and 4, completions 5 and 6 only. The method of Kenward and Roger was used to estimate the degrees of freedom for all these tests.

Participants

A total of 494 adults were assessed for study eligibility. Of these, 247 consented, completed baseline and were randomised. A total of 29 participants actively withdrew after randomisation (see Figure 2 for study flow).

Table 2 outlines the baseline characteristics of the final sample (N=218). The majority were female (74.8%), Australian born (80.3%), married or partnered (52.8%), in paid employment (74.8%) with a mean age of 39.33 years (SD: 12.37, range: 18 to 69 years) and English as the main language spoken at home (90.8%). The majority (85.3%) reported that they never required any literacy support. Nearly all participants reported that they had received a formal diagnosis of a mental illness at baseline (92.7%, n=202/218) and over half were taking prescribed medication for a mental illness (59.6%, n=130/218).

Table 2. Baseline characteristics of sample (N=218)

	Total sample
	M	SD
Age (years)	39.33	12.37
Depressive symptoms (PHQ9)	13.50	4.13
Anxiety symptoms (GAD7)	10.83	4.51
	N	%
Female	163	74.8
University education	105	48.2
Married/Partnered	115	52.8
In paid employment	163	74.8
Born in Australia	175	80.3
First Nations	5	2.3
LGBTQI+	45	20.6
English only	198	90.8
Literacy support (Never)	186	85.3
Formally diagnosed mental illness	202	92.7
Age of onset	19.91	10.83
Age of diagnosis	26.94	10.93
Taking prescribed medication	130	59.6
Likely case of Major Depressive Disorder	193	88.5
Likely case of Generalized Anxiety Disorder	134	61.5

Note: Likely case of Major Depressive Disorder (MDD) determined by a PHQ9 score ≥ 10; Likely case of Generalised Anxiety Disorder (GAD) determined by a GAD7 score ≥ 10.

Completion of text tasks

On average, participants attempted 16.41 out of the 19 tasks (SD: 4.87, range: 0 to 19). The majority completed the text tasks on a mobile device (88.5%). Participants in sequence 4 (M: 14.84, SD: 5.99) attempted significantly fewer tasks compared to those in sequence 2 (M: 17.43, SD: 3.86; t=2.843, P<.05). No other differences in completion rates were found. Table 3 presents the word counts, character counts, and proportion of dictionary words within tasks.

Table 3: Word counts, character counts, and proportion of dictionary words within tasks per task and per participant

Task	Task - word count	Task - character count	Task - proportion of dictionary words	Participant corpus - word count	Participant corpus - character count	Participant corpus -proportion of dictionary words
	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)	M (SD)
A. SMS retrieval	64.88 (76.88)	336.17 (400.13)	93.01 (5.10)	280.88 (258.14)	1455.37 (1351.33)	93.07 (2.96)
B. Social media posts	62.18 (63.30)	328.10 (331.63)	94.09 (6.04)	219.81 (180.47)	1159.87 (946.47)	94.13 (3.66)
C. Emotion diaries	94.31 (75.62)	488.09 (394.78)	95.43 (3.59)	259.70 (189.24)	1344.03 (988.37)	95.45 (2.37)
D. Expressive writing	223.01 (198.22)	1164.18 (1034.49)	95.53 (2.65)	527.50 (479.72)	2753.89 (2506.60)	95.54 (2.04)
V. Personal description	-	-	-	197.34 (144.06)	1041.64 (761.26)	92.90 (3.42)
W. Friend description	-	-	-	167.02 (142.39)	870.65 (742.69)	95.59 (2.50)
X. Reflection of a complex image	-	-	-	79.44 (53.9)	430.74 (291.26)	92.26 (4.79)
Y. Holidaying letter	-	-	-	117.21 (72.00)	603.29 (373.16)	92.64 (3.96)

Within-task associations between mental health symptoms and linguistic features

Significant correlations between mental health symptoms and the 113 linguistic features were found in three of the eight tasks. In the Task B corpus (i.e., combined social media posts), higher depressive symptoms were associated with greater expressions of “power” words (r = .27, q = .010), greater expressions of “sadness” words (r = .26, q = .010) and reduced first-person plural pronouns (r = -.26, q = .010). See Supplementary Material for correlation plots (Section 4, Figure S.4.3). As shown in Table 4, these features were not found to be consistently associated with depressive symptoms when the Task B completions were examined separately.

Table 4: Associations between the significant task-level linguistic features (power words, sadness words, first-person plural pronouns) and depressive symptoms within each Task B completion.

Task - Completion	Power words (r)	Sadness words (r)	First-person plural pronouns (r)	q-range
Task B-1	.09	.18**	-.17*	.546 - .929
Task B-2	.22**	.08	-.17*	.272 – 1.0
Task B-3	.05	.21**	-.14	.229 - .986
Task B-4	.17*	.05	-.09	.551 – 1.0

Table note: *Unadjusted P< .05, **Unadjusted P<.02.

In Task V (personal description), higher levels of depressive symptoms were associated with greater use of “cognition” words (r = .25, q = .032), greater use of “tentative” words (r = .25, q = .032), and higher levels of “negative tone” (r = .23, q = .045). When outliers were removed, “tentative” words (r = .19, q = .175) and “negative tone” (r = .17, q = .195) were non-significant. In Task W (description of friends), higher levels of depressive symptoms were associated with greater expressions of “want” words (r = .27, q = .028). See Supplementary Material for correlation plots (Section 4, Figure S.4.4 and Figure S.4.5). No other significant correlations were found.

Shown in Table 5, the within-task machine learned models predicted up to 8% (Task B corpus) of the variance in depressive symptoms and up to 4% (Task B corpus) of the variance in anxiety symptoms. The best-performing models varied depending on the task. The models that included the linguistic features from the Task B corpus together with clinical and demographic characteristics accounted for the greatest level of variance in symptoms. The observed and predicted values from the outer cross-validation folds were significantly correlated for all depression models whereas this pattern was only found for Tasks A, B, V, and X in the anxiety models. Across the models, depression scores were predicted with a mean absolute error of 4.13-4.77 points on the PHQ-9 scale, and anxiety scores were predicted with a MAE of 4.14-4.51 points on the GAD-7 scale. The importance and directionality of linguistic features in the models varied greatly between tasks, with clinical and demographic features showing high importance in some models (see Supplementary Material Section 4 Figures S.4.6 to S.4.16).

Table 5. The best performing within-task machine learning models using linguistic features to predict depressive and anxiety symptoms.

	Depressive symptoms (PHQ-9)
	Model	RFE	Demographic and clinical features	RMSE	r²	MAE	r	p	Hyperparameters for final fitted model
Task A	SVM-NL	No	Yes	5.77	.05	4.69	.22	.005	C = 1 Σ = .001
Task B	Elastic Net	No	Yes	5.27	.08	4.41	.28	<.001	α = .6 λ = 1
Task C	Random Forest	No	No	5.74	.04	4.68	.20	.006	mtry = 50
Task D	SVM-NL	No	Yes	5.17	.04	4.24	.19	.018	C = 2 Σ = .008
Task V	Random Forest	No	No	5.15	.03	4.13	.18	.012	mtry = 16
Task W	Elastic Net^	No	Yes	5.82	.05	4.77	.22	.005	α = 0 λ = 16
Task X	Random Forest	Yes	Yes	5.66	.05	4.50	.22	.005	mtry = 50
Task Y	Random Forest	No	Yes	5.62	.07	4.61	.26	<.001	mtry = 8
	Anxiety symptoms (GAD-7)
	Model	RFE	Demographic and clinical features	RMSE	r²	MAE	r	p	Hyperparameters for final fitted model
Task A	Random Forest	No	Yes	5.26	.03	4.51	.16	.042	mtry = 1
Task B	SVM-L	No	Yes	5.03	.04	4.22	.21	.007	C = .001
Task C	Random Forest	Yes	Yes	5.02	.02	4.14	.15	.058	mtry = 1
Task D	Elastic Net	Yes	No	5.03	.01	4.24	.04	.585	α = 0 λ = 2
Task V	Elastic Net	No	No	4.90	.03	4.14	.16	.027	α = .5 λ = 1
Task W	SVM-L	No	Yes	5.29	.01	4.46	.09	.257	C = .001
Task X	SVM-L	Yes	Yes	5.02	.03	4.25	.18	.022	C = .016
Task Y	SVM-L	No	Yes	5.13	.01	4.28	.12	.147	C = .0005

Table note: Task descriptions: A=SMS retrieval, B=Social media post, C=Emotion diary, D=Expressive writing, V=Personal description, W=Friends description, X=Reflection of a complex image, Y=Holidaying letter. ^α = 0 represents ridge regression in the elastic net framework.

Associations between mental health symptoms and linguistic features when all tasks were combined.

As shown in Table 6, only two of the linguistic features found to be significantly associated with depressive symptoms within the tasks were found to be significant when texts from all the tasks were combined into one corpus. In this analysis, higher levels of depressive symptoms were associated with greater use of “cognition” words (r = .24, q = .024) and “cognitive process” words (r = .22, q = .027), lower levels of “positive tone” (r = -.32, q < .001) and “positive emotion” (r = -.21, q = .037), reduced first-person plural pronouns (r = -.25, q = .016), and reduced “affiliation” words (r = -.23, q = .024). Higher levels of anxiety symptoms were associated with greater use of “cognition words” only (r = .26, q = .022). See Supplementary Material (Figures S.4.17) for the correlation plots.

Table 6. Linguistic features significantly associated with depressive and anxiety symptoms within-tasks and when tasks were combined.

Linguistic features associated with depressive symptoms (PHQ9) within-tasks	Task
	A	B	C	D	V	W	X	Y
Cognition	-	-	-	-	*	-	-	-
First-person plural pronouns	-	*	-	-	-	-	-	-
Power	-	*	-	-	-	-	-	-
Sadness	-	*	-	-	-	-	-	-
Want	-	-	-	-	-	*	-	-
Linguistic features associated with depressive symptoms (PHQ9) when combined	A	B	C	D	V	W	X	Y
Affiliation	-	-	-	-	-	-	-	-
Cognition	-	-	-	-	*	-	-	-
Cognitive processes	-	-	-	-	-	-	-	-
First-person plural pronouns	-	*	-	-	-	-	-	-
Positive emotion	-	-	-	-	-	-	-	-
Positive tone	-	-	-	-	-	-	-	-
Linguistic features associated with anxiety symptoms (GAD7) within-tasks	Task
	A	B	C	D	V	W	X	Y
None	-	-	-	-	-	-	-	-
Linguistic features associated with anxiety symptoms (GAD7) when combined	A	B	C	D	V	W	X	Y
Cognition	-	-	-	-	-	-	-	-

Table note: Linguistic features listed in left column are all those found to be associated with symptoms of depression (PHQ-9) and anxiety (GAD-7) at either the task level or when combined. * indicates the features with a significant q value for that task when examined with outliers removed. – indicates no significant q value for that task with outliers removed.

Shown in Table 7, random forest models using only linguistic features were the best performing models, predicting 8% of the variance in depressive symptoms and 3% of the variance in anxiety symptoms. The observed and predicted values from the outer cross-validation folds were significantly correlated for both the depression model (r = .28, p < .001) and the anxiety model (r = .17, p < .05). As displayed in Figure 6, “positive tone” was the most important linguistic feature for predicting depressive symptoms with negative directionality (i.e., higher values lead to lower symptoms). First-person plural pronouns, “time”, “motion”, “substance” and “wellness” words also emerged as somewhat important to the model. The most important linguistic features with positive directionality (i.e., lower values lead to higher symptoms) were conjunctions, “function” “differentiation” and “death” words. The most important linguistic features for predicting anxiety symptoms with negative directionality were past-focussed words, positive tone, “drive” words, third-person singular pronouns, and “article” words. The most important features with positive directionality were “anxiety” words, conjunctions, linguistic words, “all-or-nothing” words, and “discrepancy” words. See Supplementary Material (Figures S.4.18) for the variable stability plots.

Table 7. Best performing across-task machine learning models using linguistic features to predict mental health symptoms.

				Performance metrics			Correlation between observed and predicted test values
Mental health symptoms	Best performing model	Recursive feature elimination	Demographic and clinical features	RMSE	r²	MAE	r	p	Hyperparameters for final fitted model
Depression	Random Forest	No	No	4.87	.08	3.89	.28	<.001	mtry = 70
Anxiety	Random Forest	No	No	4.50	.03	3.70	.17	.015	mtry = 30

Impact of emotional text tasks (Task C and Task D) on participants’ mood and acceptability ratings

There were almost no immediate changes in participants’ mood, as measured by the MDMQ, after completing the emotion diary tasks (Task C, P=.057 to .987) except for higher levels of relaxation after the second completion (MD: 0.26, df=895.81, P=.030) and higher levels of agitation after the third completion (MD: 0.34, df=896.42, P=.008). For the expressive task (Task D), participants reported significantly greater fatigue (P=.001 to P=.045), lower levels of contentment (P=.000-.007), higher levels of agitation (all P<.001), lower levels of energy (P=.002 to 0.40), feeling more unwell (P=.000 to .023) and lower levels of relaxation (all P<.001) after all completions (see Supplementary Material Section 4 for more detail). There were no main effects of sequence or interaction effects for sequence by time on any of these findings.

Shown in Table 8, the acceptability ratings of the tasks were mixed. Across the whole sample, the expressive tasks (Task D) were significantly more difficult to complete (M: 3.0, SD: 1.9) than all other tasks (all P < .001). Participants were also significantly less willing to complete Task D (M: 4.7, SD: 1.9) when compared to all other tasks (all P ranged from <.001 to .012). However, participants’ levels of interest and willingness were influenced by their allocation sequence. Participants in sequence 1 reported significantly greater interest in completing tasks when compared to sequences 3 (P=.026) and 4 (P=.006). Participants in sequence 1 were also significantly more willing to repeat the tasks when compared to sequences 3 (P=.036) and 4 (P=.036).

Table 8. Omnibus test results from mixed models assessing acceptability across tasks and sequences.

	Variable	*df_num*	*df_den*	F	P
Ease of completion	Task	7	1306.66	37.51	<.001
	Sequence	3	202.64	0.53	.661
	Task * Sequence	21	1306.58	0.80	.723
Level of interest	Task	7	1308.55	10.59	<.001
	Sequence	3	206.74	4.75	.003
	Task * Sequence	21	1308.49	0.89	.610
Willingness to repeat	Task	7	1303.95	12.16	<.001
	Sequence	3	202.94	3.33	.021
	Task * Sequence	21	1303.89	3.22	<.001

This experimental study examined the utility of several types of text data for generating valid and reliable linguistic markers of depression and anxiety when collected among symptomatic individuals, using a trial design that accounted for sequence and potential task effects. The findings only partially supported the hypotheses. There were no linguistic features that were consistently associated with depressive or anxiety symptoms within tasks or across all tasks. At the task level, five linguistic features were associated with depressive symptoms but these features were elicited in only three of the eight tasks and associations were weak. Notably, the features found to be associated with depressive symptoms were different for each task and there was only some degree of reliability of these features within the repeated tasks. When all tasks were combined, six linguistic features were associated with depressive symptoms. However, only two of these features were correlated at the task-level. Only one linguistic feature was associated with anxiety, but this feature was not significant at the task level. In all the machine learning models, predicted values were weakly associated with actual values for both depressive and anxiety symptoms. Positive tone emerged as the most important linguistic feature for predicting variance in depressive symptoms, yet this feature was not significant within any of the tasks. Linguistic features were much less important in the anxiety model than the depression model. Overall, these findings indicate that there may be few valid and reliable group-level linguistic markers of depression and anxiety when examining several types of individuals’ text data.

Notably, our results do not replicate the findings of many past studies that used the same types of text tasks (e.g. ^15,17,18,46). In this trial, only three of the eight tasks revealed significant linguistic markers of depressive and anxiety symptoms when controlling for the false discovery rate: social media posts, personal description of one’s self, and description of one’s friends. The linguistic markers found within these tasks do suggest that some types of word use may reflect the low mood, social withdrawal, and impaired desire that characterise depression. The findings also suggest that writing about one’s self and one’s social network may differentiate cognitive processing in individuals with depressive symptoms (as demonstrated by higher use of the severity of a depressive state than personal communications (e.g. SMS data) or emotion processing tasks (e.g. emotional event recall). However, the markers that were significantly correlated with depressive symptoms were unique to each of the personal reflection tasks and not all were significant in the combined analysis. There was also limited reliability for the linguistic correlates in the social media data, indicating that the robustness of these features may be influenced by the amount of data available for repeated analyses. As such, the results of this trial do not provide definitive evidence of the strength of linguistic markers of depressive symptoms from within the included tasks.

Overall, there was poor generaliseability for the linguistic markers examined in this trial as none of the markers were consitent within and across all tasks. This provides further evidence of a task effect and suggests that this approach to digital phenotyping may not yet yield great benefit for detecting depressive symptoms unless the contextual factors or priming effects within the tasks are accounted for. Thus, researchers wishing to use either organic or simulated text data for the digital phenotyping of mental illness need to carefully consider the conditions under which the text data was generated and account for variations in the quantity and frequency of data collected. While the results do not point to a single or set of valid and reliable linguistic markers of depressive or anxiety symptoms, the overall degree of positive tone expressed in individuals’ text data may have some utility for predicting depressive scores at the group-level. However, this finding also requires further validation in larger samples with varying types of text data, collected again at varying frequencies and over longer periods of time, given it didn’t emerge as a within-task marker. As a goal of digital phenotyping is personalised medicine, future research may also benefit from focusing greater efforts on examining person-level variations in linguistic expression over time. This may be particualarly relevant to linguistic analysis, given prior work has showed group-level models do not have strong predictive value for individuals ⁴⁷. However, deciding which features to include in analyses of this kind remains a serious methodological challenge ⁴⁸. Interestingly, demographics and clinical characteristics were absent from some of the best-fitting models, indicating that this information did not always improve the accuracy of the machine learning. This may have broader implications for machine learning approaches in digital phenotyping, whereby more exploratory work is needed to understand any subtle interactions between personal or illness characteristics, digital signals, and different use cases. For example, when using digital phenotyping for monitoring an individual when observers may already have some knowledge of their mental state or circumstance (e.g. monitoring in clinical practice) in contrast to using digital phenotyping for tracking the mental health of a population with unknown parameters.

Lastly, while rates of task completion were high in this study, acceptability of the tasks were mixed. In particular, the expressive tasks that focused on memory recall were found to negatively impact various dimensions of participants’ mood, after all occasions, and were poorly accepted by participants. Although this task did not explicitly ask participants to focus on a negative emotional event, given the negative cognitive biases caused by depression, it is highly likely that many participants chose to reflect on negative events. Qualitative analysis of participants’ responses to this task may help to provide greater insight of the topics discussed that may be contributing to the negative effects on mood. Future research should also carefully examine the ethical implications and design implications of using some structured expressive tasks for the collection of linguistic markers among symptomatic samples.

Strengths and Limitations

A key strength of the current study was the use of a cross-over trial design with repeated measures, as the results confirmed that the order in which text tasks were administered impacted participants’ completions as well as their levels of interest and engagement. Within-task and across task analyses also enabled greater exploration of the validity and reliability of the linguistic features by providing a larger testing set of varying text data. However, the generalisability of the findings may be limited, given most participants were English-literate females. Models applied to samples with varying degrees of English proficiency may have different results. As our design focused on individuals with elevated symptoms, different results may also be found when attempting to use linguistic markers to distinguish depressed individuals from non-depressed, or stratification of sub-types of depression. There is also evidence of potential floor effects for some of the linguistic features when examined by task that were not present when all tasks were combined. This suggests that it may have been the tasks eliciting lower base rates of some linguistic features rather than individuals’ depressive and anxiety symptoms. This may also be related to the word counts, given that on average, participants produced fewer than 300 words for most of the tasks. While the cleaning of data helped to ensure the accuracy of the LIWC, the text data were not passively collected. As such, the measurement effects within this study may not be found when other types of text data of varying lengths and collection methods are examined or updated versions of LIWC are applied to the data in the future.

Conclusions

The findings suggest limited validity and reliability of group-level linguistic markers for depression and anxiety when using a variety of harvested text data. The findings provide evidence of a task-effect on linguistic expression whereby the markers generated in one context may not be generaliseable to another. This underscores the need for researchers to account for contextual variations when attempting to use linguistic analysis for digital phenotyping of mental illness. While positive tone shows some potential for predicting depressive symptoms at the group level, its utility requires validation in larger samples with diverse text data collected passively. Future research should focus on individual-level variations in linguistic expression over time, addressing methodological challenges in feature selection. Additionally, careful examination of task acceptability, particularly expressive writing tasks, is warranted for collecting linguistic markers among symptomatic samples.

Snyderman, R. Personalized health care: From theory to practice. Biotechnol. J 7, 973–979, doi:10.1002/biot.201100297 (2012).
Zarate, D., Stavropoulos, V., Ball, M., de Sena Collier, G. & Jacobson, N. C. Exploring the digital footprint of depression: a PRISMA systematic literature review of the empirical evidence. BMC Psychiatry 22, 421, doi:10.1186/s12888-022-04013-y (2022).
Liang, Y., Zheng, X. & Zeng, D. D. A survey on big data-driven digital phenotyping of mental health. Information Fusion 52, 290–307, doi:10.1016/j.inffus.2019.04.001 (2019).
DeSouza, D. D., Robin, J., Gumus, M. & Yeung, A. Natural Language Processing as an Emerging Tool to Detect Late-Life Depression. Front Psychiatry 12, 719125, doi:10.3389/fpsyt.2021.719125 (2021).
Kamath, J., Leon Barriera, R., Jain, N., Keisari, E. & Wang, B. Digital phenotyping in depression diagnostics: Integrating psychiatric and engineering perspectives. World J Psychiatry 12, 393–409, doi:10.5498/wjp.v12.i3.393 (2022).
Rimes, K. A. & Watkins, E. The effects of self-focused rumination on global negative self-judgements in depression. Behav Res Ther 43, 1673–1681, doi:10.1016/j.brat.2004.12.002 (2005).
Gotlib, I. H. & Joormann, J. Cognition and depression: current status and future directions. Annu. Rev. Clin. Psychol. 6, 285–312, doi:10.1146/annurev.clinpsy.121208.131305 (2010).
Du, J. Y., Hallford, D. J. & Busby Grant, J. Characteristics of episodic future thinking in anxiety: A systematic review and meta-analysis. Clin. Psychol. Rev. 95, 102162, doi:10.1016/j.cpr.2022.102162 (2022).
Hallford, D. J., Austin, D. W., Takano, K. & Raes, F. Psychopathology and episodic future thinking: A systematic review and meta-analysis of specificity and episodic detail. Behav Res Ther 102, 42–51, doi:10.1016/j.brat.2018.01.003 (2018).
Hallford, D. J. et al. Impairments in episodic future thinking for positive events and anticipatory pleasure in major depression. J. Affect. Disord. 260, 536–543, doi:10.1016/j.jad.2019.09.039 (2020).
Edwards, T. & Holtzman, N. S. A meta-analysis of correlations between depression and first person singular pronoun use. J of Res Pers 68, 63–68, doi:10.1016/j.jrp.2017.02.005 (2017).
Tølbøll, K. B. Linguistic features in depression: a meta-analysis. Language Works 4, 39–59 (2019).
Savekar, A., Tarai, S. & Singh, M. Structural and functional markers of language signify the symptomatic effect of depression: A systematic literature review. Eur. J. Appl. Linguist. 11, 190–224, doi:doi:10.1515/eujal-2022-0022 (2023).
Rook, L., Mazza, M. C., Lefter, I. & Brazier, F. Toward Linguistic Recognition of Generalized Anxiety Disorder. Front Digit Health 4, 779039, doi:10.3389/fdgth.2022.779039 (2022).
Lee, S. W. et al. Insights from an expressive writing intervention on Facebook to help alleviate depressive symptoms. Comput. Human Behav. 62, 613–619, doi:10.1016/j.chb.2016.04.034 (2016).
Burkhardt, H. A. et al. Behavioral Activation and Depression Symptomatology: Longitudinal Assessment of Linguistic Indicators in Text-Based Therapy Sessions. J. Med. Internet Res. 23, e28244, doi:10.2196/28244 (2021).
Havigerová, J. M., Haviger, J., Kučera, D. & Hoffmannová, P. Text-Based Detection of the Risk of Depression. Front. Psychol. 10, doi:10.3389/fpsyg.2019.00513 (2019).
Minori, G. et al. Linguistic Markers of Anorexia Nervosa: Preliminary Data from a Prospective Observational Study. (2020).
Cuteri, V. et al. Linguistic feature of anorexia nervosa: a prospective case-control pilot study. Eat Weight Disord 27, 1367–1375, doi:10.1007/s40519-021-01273-7 (2022).
Kaźmierczak, I., Sarzyńska-Wawer, J., Wawer, A. & Chądzyńska, M. Describing a critical life event and its psychological consequences: The type of language used by patients suffering from depression and its relationship with personality development. Current Psychology 41, 4368–4378, doi:10.1007/s12144-020-00944-5 (2022).
Reinhold, M., Bürkner, P.-C. & Holling, H. Effects of expressive writing on depressive symptoms—A meta-analysis. Clin. Psychol. 25, e12224, doi:10.1111/cpsp.12224 (2018).
Stamatis, C. A. et al. Prospective associations of text-message-based sentiment with symptoms of depression, generalized anxiety, and social anxiety. Depress. Anxiety 39, 794–804, doi:10.1002/da.23286 (2022).
Meyerhoff, J. et al. Analyzing text message linguistic features: Do people with depression communicate differently with their close and non-close contacts? Behav Res Ther 166, 104342, doi:10.1016/j.brat.2023.104342 (2023).
O'Dea, B., Larsen, M. E., Batterham, P. J., Calear, A. L. & Christensen, H. A Linguistic Analysis of Suicide-Related Twitter Posts. Crisis, 1–11, doi:10.1027/0227-5910/a000443 (2017).
Chancellor, S. & De Choudhury, M. Methods in predictive techniques for mental health status on social media: a critical review. npj Digital Medicine 3, 43, doi:10.1038/s41746-020-0233-7 (2020).
Park, M., Cha, C. & Cha, M. in ACM SIGKDD Workshop on Intelligence and Security Informatics (Beijing, China, 2012).
Reece, A. G. et al. Forecasting the onset and course of mental illness with Twitter data. Sci. Rep. 7, 13006, doi:10.1038/s41598-017-12961-9 (2017).
Seabrook, E. M., Kern, M. L., Fulcher, B. D. & Rickard, N. S. Predicting Depression From Language-Based Emotion Dynamics: Longitudinal Analysis of Facebook and Twitter Status Updates. J. Med. Internet Res. 20, e168, doi:10.2196/jmir.9267 (2018).
Yang, W. & Mu, L. GIS analysis of depression among Twitter users. Applied Geography, doi:10.1016/j.apgeog.2014.10.016 (2014).
Liu, D., Feng, X. L., Ahmed, F., Shahid, M. & Guo, J. Detecting and Measuring Depression on Social Media Using a Machine Learning Approach: Systematic Review. JMIR Ment Health 9, e27244, doi:10.2196/27244 (2022).
Waterloo, S. F., Baumgartner, S. E., Peter, J. & Valkenburg, P. M. Norms of online expressions of emotion: Comparing Facebook, Twitter, Instagram, and WhatsApp. New Media & Society 20, 1813–1831, doi:10.1177/1461444817707349 (2018).
Tlachac, M. L. & Rundensteiner, E. Screening For Depression With Retrospectively Harvested Private Versus Public Text. IEEE J Biomed Health Inform 24, 3326–3332, doi:10.1109/jbhi.2020.2983035 (2020).
Liu, T. et al. Different Affordances on Facebook and SMS Text Messaging Do Not Impede Generalization of Language-Based Predictive Models. Proceedings of the Seventeenth International AAAI Conference on Web and Social Media 17, 1153–1157, doi:10.1609/icwsm.v17i1.22226 (2023).
Smirnov, I. V., Ushakova, A. V. & Chudova, N. V. in Artif. Intell. (eds Sergei O. Kuznetsov, Aleksandr I. Panov, & Konstantin S. Yakovlev) 325–337 (Springer International Publishing).
Biddle, L., Donovan, J., Sharp, D. & Gunnell, D. Explaining non-help-seeking amongst young adults with mental distress: a dynamic interpretive model of illness behaviour. Sociol. Health Illn. 29, doi:10.1111/j.1467-9566.2007.01030.x (2007).
Rosenthal, S., McKeown, K., Lin, D., Matsumoto, Y. & Mihalcea, R. in Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 763–772 (Association for Computational Linguistics).
Chen, Y., Sherren, K., Smit, M. & Lee, K. Y. Using social media images as data in social science research. New Media & Society 25, 849–871, doi:10.1177/14614448211038761 (2023).
Bruns, A. in Disinformation and Data Lockdown on Social Platforms (eds Shawn Walker, Dan Mercea, & Marco Bastos) 23 (Routledge, 2021).
Wilhelm, P. & Schoebi, D. Assessing Mood in Daily Life. Eur. J. Psychol. Assess. 23, 258–267, doi:10.1027/1015-5759.23.4.258 (2007).
Boyd, R. L., Ashokkumar, A., Seraj, S., & & Pennebaker, J. W. The Development and Psychometric Properties of LIWC-22 (University of Texas at Austin, 2022).
R Core Team. R: A language and environment for statistical computing. (R Foundation for Statistical Computing, 2022).
Wickham, H. ggplot2: Elegant Graphics for Data Analysis. (Springer-Verlag 2016).
Lewis, M. J. et al. nestedcv: an R package for fast implementation of nested cross-validation with embedded feature selection designed for transcriptomics and high-dimensional data. Bioinformatics Advances 3, vbad048, doi:10.1093/bioadv/vbad048 (2023).
Kuhn, M. caret: Classification and Regression Training. R package version 6.0–93. (https://CRAN.R-project.org/package=caret, 2022).
Benjamini, Y. & Hochberg, Y. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Methodological) 57, doi:10.1111/j.2517-6161.1995.tb02031.x (1995).
Cuteri, V. & Minori, G., et al. Linguistic Feature of Anorexia Nervosa: A Prospective Case-Control Pilot Study. Research Square, 10.21203/rs.21203.rs-186615/v186611 (2021).
O'Dea, B. et al. The relationship between linguistic expression in blog content and symptoms of depression, anxiety, and suicidal thoughts: A longitudinal study. PLoS One 16, e0251787, doi:10.1371/journal.pone.0251787 (2021).
Just, M. A. et al. Retraction Note: Machine learning of neural representations of suicide and emotion concepts identifies suicidal youth. Nature Human Behaviour 7, 824–824, doi:10.1038/s41562-023-01581-1 (2023).

Funding information:

This research was funded by the National Health and Medical Research Council (GNT 1165233). BOD is supported by a National Health and Medical Research Council Medical Research Future Fund Investigator Grant (MRF1197249). MB is supported by a National Health and Medical Research Council Leadership 3 Investigator Grant (GNT2017131).

Competing interests:

No competing interests to declare.

No competing interests reported.

SupplementaryMaterial.docx

The (not-so) valid and reliable linguistic markers of depression and anxiety in symptomatic adults: A randomised cross over trial

Status:

Version 1

Abstract

Figures

1. INTRODUCTION

1.1. Linguistic markers of depression and anxiety

1.2. Objectives of current study

2. METHODS

2.1. Design

2.2. Participants and sample size

2.3. Recruitment and consent

2.4. Procedure

2.5. Measures

2.6. Data collection and analysis

3. RESULTS

4. DISCUSSION

References

Declarations

Additional Declarations

Supplementary Files

Status:

Version 1