Background: Clinical assessment and scientific research in psychiatry is largely based on questionnaires that are employed to detect and quantify psychopathology. The development of large language models offers a new perspective for the analysis of the language and terminology that these questionnaires are based on.
Methods: In the present study, we employed state-of-the-art large language models to derive numerical representations (so called ‘text embeddings’) of semantic and sentiment content of items from established questionnaires for the assessment of psychopathology. We compared the pairwise associations between empirical data and text embeddings in order to test if the empirical structure of psychopathology can be reconstructed by large language models.
Results: Across four large-scale data sets (n=1555, n=1099, n=11807, n=39755), we found a range of significant correlations between empirical item-pair associations and associations derived from text embeddings (r=0.18 to r=0.57, all p<0.05). Machine learning models based on semantic or sentiment embeddings predicted empirical item-pair associations with moderate to high accuracy (r=0.33 to r=0.81, all p<0.05). Similarly, empirical clustering of items and the grouping to established subdomain scores could be partly reconstructed by text embeddings.
Conclusion: The present results demonstrate that large language models are able to represent substantial components of the empirical structure of psychopathology. Consequently, the integration of large language models into mental health research holds the potential to unlock numerous promising avenues. These may encompass improving the process of generating novel questionnaires, optimising generalizability and redundancy of existing questionnaires or facilitating the development of novel conceptualizations of mental disorders.