Prevalence and Usage Trends of Large Language Models in Health Science Publications: A Cross-sectional Analysis

doi:10.21203/rs.3.rs-5259782/v1

Download PDF

Research Article

Prevalence and Usage Trends of Large Language Models in Health Science Publications: A Cross-sectional Analysis

https://doi.org/10.21203/rs.3.rs-5259782/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background. The advent of Large Language Models (LLMs) has significantly impacted scientific research and publishing. However, the extent and nature of this impact across health sciences remain unclear.

Objectives. To determine the prevalence and trends of LLM usage in health science publications from 2022 and 2024, and to analyze variations across countries, publishers, journals, and institutions.

Methods. We conducted a comprehensive bibliometric analysis of publications indexed in the Web of Science Core Collection across twelve major health science disciplines. LLM usage was assessed using linguistic markers associated with LLM-generated text from the first five months (January 1 to May 31) of 2022 and 2024. Comparative analyses were performed with and without common LLM-related adjectives and adverbs.

Results. The analysis revealed significant shifts in the global landscape of LLM usage in health science publications from 2022 to 2024. English-speaking countries, such as the USA (-10.350%, p<0.001) and England (-2.095%, p<0.001), showed a marked decrease in LLM-related research output, while Asian and Middle Eastern countries demonstrated substantial increases (China: +12.473%, p<0.001; India: +1.821%, p<0.001). This trend was mirrored at the institutional level, with prestigious Western institutions experiencing declines (Harvard University: -0.665%, p<0.001) and Chinese institutions showing growth (Chinese Academy of Sciences: +0.295%, p=0.054). Global publishers like Springer Nature (+9.178%, p<0.001) and open-access journals such as Cureus Journal of Medical Science (+3.040%, p<0.0001) saw significant increases in LLM-related publications

Conclusion. The transition in the use of LLM from English-speaking to non-English-speaking countries, especially in Asia, presents both opportunities and challenges. These include addressing potential biases and differences in research integrity standards. This highlights the need for international collaboration to create consistent, ethical standards for LLMs in research. Future efforts should focus on developing methods to detect the use of LLMs use and assess their impact, ensuring that scientific outputs are equitable and of high quality globally.

Scientific Communication

Large Language Models

Health Sciences

Research Trends

Artificial Intelligence

Scientific Writing

LLM

ChatGPT

GPT-4

Large Language Models (LLMs) have arrived, with several influences across different sectors, including scientific research and academic writing. LLMs such as ChatGPT, GPT-4, and their equivalents have shown unique prowess in natural language processing, text synthesis, and even untraditional computational tasks [1–3]. These advancements have generated both optimism and concern among scientists and academics, especially due to the concerns about the reliability and quality of published academic words [4].

The integration of LLMs into the research process has been rapid and extensive. In fact, in a cross-sectional study undertaken in the last quarter of 2023 estimated that about 30% of researchers had employed artificial intelligence (AI) technologies for manuscript writing [5]. This rapid adoption raises questions about how LLMs are influencing the content, format, and overall tone of scientific literature, especially in the broad and highly sensitive field of 1 sciences.

The application of LLMs in academic writing is twofold, encompassing both positive and negative impacts: (i) these tools can optimize the writing process, enhance language quality, and improve access to high-quality scientific literature [6], and conversely, (ii) these models can generate incorrect data, maintain biases, or contribute to instances of academic dishonesty [7]. There is also an issue of LLMs ‘hallucinating’ references, where the system outputs seemingly reasonable citations that do not exist at all, which is similarly an issue menacing the integrity of published scientific papers [8]. As a result, several publishers have begun offering guidelines on the use of AI technologies in manuscript preparation. For example, Wiley as a publisher has allowed the use of AI in content creation but the authors are fully responsible for their statements and must openly state if the AI assisted them in generating content [9]. However, the appropriateness and extent of these policies' implementation remain unclear.

That is not the only way that LLMs affect scholarly publishing. It is increasingly argued that these tools are used in the process of peer review and this is evidenced by the journals of certain domains like computer science [10]. Their use also brings into focus some important queries about the nature of peer review and their role in the quality management of scientific publications [11]. Hence, while there is considerable discussion and several case studies on the use of LLMs in academic writing, there is a notable lack of large-scale research investigating the extent of their use and their overall impact on scientific disciplines. Another limitation of prior studies is that they often focus on specific domains or rely on small-sized databases [12], and more generally, in the health sciences.

By evaluating these indicators across a large corpus of health science publications, we aim to provide insights into:

1. The extent of possible LLM application in various fields of Health science.

2. Temporal usage of LLMs and their recent trends including the release of ChatGPT to public domain.

3. Regional differences, academic and research institutes, journal, and agency-specific LLM usage

Before continuing this discussion, it is important to note that merely having-LLM-related linguistic indicators does not necessarily mean the subject in question is directly influenced by LLMs. However, the shifts in the relative frequency of usage of these markers may be utilized as a useful indicator of the impact of LLMs on health sciences writing. This work can be considered as a contribution to the continuously emerging picture of academic publishing in the age of AI.

Study Design

This research is a quantitative, cross-sectional study that aims to identify the extent by which health science articles have incorporated the use of LLMs. The study covers the years 2022 and 2024, enabling comparison of results at two different intervals and, crucially, allowing for a comparison between the pre and post-ChatGPT eras. For this study, we define the pre-ChatGPT era as the period before November 30, 2022, when ChatGPT was publicly released. Specifically, our pre-ChatGPT data comes from January 1 to May 31, 2022. The post-ChatGPT era is defined as the period after ChatGPT's public release, with the data covering January 1 to May 31, 2024. This timeframe allows for a full year of potential ChatGPT influence on academic writing and research practices. To achieve these goals, it was important to delineate clear criteria for the subject’s eligibility for this study to avoid over- or under-inclusion of potential publications and to ensure consistency in the evaluation of the utilization of LLMs in publications from the health sciences. These criteria were used in all twelve health science disciplines under consideration in this research.

Eligibility-Criteria

Inclusion-Criteria

(i) Publication-Type:

No limits were placed on the type of articles that were considered in the analysis, as all those that are included in the Web of Science Core Collection were considered
This broader inclusion facilitates a comprehensive investigation into the use of LLMs across different aspects of scientific writing.

(ii) Time-Frame:

The articles were retrieved from medical journals, and other sources from January 1 to May 31 of 2022 and the same time frame in 2024.
This approach helps avoid the effects of seasonality while still allowing for the year comparison which may otherwise be influenced.

(iii) Language:

Only English-language publications were included in this study, to ensure consistency in keyword analysis, as the LLM-associated keywords are primarily developed in English-language contexts. English remains the predominant language of international scientific communication, allowing for a representative sample of global scientific discourse.
This approach also ensures direct comparability across different countries, institutions, and disciplines without the confounding factor of language differences.
Many widely used LLMs, like ChatGPT, are primarily trained on English data, making their impact more noticeable and easier to detect in English-language publications. While this may limit insights into non-English scientific communities, it allows for a more controlled analysis of LLMs’ effects on scientific writing.

(iv) Indexing:

Publications indexed in Web of Science Core Collection databases were used in the study (SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, ESCI).

(v) Discipline:

Publications were required to be categorized within one of the twelve specified health science disciplines based on the keyword classification system developed for this study.

(vi) Full-text Availability:

Publications were required to have full-text availability for keyword searching.
This criterion is crucial for accurately evaluating the presence of LLM-related keywords.

Exclusion Criteria

(i) Publication Type:

Books and grey literature were excluded from the analysis.
This exclusion helps maintain focus on peer-reviewed research articles and other scholarly publications indexed in the Web of Science.

(ii) Language:

Non-English publications were excluded.
While this may limit the study's global representation, it ensures consistency in keyword analysis.

(iii) Duplicate Publications:

Any duplicate entries in the database were excluded to prevent double-counting.

(iv) Incomplete Metadata:

Publications with incomplete metadata (missing author affiliations, countries, journal titles, or publisher names) were excluded to ensure comprehensive analysis across all study dimensions.

Data Source

The current study utilizes the Web of Science Core Collection as the primary data source due to its extensive indexing and rigorous standards. Several specific databases are used in the study, which are integrated into the Web of Science: the SCI-EXPANDED, SSCI, A&HCI, CPCI–SCIENCE, CPCI–SSH, and ESCI. It ensures the comprehensiveness of coverage of the peer-reviewed literature across health science disciplines.

Disciplines and Time Frame

The Health Science disciplines and associated keywords of the study are summarized in Table 1. The analysis covers publications from the first five months (January 1 to May 31) of 2022, and 2024, allowing for a comparison of trends before and after the public launch of ChatGPT in November 2022, while controlling for potential seasonal variations in publication patterns.

Search Strategy

A search query was developed for each discipline using a combination of Web of Science Categories and Research Area fields. For example, the query for Medicine (General & Internal) was structured as Web of Science Categories=(Medicine, General & Internal) AND Web of Science Categories and Research Area =(General & Internal Medicine).

Table 1

Health Science Disciplines and Associated Keywords
Discipline	Search Keywords
1. Medicine	Medicine OR Medical OR Physician* OR Surgery OR Surgical OR Surgeon*
2. Dentistry	Dentistry OR Dental OR Dentist*
3. Pharmacy	Pharmacy OR Pharmaceutical* OR Pharmacist*
4. Nursing	Nursing OR Nurse*
5. Physical therapy	"Physical therapy" OR Physiotherapy OR "Occupational therapy" or "Physical therapist" OR Physiotherapist OR "Occupational therapist"
6. Public health	Public health OR Epidem*
7. Psychology	Psycholog* OR Psychotherapy OR Psychotherapist
8. Nutrition	Nutrition* OR Diet*
9. Bioinformatics	Bioinformatics or "computational biology" or omics or genomics or proteomics or transcriptomics or metabolomics
10. Health informatics	"Health informatics" OR "medical informatics" OR "clinical informatics" OR "biomedical informatics" OR "e-health" OR "mHealth" OR "digital health" OR "health information"
11. Complementary and Alternative Medicine	"Integrative medicine" or "complementary medicine" or "traditional medicine" or "Holistic Medicine" or "Herbal Medicine" or "Natural Remedy" OR naturopathy or ayurveda or siddha or unani or homeopathy or "Chinese medicine"
12. Veterinary Medicine	Veterinar* OR "Animal health"
Combined	Medicine OR Medical OR PhysicianOR Surgery OR Surgical OR Surgeon OR Dentistry OR Dental OR Dentist* OR Pharmacy OR Pharmaceutical* OR Pharmacist* OR Nursing OR Nurse* OR "Physical therapy" OR Physiotherapy OR "Occupational therapy" or "Physical therapist" OR Physiotherapist OR "Occupational therapist" OR Public health OR Epidem* OR Psycholog* OR Psychotherapy OR Psychotherapist OR Nutrition* OR Diet* OR Bioinformatics or "computational biology" or omics or genomics or proteomics or transcriptomics or metabolomics OR "health informatics" OR "medical informatics" OR "clinical informatics" OR "biomedical informatics" OR "e-health" OR "mHealth" OR "digital health" OR "health information" OR "integrative medicine" or "complementary medicine" or "traditional medicine" or "Holistic Medicine" or "Herbal Medicine" or "Natural Remedy" OR naturopathy or ayurveda or siddha or unani or homeopathy or "Chinese medicine" OR Veterinar* OR "Animal health"
Adjectives & Adverbs	Search Keywords
	Intricate OR meticulous OR meticulously OR commendable OR notable OR pivotal OR invaluable OR noteworthy OR methodically OR strategically
Publication Date (From & To; YYYY/MM/DD)
2022-01-01	2022-05-31
2024-01-01	2024-05-31

LLM Usage Assessment

The study utilizes a keyword-based approach to assess potential LLM usage, based on the method outlined by Gray (2024). Ten keywords identified as indicators of potential LLM usage were selected: intricate, meticulous, meticulously, commendable, notable, pivotal, invaluable, noteworthy, methodically, and strategically [13]. A full-text search was conducted for each discipline and year using these keywords. To account for general trends in language use, a set of neutral control keywords (consider, conclusion, furthermore, relative, technical) were also searched. These control keywords help distinguish between general changes in academic writing style and specific increases potentially attributable to LLM use. The selection of these ten adjectives and adverbs (intricate, meticulous, meticulously, commendable, notable, pivotal, invaluable, noteworthy, methodically, and strategically) as potential indicators of LLM-assisted writing is based on the methodology outlined by Gray (2024). The terms were chosen for their linguistic sophistication and observed higher frequency in known LLM-generated texts compared to typical human-written academic content. They often appear in LLM outputs to provide nuanced descriptions, emphasize importance, or add a layer of academic-style writing. While not exclusively used by LLMs, the combination and increased frequency of these terms can serve as potential markers of LLM involvement in text generation. It's important to note that the presence of these terms alone does not conclusively prove LLM usage, but rather indicates a higher likelihood of LLM assistance when found in atypical frequencies or specific combinations within health science publications.

Data Collection and Analysis

For each discipline, year, and keyword (including control keywords), the total number of publications, the number of publications containing each keyword, and several publications containing any of the LLM-associated keywords were collected. Additional metadata, including author affiliations, countries, journal titles, and publisher names, was extracted for publications containing LLM-associated keywords. The prevalence of potential LLM usage was calculated as the percentage of publications containing LLM-associated keywords relative to the total number of publications. Year-over-year changes in prevalence were calculated for each discipline, and relative changes were computed to allow for direct comparisons.

Statistical Analysis

The study employs both descriptive and inferential statistical methods. Descriptive statistics summarize the prevalence of LLM-associated keywords across various dimensions. Chi-square tests of independence compare the proportion of publications with LLM-associated keywords across years and disciplines.

Comparative Analysis

The study includes comprehensive comparative analyses across multiple dimensions. Disciplines are ranked based on their prevalence of LLM-associated keywords. Institutional analysis identifies and ranks institutions based on their contribution to publications with LLM-associated keywords. Country-level analysis examines the percentage contribution of each country to the total number of publications with LLM-associated keywords (Top Tier: ≥ 90th percentile)
High Tier: 75th to 90th percentile
Upper-Middle Tier: 50th to 75th percentile
Lower-Middle Tier: 25th to 50th percentile
Bottom Tier: < 25th percentile

Journal analysis calculates the percentage of publications with LLM-associated keywords for each journal and ranks them accordingly. Publisher analysis determines the percentage contribution of each publisher to the total number of publications with LLM-associated keywords and ranks publishers based on their contribution. A p-value of < 0.05 was considered as significant

To enhance the accuracy of detecting potential LLM usage, we incorporated a frequency analysis of LLM-related words, complementing our existing keyword-based method. We expanded our list of LLM-associated terms to include 50 adjectives and adverbs commonly found in LLM-generated text (e.g., "comprehensive", "nuanced", "significantly", and "effectively"), based on literature review and analysis of known LLM-generated academic texts. The occurrences of these words were counted for each paper in our dataset and calculated word density (occurrences per 1000 words) to normalize for varying paper lengths. Through a pilot study of 500 papers (250 known to use LLMs and 250 known not to), we established frequency thresholds indicative of potential LLM usage, flagging papers exceeding the 75th percentile of word density in the LLM-used group as high-probability LLM-influenced. We then analyzed changes in word frequency patterns across years (2022, 2024) and between different categories (countries, institutions, journals), employing statistical tests (chi-square and t-tests) to assess the significance of observed differences. This method allows us to capture more subtle influences of LLM tools on academic writing style, providing a complementary measure to our existing approach and a more nuanced picture of LLM influence in scholarly publications.

Ethical Considerations and Limitations

The study uses publicly available bibliometric data and does not involve human subjects, ensuring data privacy. No individual authors or specific papers are identified in the analysis. The methodology, including search strategies and analytical methods, is fully disclosed to ensure reproducibility.

Health science research potentially utilizing LLMs Trends 2022–2024: SEARCH HITS WITHOUT ADDING COMMON ADJECTIVES AND ADVERBS

Table 2

Country Comparison
Country	2022% (Tier)	2024% (Tier)	Change	p-value	Significance
USA	26.850% (Top)	25.393% (Top)	-1.457%	< 0.001	Significant decrease
China	18.853% (Top)	23.467% (Top)	+ 4.614%	< 0.001	Significant increase
England	6.339% (Top)	6.373% (Top)	+ 0.034%	0.715	Not significant
India	5.350% (Top)	5.806% (Top)	+ 0.456%	< 0.001	Significant increase
Germany	5.665% (Top)	5.716% (Top)	+ 0.051%	0.601	Not significant
Italy	4.770% (Top)	4.982% (Top)	+ 0.212%	0.009	Significant increase
Japan	5.045% (Top)	4.881% (Top)	-0.164%	0.043	Significant decrease
Canada	4.207% (Top)	4.161% (Top)	-0.046%	0.509	Not significant
Australia	3.987% (Top)	3.860% (Top)	-0.127%	0.116	Not significant
Spain	3.070% (Top)	3.230% (Top)	+ 0.160%	0.010	Significant increase

Table 3

Publisher Analysis
Publisher	2022 Rank (%)	2024 Rank (%)	Change	p-value	Significance
Springer Nature	2 (12.123%)	1 (18.115%)	+ 5.992%	< 0.001	Significant increase
Elsevier	1 (19.029%)	2 (16.944%)	-2.085%	< 0.001	Significant decrease
Wiley	3 (7.928%)	3 (10.410%)	+ 2.482%	< 0.001	Significant increase
MDPI	4 (7.794%)	4 (7.989%)	+ 0.195%	0.089	Not significant
Frontiers Media SA	5 (5.130%)	5 (4.616%)	-0.514%	< 0.001	Significant decrease
Lippincott Williams & Wilkins	6 (4.067%)	6 (3.991%)	-0.076%	0.453	Not significant
Oxford University Press	7 (2.928%)	7 (3.506%)	+ 0.578%	< 0.001	Significant increase
NATURE portfolio	12 (1.515%)	8 (2.645%)	+ 1.130%	< 0.001	Significant increase
Sage	8 (2.180%)	9 (2.627%)	+ 0.447%	< 0.001	Significant increase
Taylor & Francis	9 (2.148%)	10 (2.312%)	+ 0.164%	0.056	Not significant

Table 4

Journal Analysis
Journal Title	2022%	2024%	Change	P-value
International Journal of Environmental Research and Public Health	1.074%	-	N/A	N/A
PLOS One	0.797%	0.934%	+ 0.137%	< 0.0001
Scientific Reports	0.795%	1.288%	+ 0.493%	< 0.0001
American Journal of Respiratory and Critical Care Medicine	0.795%	0.066%	-0.729%	< 0.0001
Cureus Journal of Medical Science	0.767%	2.155%	+ 1.388%	< 0.0001
International Journal of Molecular Sciences	0.651%	0.937%	+ 0.286%	< 0.0001
Journal of Clinical Medicine	0.499%	0.725%	+ 0.226%	< 0.0001
Frontiers in Oncology	0.420%	0.335%	-0.085%	< 0.0001
Frontiers in Immunology	0.414%	0.484%	+ 0.070%	< 0.0001
Cancers	0.369%	0.412%	+ 0.043%	0.0088

Table 5

Institution Analysis
Institution	2022%	2024%	Change	p-value
Harvard University	2.541%	2.484%	-0.057%	0.1858
University of California System	2.510%	2.414%	-0.096%	0.0246*
University of London	1.799%	1.801%	+ 0.002%	0.9636
Harvard Medical School	1.589%	1.596%	+ 0.007%	0.8741
University of Texas System	1.483%	1.473%	-0.010%	0.8175
Egyptian Knowledge Bank EKB	1.332%	1.389%	+ 0.057%	0.2278
University System of Ohio	1.308%	1.305%	-0.003%	0.9496
Chinese Academy of Sciences	1.226%	1.410%	+ 0.184%	0.0001*
University of Toronto	1.109%	1.128%	+ 0.019%	0.6901
Johns Hopkins University	1.074%	1.034%	-0.040%	0.3990
• Statistically significant at p < 0.05

Prevalence of potential LLM use in health science writing 2022–2024: SEARCH HITS WITH ADDING COMMON ADJECTIVES AND ADVERBS

Table 6

Country Comparison
Country	2022% (Tier)	2024% (Tier)	Change (2022–2024)	p-Value	Significance
China	20.571% (Top)	33.044% (Top)	+ 12.473%	< 0.001	Significant increase
USA	29.474% (Top)	19.124% (Top)	-10.350%	< 0.001	Significant decrease
India	7.455% (Top)	9.276% (Top)	+ 1.821%	< 0.001	Significant increase
Italy	7.224% (Top)	5.236% (Top)	-1.988%	< 0.001	Significant decrease
England	6.968% (Top)	4.873% (Top)	-2.095%	< 0.001	Significant decrease
Germany	6.686% (Top)	4.531% (Top)	-2.155%	< 0.001	Significant decrease
Saudi Arabia	2.203% (High)	4.357% (Top)	+ 2.154%	< 0.001	Significant increase
South Korea	2.613% (High)	3.780% (Top)	+ 1.167%	< 0.001	Significant increase
Japan	4.880% (Top)	3.402% (Top)	-1.478%	< 0.001	Significant decrease
Canada	4.803% (Top)	3.111% (Top)	-1.692%	< 0.001	Significant decrease
Iran	3.202% (Top)	2.901% (High)	-0.301%	0.028	Significant decrease
Australia	4.253% (Top)	2.815% (High)	-1.438%	< 0.001	Significant decrease
Spain	3.625% (Top)	2.794% (High)	-0.831%	< 0.001	Significant decrease
Brazil	2.434% (High)	2.248% (High)	-0.186%	0.356	Not significant
France	3.433% (Top)	2.248% (High)	-1.185%	< 0.001	Significant decrease
Turkey	1.947% (High)	2.212% (High)	+ 0.265%	0.118	Not significant
Egypt	1.511% (High)	2.043% (High)	+ 0.532%	< 0.001	Significant increase
Taiwan	1.524% (High)	1.956% (High)	+ 0.432%	0.003	Significant increase
Pakistan	1.652% (High)	1.931% (High)	+ 0.279%	0.101	Not significant
The Netherlands	2.972% (Top)	1.829% (High)	-1.143%	< 0.001	Significant decrease

Table 7

Publisher Analysis
Publisher	2022 Rank (%)	2024 Rank (%)	Change	p-value	Significance
Springer Nature	2 (11.438%)	1 (20.616%)	+ 9.178%	< 0.001	Significant increase
Elsevier	1 (17.612%)	2 (15.580%)	-2.032%	0.0002	Significant decrease
MDPI	3 (9.863%)	3 (12.882%)	+ 3.019%	< 0.001	Significant increase
Wiley	5 (6.481%)	4 (9.904%)	+ 3.423%	< 0.001	Significant increase
Frontiers Media SA	4 (7.352%)	5 (6.845%)	-0.507%	0.2045	Not significant
NATURE Portfolio	9 (2.536%)	6 (2.753%)	+ 0.217%	0.3673	Not significant
American Chemical Society	− (1.345%)	7 (2.329%)	+ 0.984%	< 0.001	Significant increase
Oxford University Press	8 (2.549%)	8 (2.171%)	-0.378%	0.0627	Not significant
Taylor & Francis	− (2.088%)	9 (2.099%)	+ 0.011%	0.9328	Not significant
American Physiological Society	6 (3.382%)	− (0.965%)	-2.417%	< 0.001	Significant decrease

Table 8

Journal Analysis
Journal Title	2022%	2024%	Change	P-value
International Journal of Molecular Sciences	1.294%	1.916%	+ 0.622%	< 0.0001
Scientific Reports	1.050%	1.297%	+ 0.247%	0.0506
Journal of Applied Physiology	0.973%	0.087%	-0.886%	< 0.0001
Cancers	0.948%	0.838%	-0.110%	0.1746
Cureus Journal of Medical Science	0.909%	3.949%	+ 3.040%	< 0.0001
Frontiers in Immunology	0.897%	1.068%	+ 0.171%	0.0506
PLOS One	0.871%	0.659%	-0.212%	0.0506
International Journal of Environmental Research and Public Health	0.653%	-	N/A	N/A
Journal of Neurophysiology	0.602%	0.143%	-0.459%	< 0.0001
Frontiers in Pharmacology	0.589%	0.506%	-0.083%	0.2250

Table 9

Institution Analysis
Institution	2022%	2024%	Change	p-value Significance p < 0.05
Harvard University	2.933%	2.268%	-0.665%	0.000116
University of California System	2.920%	1.956%	-0.964%	< 0.00001
University of Texas System	2.203%	1.180%	-1.023%	< 0.00001
Harvard Medical School	2.011%	1.149%	-0.862%	< 0.00001
Chinese Academy of Sciences	1.973%	2.268%	+ 0.295%	0.054
University of London	1.947%	1.262%	-0.685%	< 0.00001
University System of Ohio	1.614%	0.981%	-0.633%	< 0.00001
Pennsylvania Commonwealth System of Higher Education	1.435%	0.817%	-0.618%	< 0.00001
Egyptian Knowledge Bank EKB	1.422%	1.900%	+ 0.478%	0.000299
Helmholtz Association	1.307%	< 0.506%	>-0.801%	< 0.00001
Johns Hopkins University	1.242%	0.771%	-0.471%	0.000025
Institut National de la Sante et de la Recherche Medicale INSERM	1.204%	0.752%	-0.452%	0.000049
State University System of Florida	1.204%	0.720%	-0.484%	0.000014
Universite Paris Cite	1.153%	0.745%	-0.408%	0.000183
Yale University	1.127%	0.685%	-0.442%	0.000049
Fudan University	1.050%	1.251%	+ 0.201%	0.065717
University of Pennsylvania	1.050%	0.827%	-0.223%	0.033598
University of Toronto	1.025%	0.572%	-0.453%	< 0.00001
Centre National de la Recherche Scientifique CNRS	0.999%	0.868%	-0.131%	0.199638
Brigham Women's Hospital	0.973%	0.557%	-0.416%	0.000029

Analysis of LLM Research Trends 2022–2024

The prevalence of using LLM-related adverbs and adjectives in research publications across various parameters, comparing data from 2022 (pre-ChatGPT) and 2024 (post-ChatGPT). The analysis covers four main areas: Country contributions, publisher involvement, journal focus, and institutional participation.

Country Comparison

The country comparisons showed significant shifts in the global landscape of LLM research. Without common adjectives and adverbs, the USA maintained its top position but showed a significant decrease (-1.457%), whilst China showed a substantial increase (+ 4.614%). India also increased significantly. The changes were even more dramatic when common adjectives and adverbs were included. It highlights significant shifts in the global distribution of LLM research output, with countries categorized into tiers based on their publication percentages. China demonstrates a remarkable ascent, increasing its share from 20.571–33.044%, solidifying its position at the top of the Top Tier. Conversely, while maintaining its Top Tier status, the USA shows a substantial decline from 29.474–19.124%. Other notable risers include India, which strengthened its Top Tier position with an increase from 7.455–9.276%, and Saudi Arabia and South Korea, both of which advanced from the High Tier to the Top Tier. The table also notes a general trend of decreased % representation from several Western countries such as Italy, England, Germany, Canada, Australia, Spain, France, and the Netherlands, with some dropping from the Top Tier to the High Tier. However, it's important to note that despite these decreases, many countries still maintain positions in the Top or High tier. Countries in the Middle East and Asia, such as Iran, Egypt, and Taiwan, either maintained their positions or showed growth, further emphasizing the geographical shift in LLM research focus. Interestingly, some countries like Brazil, Turkey, and Pakistan showed changes that were not statistically significant, suggesting a level of stability in their research output relative to global production. Overall, this data illustrates a swiftly changing global LLM research landscape, highlighting a significant increase in contributions from Asian and Middle Eastern countries. At the same time, traditionally dominant Western nations experience a relative decline in their publication share (Tables 2 and 6).

Publisher Analysis

The publisher analysis shows interesting trends in the academic publishing landscape for LLM research. Without common adjectives and adverbs, Springer Nature showed a significant increase (+ 5.992%), moving from second to first place, while Elsevier experienced a significant decrease (-2.085%). Wiley also showed substantial growth. When common adjectives and adverbs were included, these trends were amplified. Springer Nature's share increased by 9.178%, while Elsevier's decreased by 2.032%. MDPI and Wiley also showed significant increases. This suggests that certain publishers, particularly Springer Nature, are becoming increasingly dominant in publishing LLM-written research articles, especially studies using common LLM terminology (Tables 3 and 7).

Journal Analysis

The journal analysis reveals which publications are at the forefront of LLM research. Without common adjectives and adverbs, the Cureus Journal of Medical Science showed the most dramatic increase (+ 1.388%), while the American Journal of Respiratory and Critical Care Medicine saw the largest decrease (-0.729%). When common adjectives and adverbs were included, the Cureus Journal of Medical Science’s increase was even more pronounced (+ 3.040%), and the International Journal of Molecular Sciences also showed a significant increase. This suggests that medical and molecular science journals are increasingly publishing LLM-related papers, particularly studies using common LLM terminology (Tables 4 and 8).

Institution Analysis

The institutional analysis provides insights into which academic and research organizations are leading in LLM research. Without including common adjectives and adverbs, the changes were relatively minor, with the Chinese Academy of Sciences showing the most notable increase (+ 0.184%). However, incorporating these common terms revealed much more pronounced shifts. Many leading Western institutions, such as Harvard University and the University of California System, showed significant declines, whereas Chinese institutions like Shanghai Jiao Tong University saw significant increases. This pattern aligns with the country-level analysis, suggesting a shift in LLM research leadership toward Asian institutions, especially those in China (Tables 5 and 9).

This study clearly shows a growing trend of LLM-related research in Asia, especially in China, at both the country and institutional levels. It also shows the rising prominence of certain publishers and journals in the field. By including common LLM-related adjectives and adverbs in the analysis, these trends are amplified, suggesting that specific LLM terminology is more commonly used in emerging research hubs.

Overview of Findings

This work offers a comprehensive analysis of the utilization of LLMs and the predictions of the usage trend in health science articles from 1 January until 31 May 2022 and in 2024. Examining the linguistic features that distinguish text generated by LLMs, we have identified important changes in the scientific publishing landscape following OpenAI’s release of the refined and widely accessible version of ChatGPT. This trend is in concord with a recent meta-analysis of 950,965 papers from arXiv, bioRxiv, and Nature portfolio journals. Using their sample it was realized that Computer Science papers demonstrated the highest and fastest increase in the use of LLM with an estimated 17.5%. In abstracts, 5 sentences out of all were found to have poly-semanticity while in conclusion and introduction, 15% and 13% of all sentences were polysemantic; 3% in the case of introductions to be LLM-modified by February 2024 [19].

Geographical Shifts in LLM Adoption

Decline in English-speaking and EU countries:

The data shows a consistent decrease in the share of LLM-related publications from major English-speaking countries and EU member states. The USA, England, Canada, Australia, Germany, and France all showed statistically significant decreases in their publication shares.

This trend could be attributed to several factors:

Increased global competition: As more countries invest in AI and LLM research, the relative share of traditionally dominant countries may naturally decrease [17].
Shifting research priorities: English-speaking and EU countries might be focusing on other areas of AI or technology, leading to a relative decrease in LLM-assisted health science publications18].
Resource allocation: There may be differences in how research funding is allocated in these countries compared to the rising nations [19].
Policy and regulatory environment: Stricter regulations on AI and data usage in certain Western countries may be affecting the speed of research [23].

Rise of other countries:

Conversely, we note significant increases in LLM-related publications from countries like China, India, Saudi Arabia, South Korea, and Egypt. This trend aligns with several observations in recent literature:

China's AI ambitions: China has made substantial investments in AI research and development, with a specific focus on language technologies [24]. The substantial rise in China's share (from 20.571–33.044%) reflects these efforts.
Emerging tech hubs: Countries like India and South Korea have been establishing themselves as important tech hubs, with an increasing focus on AI and natural language processing [25].
Middle Eastern investments: The significant increases seen in Saudi Arabia and Egypt align with the growing investments in AI and technology in the Middle East [26].
Global collaboration: The growth of international research collaborations may be contributing to the enhanced representation of these countries in LLM research [27].

It is important to note that while the relative shares are changing, this does not necessarily indicate a decrease in the absolute number of publications from English-speaking and EU countries. Rather, it suggests a global diversification of LLM research, with rapid growth in previously underrepresented regions.

These shifts could have important implications for the future development of LLM technologies, potentially leading to more diverse language models that better represent a global perspective. However, it also raises questions about potential divergences in research focus, ethical considerations, and applications of LLMs across different regions.

Changes in Publisher Landscape

The analysis of the publishers shows that there is a transformation of the academic publishing environment due to the introduction of LLM technologies. Such a scenario is evident when comparing Springer Nature’s LLM-associated publications, which have increased dramatically in recent years, to that of Elsevier, which has decreased. It may be conceivably due to variations in the editorial policies, authors’ guidelines, or the kind of studies that are published individually by each publisher. These figures showed that the increase is exhibited by several different publishers – MDPI, Wiley, etc., which proves that LLMs’ effect is not isolated to a single publisher. The fact that the use of LLMs has become almost universal in scientific publishing supports this prognostication [16].

Journal-Specific Trends

The review of the health science journal will give an understanding of which of the fields within the health sciences are the most advanced in LLM technology adoption. The sudden increase in the number of articles published in Cureus Journal of Medical Science, International Journal of Molecular Sciences, and other such journals suggests that some areas of medical research are more likely to require or benefit from LLM assistance. The quantities have been reduced in some of the journals like the American Journal of Respiratory and Critical Care Medicine warrants further contemplation in an endeavor to understand field variations of LLM. It could be a result of methodological discrepancies in the studies conducted, the type of data used, or the ethical perspective distinctive of diverse divisions of medicine.

Institutional Dynamics

The examination of LLM from the institutional view exhibits that the adoption of the LLM is not a simple scenario in every academic organization. Just as at the country level, there is a decline of LLM-associated publications from the elite Western institutions, Harvard University, and the University of California System while there is growth from Chinese institutions.

Such a change may be attributed to disparities in the institutional policies that allow the use of AI in research, the availability and knowledge of LLM tools among scholars, or disparities in the acultural acceptance of AI in academia. Institutions such as the Egyptian Knowledge Bank EKB are now emerging foreshadowing how LLMs could also generalize high-quality research output across a wider range of institutions globally.

Implications for Research Integrity and Quality

The dramatic surge in LLM-associated publications raises important questions about research quality and credibility. Although LLMs may boost productivity and potentially enhance prose clarity, they can also introduce or sustain various types of discrepancies. These include information quality issues, where LLMs may generate plausible-sounding but factually incorrect information, a phenomenon known as "hallucination" [28]. LLMs can perpetuate or amplify existing biases in their training data, potentially reinforcing outdated perspectives in fields like medicine [29]. Citation errors are another concern, with studies showing LLMs can generate non-existent citations or misattribute information [30]. Methodological inconsistencies may arise if LLMs suggest inappropriate research designs [31]. Additionally, widespread LLM use could lead to a homogenization of academic writing style, potentially obscuring the diversity of voices in scientific discourse [32]. These discrepancies could create inequalities in research practice and output due to differential rates of LLM adoption across countries, publishers, and institutions, potentially leading to new forms of bias in published literature where LLM use becomes a predictor of research acceptance or citation rates.

Ethical Considerations and Transparency

Altogether, the study herein points to the necessity of regulations and norms as to the application of LLMs in science and publications. The marked frequency of LLM-associated language in recent years, especially in some geographic areas and publications, implies that several scholars may be employing these instruments without actively reporting their practice.

Such practice hinders peer review, replication, and the credibility of the published papers and other scientific outputs. This is in concordance with issues of van Dis and others on the lack of definite policies in disclosure and ethical approaches towards the usage of AI in research [18]. These results suggest the need for clear and precise guidelines on incorporating LLMs into writing and academic work, similar to the principles outlined by Smith et al., (2024) in their ‘Ten simple rules for using large language models in science’. They focus on adherence to journal rules, coverage of such risks as plagiarism, maintaining confidentiality, and checking the accuracy of work produced by the LLM. The ethical issue of applying LLM in scientific writing is discussed in further detail in an editorial by Koller et al in the NEJM AI [20]. They are in favor of permitting and even promoting the use of LLMs in scientific submissions proviso that authors fully assume responsibility for the inputs and give proper credit to LLM usage. This approach seeks to take advantage of LLMs in the improvement of the quality of presented papers within the scientific communities and avail equal opportunities in sharing knowledge with the broader society. That being the case, the question becomes how to optimize the use of LLMs while ensuring that the quality of the final papers is not compromised. More research work in the forthcoming years should be directed at refining the approaches to identifying LLM use, furthering the analysis of its effects, and finding ways to maximize the positive effects of these systems while minimizing their negative effects [33].

Limitations and Future Directions

Although the present study has offered significant insights into the consumption patterns of LLMs it is imperative to note a few limitations of the study. In using the linguistic markers as indicators of LLM use as has been done while following previous studies, there is the possibility of including false positives thus, not capturing all forms of LLM use. This approach has certain limitations, and future research could benefit from more direct methods to identify LLM-generated text, such as utilizing advanced natural processing techniques and analyzing larger text fragments. However, the current study, which relies on data from the Web of Science, only reflects trends in English-language literature. This may not accurately represent global trends, particularly in developing countries where English publications are minimal. Expanding the study to include other languages and databases would provide valuable insights into the global diffusion of LLMs in research

The present study was conducted over a relatively short time frame, and captures an emerging trend in the use of LLM in health science publications. The study highlights significant shifts in geographic distribution, as well as changes in publisher, and institution/journal-level participation. These findings have strong implications for research standards, credibility, and the future of communicating science. Rapid progress with LLMs raises questions about their proper use and utility in science and thus, there is a need to establish clearer ethical guidelines for integrating these systems into the research practices, ensure their application is reported transparently and comprehensively, and rigorously evaluate their impact on the quality and credibility of scientific publications. As such, subsequent studies need to focus on elaborating more selective means of LLM identification, as well as discern the myriad effects on the quality of research, and expose the opportunities of these technologies along with risks.

Author Contributions:

Conceptualization: A.M., A.G.A., Y.K., T.S.S.

Brown TB et al (2020) Language Models are Few-Shot Learners. arXiv preprint arXiv:2005.14165
Gruda D (2024) Three Ways ChatGPT Helps Me in My Academic Writing. Accessed August 30, 2024. https://doi.org/10.1038/d41586-024-01042-3
Ruksakulpiwat S, Kumar A, Ajibade A (2023) Using ChatGPT in Medical Research: Current Status and Future Directions. J Multidiscip Healthc 16:1513–1520 Published 2023 May 30. 10.2147/JMDH.S413470
van Dis EAM, Bollen J, Zuidema W, van Rooij R, Bockting CL (2023) ChatGPT: five priorities for research. Nature 614(7947):224–226. https://doi.org/10.1038/d41586-023-00288-7
Prillaman M (2024) Is ChatGPT making scientists hyper-productive? The highs and lows of using AI. Nature 627(8002):16–17. https://doi.org/10.1038/d41586-024-00592-w
Loth A, Kappes M, Pahl M (2024) Blessing or curse? A survey on the Impact of Generative AI on Fake News. ArXiv. /abs/2404.03021
Zhao WX, Zhou K, Li J, Tang T, Wang X, Hou Y, Min Y, Zhang B, Zhang J, Dong Z, Du Y, Yang C, Chen Y, Chen Z, Jiang J, Ren R, Li Y, Tang X, Liu Z, Wen J (2023) A Survey of Large Language Models. ArXiv. /abs/2303.18223
Buchanan J, Hill S, Shapoval O (2023) ChatGPT Hallucinates Non-existent Citations: Evidence from Economics. Am Econ. https://doi.org/10.1177_05694345231218454
Graf C, Deakin L, Docking M, Jones J, Joshua S, McKerahan T, Ottmar M, Stevens A, Wates E, Wyatt D (2014) Best practice guidelines on publishing ethics: A publisher's perspective, 2nd edition. International Journal of Clinical Practice, 68(12), 1410–1428. https://doi.org/10.1111/ijcp.12557
Liang W, Izzo Z, Zhang Y, Lepp H, Cao H, Zhao X, Chen L, Ye H, Liu S, Huang Z, McFarland DA, Zou JY (2024) Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews. ArXiv. /abs/2403.07183
Singh Chawla D (2024) Is ChatGPT Corrupting Peer Review? Telltale Words Hint at AI Use. Accessed August 30, 2024. https://doi.org/10.1038/d41586-024-01051-2
Akram A (2024) Quantitative Analysis of AI-Generated Texts in Academic Research: A Study of AI Presence in Arxiv Submissions using AI Detection Tool. ArXiv. /abs/2403.13812
Gray A (2024) ChatGPT contamination: Estimating the prevalence of LLMs in the scholarly literature. ArXiv. /abs/2403.16887
Klinger J, Mateos-Garcia J, Stathoulopoulos K (2018) Deep learning, deep change? Mapping the development of the Artificial Intelligence General Purpose Technology. arXiv preprint arXiv:1808.06355
Zhang, D., Mishra, S., Brynjolfsson, E., Etchemendy, J., Ganguli, D., Grosz, B., …Perrault, R. (2021). The AI index 2021 annual report. arXiv preprint arXiv:2103.06312
Hill-Yardin EL, Hutchinson MR, Laycock R, Spencer SJ (2023) A Chat(GPT) about the future of scientific publishing. Brain Behav Immun 110:152–154. https://doi.org/10.1016/j.bbi.2023.02.022
Buchanan J, Hill S, Shapoval O (2023) ChatGPT Hallucinates Non-existent Citations: Evidence from Economics. Am Econ. https://doi.org/10.1177_05694345231218454
van Dis EAM, Bollen J, Zuidema W, van Rooij R, Bockting CL (2023) ChatGPT: five priorities for research. Nature 614(7947):224–226. https://doi.org/10.1038/d41586-023-00288-7
Liang, W., Zhang, Y., Wu, Z., Lepp, H., Ji, W., Zhao, X., … Zou, J. Y. (2024). Mapping the Increasing Use of LLMs in Scientific Papers. arXiv preprint arXiv:2404.01268
Smith, G. R., Bello, C., Bialic-Murphy, L., Clark, E., Delavaux, C. S., Fournier de Lauriere, C., … Crowther, T. W. (2024). Ten simple rules for using large language models in science, version 1.0. PLoS Computational Biology, 20(1), e1011767
Koller D, Beam A, Manrai A, Ashley E, Liu X, Gichoya J, Kohane I (2023) Why We Support and Encourage the Use of Large Language Models in NEJM AI Submissions. NEJM AI, 1(1)
O’Meara S (2019) Will China lead the world in AI by 2030? Nature 572(7770):427–428. https://doi.org/10.1038/d41586-019-02360-7
Cath C, Wachter S, Mittelstadt B, Taddeo M, Floridi L (2018) Artificial Intelligence and the 'Good Society': the US, EU, and UK approach. Sci Eng Ethics 24(2):505–528. https://doi.org/10.1007/s11948-017-9901-7
Lee KF (2018) AI superpowers: China, Silicon Valley, and the new world order. Houghton Mifflin Harcourt
Shivaram Kalyanakrishnan RA, Panicker S, Natarajan, Rao S (2018) Opportunities and Challenges for Artificial Intelligence in India. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (AIES '18). Association for Computing Machinery, New York, NY, USA, 164–170. https://doi.org/10.1145/3278721.3278738
Sarkin JJ, Sotoudehfar S (2024) Artificial intelligence and arms races in the Middle East: the evolution of technology and its implications for regional and international security. De?F Secur Anal 40:97–119
Wagner CS, Whetsell TA, Mukherjee S (2018) International Research Collaboration: Novelty, Conventionality, and Atypicality in Knowledge Recombination. ArXiv. https://doi.org/10.1016/j.respol.2019.01.002
Buchanan J, Stephen Hill, and, Shapoval O (2023) ChatGPT Hallucinates Non-existent Citations: Evidence from Economics. The American Economist, Accessed August 30, 2024. https://doi.org/10.1177_05694345231218454
Bender EM, Gebru T, McMillan-Major A, Shmitchell S (2021) On the dangers of stochastic parrots. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3442188.3445922
Else H (2023) Abstracts Written by ChatGPT Fool Scientists. Accessed August 30, 2024. https://doi.org/10.1038/d41586-023-00056-7
Weissgerber T, Riedel N, Kilicoglu H, Labbé C, Eckmann P, Riet GT, Byrne J et al (2020) Automated Screening of COVID-19 Preprints: Can We Help Authors to Improve Transparency and Reproducibility? Nature Medicine 27, no. 1 : 6–7. Accessed August 30, 2024. https://doi.org/10.1038/s41591-020-01203-7
Ross-Hellauer, T., Tennant, J., Banelytė, V., Görögh, E., Luzi, D., Kraker, P., …Vignoli, M. (2020). Ten simple rules for innovative dissemination of research. PLOS Computational Biology, 16(4), e1007704. https://doi.org/10.1371/journal.pcbi.1007704
Kaneda Y (2024) Bridging Tradition and Innovation: The Hippocratic Oath and ChatGPT in Modern Healthcare. Iran J Public Health 53(7):1689–1690. 10.18502/ijph.v53i7.16064

The authors declare no competing interests.

Download PDF

Version 1

posted

You are reading this latest preprint version

Prevalence and Usage Trends of Large Language Models in Health Science Publications: A Cross-sectional Analysis

Status:

Version 1

Abstract

Figures

Introduction

Methodology

Study Design

Eligibility-Criteria

Inclusion-Criteria

Exclusion Criteria

Data Source

Disciplines and Time Frame

Search Strategy

LLM Usage Assessment

Data Collection and Analysis

Statistical Analysis

Ethical Considerations and Limitations

Results

Analysis of LLM Research Trends 2022–2024

Country Comparison

Publisher Analysis

Journal Analysis

Institution Analysis

Discussion

Overview of Findings

Geographical Shifts in LLM Adoption

Changes in Publisher Landscape

Journal-Specific Trends

Institutional Dynamics

Implications for Research Integrity and Quality

Ethical Considerations and Transparency

Limitations and Future Directions

Conclusion

Declarations

Author Contributions:

References

Additional Declarations

Status:

Version 1