Performance of GPT-4 and mainstream Chinese Large Language Models on the Chinese Postgraduate Examination dataset: Potential for AI-assisted Traditional Chinese Medicine

doi:10.21203/rs.3.rs-4392855/v1

Download PDF

Article

Performance of GPT-4 and mainstream Chinese Large Language Models on the Chinese Postgraduate Examination dataset: Potential for AI-assisted Traditional Chinese Medicine

https://doi.org/10.21203/rs.3.rs-4392855/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

ChatGPT is a well-known example of a Large Language Models(LLMs) that has performed notably well in the medical licensing exams of many countries. Tradition Chinese Medicine(TCM) has gained increasing attention and concern from the international medical community. In China, the medical master’s degree exam carries more weight than the licensing exam due to the multi-level doctor training system. However, the current study lacks an evaluation of the performance of various LLMs in TCM postgraduate examinations. Consequently, we created a test dataset of LLMs using postgraduate examination questions to comprehensively evaluate the performance of GPT-4 and mainstream Chinese LLMs in responding to knowledge and clinical inquiries about TCM. Besides computing the exam scores and presenting LLM's performance on various subjects, we evaluated the output responses based on three qualitative metrics: logical reasoning, and the ability to use internal and external information. The results indicated that Ernie Bot and ChatGLM's expertise in TCM surpassed the passing threshold for the postgraduate selection examination, showcasing their enormous potential as TCM support tools.

Health sciences/Medical research/Translational research

Biological sciences/Computational biology and bioinformatics/Databases

Large Language Models (LLMs)

Traditional Chinese Medicine

ERNIE Bot

ChatGLM

SparkDesk

GPT-4

Physicians in most high-income countries undergo similar levels of educational training and share a homogeneous academic background¹. In response to the demand for healthcare, China has developed a complex medical education system aimed at increasing the number of physicians. Furthermore, the criteria for graduation and employment of medical doctors vary. Doctors with lower levels of education mainly practice in townships and rural areas in China, while those with higher levels of education are more likely to work in urban areas¹. Within the current system, there is a 3-year junior college medical program, a 5-year Bachelor of Medicine degree program, a "5 + 3" Master of Medicine degree program, and an 8-year Doctor of Medicine program². The medical profession has established degree levels to meet the demand for professional healthcare and ensure the provision of basic and primary medical services to the public³. Despite undergoing multiple reforms and advancements, there remains a significant gap between the medical education system and elite medical education.

A growing number of undergraduate students in China are now pursuing advanced degrees like master's or doctoral programs². They work hard to get higher degrees and educational achievements to improve their job prospects. In the past three years, over 4 million candidates have taken postgraduate exams in China annually. Recent data on postgraduate enrollment showed medicine is among the top five disciplines in terms of enrollment size, and it had the highest growth rate in 2021⁴. As of the end of 2020, 59.5% of the physicians in the workforce had a bachelor's degree or higher.

Due to the current practitioner law and the multi-level medical doctor training system in China, the medical master’s degree examination carries higher importance in the selection process than the Chinese licensing examination⁵. Existing studies have primarily focused on qualifying exams in medicine, with little attention given to the performance of Large Language Models (LLMs) in selective exams for medical schools. Our objective was to evaluate the performance of mainstream LLMs currently available in answering the Chinese Postgraduate Examination for Traditional Chinese Medicine (TCM). This examination offers several advantages: 1) Comprehensive coverage of diverse topics including vital subjects for learners. 2) Authored by field experts to guarantee credibility. 3) Objectively truthful, accurate, and standardized content. 4) Well-defined anchor value (i.e., the passing score). In particular, we use the entire question set for assessment, enabling a more realistic simulation of the actual examination.

Significant advancements have been observed in Artificial Intelligence (AI) modeling in recent years, leading to the rapid development of LLMs technology. Particularly noteworthy is the emergence of products like ChatGPT (OpenAI, 2022), signaling a new era for the deployment of general-domain AI⁶. Currently, LLMs have demonstrated promising applications in various areas of medical research, including diagnostics⁷, medical image analysis⁸, medical writing⁹, and personalized medicine¹⁰. The Medical Licensing Examination is commonly used to assess the performance of LLMs in the medical field due to its high standardization, regulation, and comprehensive coverage of various subjects. Multiple studies have systematically evaluated ChatGPT's performance on standardized tests across various languages. Notably, it has demonstrated excellent performance on assessments such as the United States Medical Licensing Examination (USMLE)^11–13, the Japanese Medical Licensing Examination (JMLE)¹⁴, the Saudi Medical Licensing Examination (SMLE)¹⁵, the Polish medical specialization licensing exam (PES)¹⁶ and Taiwan's medical licensing exams¹⁷. However, over the past five years of the Chinese National Medical Licensing Examination (NMLE), ChatGPT scores have consistently fallen below the passing threshold. The main reason is that ChatGPT's training data is mainly in English, with only a small amount in Chinese. Additionally, ChatGPT may face challenges in accurately comprehending healthcare policies within non-English speaking nations^18,19. Cai et al. (2023) introduced a comprehensive benchmark for the Chinese medical domain and evaluated both general-domain and medical LLMs in Chinese. The findings suggest that general-domain LLMs possess substantial medical knowledge and may perform better than medical LLMs²⁰.

TCM represents the culmination of centuries of invaluable experiences gained by the Chinese populace in their daily lives, production activities, and medical practices. This system has exerted a significant impact on healthcare, disease prevention, and treatment globally. Given TCM's remarkable efficacy in disease management, it has gained increasing attention and concern from the international medical community. TCM has now spread to 196 countries and regions, with acupuncture accepted for application in 113 WHO member states. As the rapid promotion and application of LLMs in TCM, it is becoming increasingly crucial and urgent to systematically evaluate the reliability of their answers to TCM questions. TCM theories have unique scientific implications, requiring LLMs to have advanced information processing and comprehension capabilities in the Chinese language environment. Therefore, both quantitative and qualitative feedback on the performance of LLMs are necessary to explore their potential applications and perspectives in academic and medical practice within TCM laying the groundwork for future research on how generative AI can improve TCM healthcare.

To evaluate the performance of GPT-4 and mainstream Chinese LLMs in answering questions related to the Chinese Postgraduate Examination for TCM, in addition to the test scores, qualitative analyses were conducted focusing on three aspects: logical reasoning ability, and ability to use internal and external information. The latter two dimensions encompassed responses that incorporated information relevant to the question or incorporated external information beyond its scope. Our objective was to evaluate the reliability of the LLMs in responding to knowledge and clinical inquiries about TCM. Additionally, we aimed to explore its potential applications in the TCM field and provide insights into its current practical value within the industry.

Accuracy rate and scoring performance

The passing threshold for the examination was set at 117. Ernie Bot, the highest achiever, attained a score of 146 by correctly answering 83 questions (50.30%). ChatGLM also successfully passed the exam with a score of 136.5, demonstrating an accuracy of 46.67%. However, SparkDesk and GPT-4 did not pass the postgraduate examination, scoring 86 and 112 points, respectively.

Performance on different subjects

Table 1 summarizes the test results for each subject. Statistical significance was observed in the performance differences across various subjects among Ernie Bot, ChatGLM, and GPT-4. All models involved in the test exhibited the highest accuracy rates in questions related to the medical humanistic spirit.

Table 1

The performance of LLMs on different subjects
	Number of questions	Ernie Bot correct n(%)	ChatGLM correct n(%)	SparkDesk correct n(%)	GPT-4 correct n(%)
Overall	n = 165	83(50.3)	77(46.7)	58(35.2)	62(37.6)
Basic Theory of TCM	n = 23	8(34.8)	12(52.2)	8(34.8)	10(43.5)
Diagnostics of TCM	n = 23	8(34.8)	9(39.1)	8(34.8)	10(43.5)
Pharmacology of TCM	n = 23	14(60.9)	9(39.1)	6(26.1)	5(21.7)
Formulas of TCM	n = 23	13(56.5)	7(30.4)	9(39.1)	4(17.4)
Internal Medicine of TCM	n = 43	26(60.5)	26(60.5)	14(32.6)	20(46.5)
Acupuncture and Moxibustion	n = 22	7(31.8)	7(31.8)	7(31.8)	5(22.7)
Medical Humanistic Spirit	n = 8	7(87.5)	7(87.5)	6(75)	8(100)
P		0.020	0.026	0.340	< .001

Qualitative analysis of LLM’s response quality

The performance of LLMs was evaluated using three qualitative indicators: logical reasoning, ability to use internal information, and ability to use external information (Table 2).

Logical reasoning: ChatGLM and GPT-4 could provide logical explanations for every answer selection. However, some of the responses from Ernie Bot and SparkDesk lacked logical reasoning. In the case of Ernie Bot, the presence or absence of logical reasoning did not have a significant effect on the accuracy rate of response (P = .12). However, a significant effect was observed on the presence of logical reasoning in SparkDesk (P < .001), suggesting that logical reasoning may impact correctness rates.

Ability to use internal information: Both ChatGLM and GPT-4 utilized internal information in all question explanations. In contrast, SparkDesk provided significantly less internal information in its parses compared to the other three models. The presence or absence of internal information in the response significantly impacted the accuracy rate for SparkDesk ( P < .001 ).

Ability to use external information: For Ernie Bot, 75.9% (63/83) of correct responses and 80.5% (66/82) of incorrect responses presented external information (difference of 4.6%; P = .30 ). Similarly, for ChatGLM, 64.9% (50/77) of correct responses and 71.6% (63/88) of incorrect responses contained external information (difference of 6.7%; P = .23 ). However, in SparkDesk, only 8.6% (5/58) of correct responses and 32.7% (35/107) of incorrect responses presented external information (difference of 24.1%; P < .001 ). There was a significant effect on the accuracy rate for SparkDesk depending on the presence of external information. For GPT-4, the difference was 7.1% (P = .21).

Among the reasons for incorrect responses in the four LLMs, informational errors were the most common, followed by instances where both logical and informational errors occurred at the same time (Table 3).

Table 2

Performance of LLMs' responses in evaluation metrics
	Ernie Bot				ChatGLM
	Overall n = 165,n(%)	Correct n = 83,n(%)	Incorrect n = 82,n(%)	P	Overall n = 165,n(%)	Correct n = 77,n(%)	Incorrect n = 88,n(%)	P
Logical reasoning
YES	162(98.2)	83(100)	79(96.3)	0.120	165(100)	77(100)	88(100)
NO	3(1.8)	0(0)	3(3.7)	0.120	0(0)	0(0)	0(0)
Internal information
YES	161(97.6)	82(98.8)	79(96.3)	0.306	165(100)	77(100)	88(100)
NO	4(2.4)	1(1.2)	3(3.7)	0.306	0(0)	0(0)	0(0)
External information
YES	129(78.2)	63(75.9)	66(80.5)	0.300	113(68.5)	50(64.9)	63(71.6)	0.226
NO	36(21.8)	20(24.1)	16(19.5)	0.300	52(31.5)	27(35.1)	25(28.4)	0.226
	SparkDesk				GPT-4
	Overall n = 165,n(%)	Correct n = 58,n(%)	Incorrect n = 107,n(%)	P	Overall n = 165,n(%)	Correct n = 62,n(%)	Incorrect n = 103,n(%)	P
Logical reasoning
YES	72(43.6)	12(20.7)	60(56.1)	< .001	165(100)	62(100)	103(100)	—
NO	93(56.4)	46(79.3)	47(43.9)	< .001	0(0)	0(0)	0(0)	—
Internal information
YES	72(43.6)	12(20.7)	60(56.1)	< .001	165(100)	62(100)	103(100)	—
NO	93(56.4)	46(79.3)	47(43.9)	< .001	0(0)	0(0)	0(0)	—
External information
YES	40(24.2)	5(8.6)	35(32.7)	< .001	119(72.1)	42(67.7)	77(74.8)	0.213
NO	125(75.8)	53(91.4)	72(67.3)	< .001	46(27.9)	20(32.3)	26(25.2)	0.213

Table 3

Statistical analysis of the reasons behind incorrect responses
	Ernie Bot		ChatGLM		SparkDesk		GPT-4
	n = 82	n(%)	n = 88	n(%)	n = 58	n(%)	n = 103	n(%)
Logical error	10	12.2	3	3.4	2	1.9	1	1
Information error	51	62.2	77	87.5	86	80.4	101	98
Logical and information errors	21	25.6	8	9.1	19	17.7	1	1

Principal Findings

Our results indicated that Ernie Bot performed the best in terms of scores and correctness, followed by ChatGLM. Both of them exceeded the passing score, demonstrating their expertise in TCM surpassed the passing threshold for the postgraduate examination. The performance of SparkDesk and GPT-4 was unsatisfactory. This suggests that Ernie Bot and ChatGLM have the potential to be used in TCM and could serve various roles in the future.

Differences in AI accuracy across input languages may partially stem from the linguistic composition of its dataset, as LLMs often exhibit a preference for languages more aligned with their training data²¹. ChatGPT performs best with English input, emphasizing the influence of language on its accuracy²². Peng et al (2024)²³ discovered that ChatGPT demonstrated medium accuracy in answering open-ended medical questions in Chinese, with an accuracy rate of 31.5%. Considering the disparities in English and Chinese inputs, it's clear that ChatGPT needs further improvements to handle medical questions in Chinese. Furthermore, the performance of GPT-4 did not meet expectations in this study, likely because it is a general-domain model with relatively limited training data in TCM.

The significant differences in the performance of Ernie Bot, ChatGLM, and GPT-4 across various subjects suggested that they were not effective as a comprehensive TCM learning tool. These four LLMs did well in responding to the questions on the medical humanistic spirit, possibly because answering such questions does not necessitate specialized medical knowledge or clinical experience. In TCM education, students first learn foundational subjects before moving on to clinical subjects. The ability to effectively address clinical questions relies, in part, on mastering these fundamentals and applying them correctly. Ernie Bot showed a higher accuracy rate in answering questions about Pharmacology (foundational subjects) and Internal Medicine of TCM (clinical subjects), while ChatGLM and GPT-4 performed well in Internal Medicine of TCM. Consequently, in the present study, the remarkable performance of LLMs in Internal Medicine of TCM (clinical subjects) gave a huge impression. This may be because the Internal Medicine of TCM questions were mostly presented in case formats, where LLMs showed greater proficiency in extracting and processing internal information. Subject-level accuracy illustrates the suitability of various models for distinct domains, providing invaluable guidance for users seeking to choose a model tailored to their specific requirements²⁴.

In the qualitative analysis of responses, 1) In terms of logical reasoning ability, when the prompt specifically asked the LLMs to generate an answer parsing, Ernie Bot, ChatGLM, and GPT-4 produced more context-focused answers, which were better representations of the deductive reasoning process. SparkDesk had 56.4% of answers lacking logical reasoning. Meanwhile, the accuracy rate of responses showing logical reasoning exceeded those without it. This suggests an urgent need for SparkDesk to improve its deductive reasoning abilities to increase accuracy rates. 2) In terms of the ability to use internal and external information, Ernie Bot, ChatGLM, and GPT-4 incorporated internal information in over 95% of their answer parses, with over 60% integrating external information. The frequency of correct answers with external information did not significantly surpass that of incorrect answers. This suggested that although LLMs could connect questions to extra data, such information did not substantially aid in making the correct choice.

The causes of incorrect responses were categorized into three groups: logical errors, information errors, and both types of errors at the same time. General-domain LLMs were mainly trained on widely accessible public datasets, which might have limitations due to the lack of training data specific to professional knowledge within specialized domains²⁵. When confronted with issues necessitating domain-specific knowledge, general-domain LLMs can be significantly illusory, often resulting in factual fabrications²⁶. It closely aligns with the findings of the present study, where informational errors were identified as the most prevalent factor. Hence, prioritizing data selection and filtering from pre-trained corpora to gather high-quality TCM knowledge data is crucial. Moreover, considering the intricate terminology in TCM, improving the proficiency of LLMs in conducting expert analyses on TCM matters could be achieved through the selection of instruction-tuning data.

In this study, the evaluation of four LLMs' performance in the postgraduate examination revealed that Ernie Bot and ChatGLM were able to pass the exam. Furthermore, over 95% of responses demonstrated logical reasoning and internal information. The majority of responses also presented external information, indicating that they could justify the response in most cases by demonstrating logical reasoning and contextual information. Ernie Bot and ChatGLM could process and comprehend natural language inputs, leveraging their extensive databases to deliver personalized and parsed responses. As a result, the superior performance and accuracy of ChatGLM and Ernie Bot imply a greater proficiency in addressing TCM exam questions. This offers preliminary data that they can lead to integrated TCM applications, showcasing their enormous potential as TCM support tools. The knowledge accuracy and interpretive ability of SparkDesk and GPT-4 need further improvement.

Potential Application of LLMs in TCM

Several application scenario studies have demonstrated the application potential of LLMs. For instance, in the case of medical records: for every hour a physician dedicates to a patient, they need to allocate nearly two hours to electronic medical records and documentation tasks during their work²⁷. ChatGPT could help with writing and organizing medical records²⁸, significantly enhancing the productivity of medical staff. During medical consultations, LLMs could rapidly and accurately translate complex medical terminology, reducing the communication gap between patients and doctors²⁹. LLMs could serve as medical assistants for patients, helping with tasks like scheduling appointments, arranging checkups, coordinating treatments, and managing their health information³⁰. In clinical specialties like radiology³¹ and psychiatry³², LLMs could help with clinical care by optimizing workflow or improving diagnostic accuracy and efficiency. Nonetheless, LLMs have several limitations. For example, the most basic consultation cannot adequately replace direct interaction between an experienced doctor and a patient. The same applies to considerations of medical ethics and law²⁷. Currently, related studies in TCM are still limited. Biomedicine and TCM are both part of the field of medicine and have a certain degree of transferability. The potential application of LLMs in TCM can be envisioned by drawing upon the research insights gleaned from LLMs in the field of biomedicine.

The logic and contextual information demonstrated by LLMs in most of the answers highlight their potential as valuable tools for medical education and learning support¹⁸. In the future, LLMs could serve as "virtual assistants" for TCM education³³. In medical education, multiple-choice questions constitute the primary examination format³⁴. LLMs can generate questions and quizzes to facilitate both practice and assessment in class sessions¹⁸. Choosing the appropriate LLMs can offer explanations and feedback for each question and clarify ambiguous and complex concepts that students might face. Using this tool, students can ask about specific medical concepts, diagnoses, or treatments and receive precise and personalized responses that enhance their learning and contribute to their knowledge base¹². In addition, LLMs are able to edit basic medical reports, which could help students identify areas for improvement³⁵. They can serve as real-time learning tools for students. They give teachers an efficient way to assess their pupils' skills and pinpoint their areas of weakness. More importantly, users need to exercise caution. It is necessary to validate all responses and information provided by LLMs to ensure the accuracy of disseminated TCM knowledge. LLMs are paving the way for a new era of teaching and learning, pushing the boundaries for educators and students. By continuously training and refining the models, their usefulness in medical education and examinations will expand, possibly revolutionizing our approach to exam preparation and lifelong learning¹⁵.

In general, LLMs offer a diverse array of applications in both TCM clinical practice and medical education. Professionals in relevant fields should stay informed about advancements in AI technology and contemplate its potential applications.

We propose further evaluation of LLMs: a broader range of question formats, such as open-ended questions, could be employed to gain a more comprehensive understanding of their real-world applicability. Secondly, conducting evaluations with a larger question pool reduces uncertainty about perceptions of LLMs. In addition, refining the subjects under assessment (e.g. by sub-disciplines) will yield a more detailed insight into the model's capabilities, allowing for speculation on its application in specific medical scenarios. A more thorough evaluation will enable LLMs to be more precisely situated within medical practice, encouraging their improvement and quick iteration. With training on a larger dataset of high-quality Chinese medical information (such as medical literature and electronic medical records), a Chinese LLM like Ernie Bot could achieve even greater performance, significantly improving its applicability in TCM.

Limitations

This study had the following limitations. Firstly, the postgraduate examination evaluated the basic competence of TCM students comprehensively but did not include a more detailed specialist assessment. Moreover, the exam exclusively comprised objective questions and did not include any subjective questions. Second, this study only explored how LLMs performed on cross-sectional data. However, the exam content differs annually, leading to corresponding changes in passing scores.

Models

Aside from GPT-4, which currently outperformed others in all aspects, we selected three mainstream Chinese LLMs, Ernie Bot, ChatGLM, and SparkDesk.

GPT-4

ChatGPT, developed by OpenAI, has undergone multiple version updates since its release. The latest iteration, GPT-4, was launched on March 14, 2023, integrating enhanced language generation capabilities with improved multi-round dialogue processing²⁴.

Ernie Bot³⁷

Through introducing a self-supervised comparative learning pre-training technique for word mixture and a word-mixing data-enhanced self-adversarial fine-tuning technique, the ERNIE 3.0 series achieved a substantial performance boost across eight Chinese mainstream downstream tasks.

ChatGLM³⁸

ChatGLM is derived from GLM-130B, a bilingual pre-trained language model with 130 billion parameters designed to process both English and Chinese languages. The model architecture of GLM is designed with an autoregressive fill-in-the-blanks structure, facilitating autoregressive prediction of masked fields by employing bidirectional attention mechanisms.

SparkDesk³⁹

SparkDesk is built upon iFlytekSpark-13B. Recent reports indicated that SparkDesk V3.5 surpasses GPT-4 Turbo in mathematical computation, language comprehension, and voice interaction functionalities.

Medical examination datasets

We collected questions from the Chinese Postgraduate Examination for TCM in 2023, for a total of 165 questions. The examination comprised single-choice and multiple-choice questions spanning seven distinct disciplines: Basic Theory of TCM, Diagnostics of TCM, Pharmacology of TCM, Formulas of TCM, Internal Medicine of TCM, Acupuncture and Moxibustion, and Medical Humanistic Spirit (Table 4).

Table 4

Subjects examined in the test questions
Subjects	No.	Sample questions
Basic Theory of TCM	23	According to mutual generation and control among the five elements, the principle of restraining the strong and supporting the weak is demonstrated by ( )
		A. Warming the kidney and tonifying the spleen
		B. Tonifying the kidney and enriching the liver
		C. Purging the heart and clearing the liver
		D. Soothing the liver and tonifying the spleen
Diagnostics of TCM	23	What is not a common indication of bluish and black facial complexion?
		A. Blood stasis pattern, Pain
		B. Blood stasis pattern, Infantile convulsions
		C. Cold pattern, Pain
		D. Cold pattern, Blood stasis pattern
Pharmacology of TCM	23	According to the theory of medicinal properties, most of the drugs used to treat edema and inhibited urination have the flavor of ( )
Pharmacology of TCM	23	A. Astringent flavor B. Bland flavor C. Salty flavor D. Bitter flavor
Formulas of TCM	23	The drug found in both Qianghuo Shengshi Decoction and Daqinjiao Decoction is ( ).
		A. Atractylodis rhizoma and Angelicae pubescentis radix
		B. Chuanxiong rhizoma and Saposhnikoviae radix
		C. Ligustici rhizoma et radix and Angelicae dahuricae radix
		D. Asari radix et rhizoma and Glycyrrhizae radix et rhizoma
Internal Medicine of TCM	43	A 34-year-old woman with a 3-month history of coughing showed increased texture in the bilateral lower lungs on a chest X-ray. Coughing, distension in the sub-costal region, dry throat and a bitter taste, scanty and sticky phlegm, red tongue with thin, yellow coating, and wiry pulse, belong to ( )
		A. Wind dryness damaging the lung pattern
		B. Lung yin deficiency pattern
		C. Liver fire affecting the lung pattern
		D. Lung heat and yin deficiency pattern
Acupuncture and Moxibustion	22	Located 7 inches below the cubital crease is ( )
Acupuncture and Moxibustion	22	A.LU6 B.LI6 C.PC4 D.SJ6
Medical Humanistic Spirit	8	The medical profession is the community that is accountable for the commitments expected by the public, and the following narrative is incorrect ( )
		A. The medical profession does not require systematic training.
		B. Professional community self-regulation is essential for the medical profession.
		C. The medical profession earns community trust by enforcing industry standards.
		D. Professional integrity within the medical profession is upheld through a rigorous admission system.

Prompt Engineering

The prompts have a significant impact on the output of LLMs ⁴⁰. To obtain the best form and content of answers, we supplied prompts and instructed LLMs to furnish explanations and justifications for their selected answers.

Each question was input in the specified format: "The following is a single-choice question from the Chinese Comprehensive Ability of Clinical Medicine (TCM) Examination. Please choose the correct answer and provide a correct and reasonable answer analysis" (single-choice question) or " The following is a multiple-choice question from the Chinese Comprehensive Ability of Clinical Medicine (TCM) Examination. For each sub-question, there are four options labeled A, B, C, and D, among which at least two meet the requirements of the question. Please choose the correct answers and generate a correct and reasonable answer analysis" (multiple-choice question). Next, input the question and its corresponding options. Table 5 displays an example question along with the answers provided by the LLMs.

Table 5

Response scenarios for the identical question from Ernie Bot and GPT-4
Questions
Input question: The following is a single-choice question from the Chinese Comprehensive Ability of Clinical Medicine (TCM) Examination. Please choose the correct answer and provide a correct and reasonable answer analysis. A 49-year-old female presenting with insomnia persisting for 4 months, compounded by vexation with insomnia, difficulty in falling asleep, palpitation and dreaming with dizziness and tinnitus, along with low back soreness and knee weakness, tidal fever, night sweats, feverish sensations in palms, soles and chest, dry throat with little fluids, accompanied by a red tongue displaying scant coating and thready, rapid pulse, is suitable for ( ). A.Zuojin Wan and Suanzaoren Decoction B.Zhibai Dihuang Wan C.Liuwei Dihuang Wan and Jiaotai Wan D.Tianma Gouteng Decoction
Ernie Bot The correct answer is B.Zhibai Dihuang Wan The patient, a middle-aged woman, is experiencing insomnia, compounded by vexation with insomnia, difficulty in falling asleep, palpitations, dreaminess, dizziness, and tinnitus. Additionally, she is struggling with low back soreness and knee weakness, tidal fever and night sweats, feverish sensations in palms, soles and chest, as well as dry throat with little fluids, a red tongue with scant coating and a thready, rapid pulse. After assessing her symptoms, tongue manifestations, and pulse conditions, she was diagnosed with insomnia caused by yin deficiency leading to fire hyperactivity. The treatment aims to nourish and tonify the blood and tranquilize the spirit. The formula is a modified version of Suanzaoren Decoction (A is incorrect). This question also includes symptoms of a dry throat and little fluids. The Suanzaoren Decoction does not include the treatment of nourishing yin and clearing the lungs to benefit the throat, so it is excluded. The patient's insomnia, attributed to yin deficiency leading to fire hyperactivity, warrants treatment focusing on heart and kidney perspectives. Coptidis Rhizoma in Zuojin Wan and Suanzaoren Decoction is specially formulated to clear the heart and tranquilize the spirit, effectively treating insomnia. However, in this case, kidney yin deficiency is predominant and treatment should be based on nourishing and supplementing kidney yin. So Zuojin Wan is not appropriate (A is incorrect). The patient's insomnia is caused by yin deficiency leading to fire hyperactivity. Treatment should focus on nourishing and tonifying the blood to tranquilize the spirit. The patient's symptoms are mainly low back soreness and knee weakness, dizziness and tinnitus, tidal fever and night sweats, feverish sensations in palms, soles and chest, dry throat and little fluids, red tongue with little coating, and a thready and rapid pulse. These indicate insomnia resulting from yin deficiency leading to fire hyperactivity. Liuwei Dihuang Wan is formulated to tonify the kidney. However, since kidney-yin deficiency predominates in this case, treatment should prioritize nourishing and supplementing kidney-yin. Therefore, Liuwei Dihuang Wan is not suitable (C is incorrect). Tianma Gouteng Decoction is weak in smoothing the liver, submerging yang, nourishing blood, and calming the mind. It is not effective in treating insomnia caused by yin deficiency leading to fire hyperactivity (D is incorrect). Zhibai Dihuang Wan is effective in enriching yin and reducing fire, making it suitable for treating insomnia caused by yin deficiency leading to fire hyperactivity (B is correct).
GPT-4 The correct answer is: B.Zhibai Dihuang Wan Answer Analysis: Based on the patient's symptoms including insomnia, palpitations, dreaminess, dizziness, tinnitus, low back soreness and knee weakness, tidal fever, night sweats, feverish sensations in palms, soles, and chest, dry throat with little fluids, as well as tongue and pulse observations (red tongue with scanty coating and thready, rapid pulse), the pattern corresponds to insomnia resulting from yin deficiency leading to fire hyperactivity. Zhibai Dihuang Wan, a classic formula for nourishing yin and reducing fire, is primarily prescribed to alleviate symptoms like tidal fever, night sweats, feverish sensations in the palms, soles, and chest, and dry mouth resulting from kidney-yin deficiency. A. Zuojin Wan combined with Suanzaoren Decoction is primarily employed to address insomnia resulting from the transformation of qi stagnation into fire. C. Liuwei Dihuang Wan combined with Jiaotai Wan is primarily utilized to treat symptoms arising from kidney-yin deficiency and kidney-yang deficiency. However, since the main symptom of the patient is yin deficiency leading to fire hyperactivity, Jiaotai Wan, being warm, is not suitable for this pattern of fire hyperactivity. D. Tianma Gouteng Decoction is mainly used for the internal stirring of liver wind pattern. Therefore, considering the patient's symptoms, the most appropriate option is Zhibai Dihuang Wan. It effectively enriches yin and reduces fire, making it suitable for treating insomnia resulting from yin deficiency leading to fire hyperactivity.

Data Analysis

All model testing was conducted by inputting questions and obtaining answers provided by the LLMs. All responses were recorded in an electronic spreadsheet, and the chosen answers from each response were extracted and compared to the standard answers. Each question was considered "correct" only if the answer choices matched the standard answer, whereas incorrect choices, missing options, or ambiguous responses were labeled as "incorrect". The accuracy of LLMs' responses was calculated after the process.

Each multiple-choice question in the Basic Theory of TCM, Diagnostics of TCM, Pharmacology of TCM, and Formulas of TCM modules was worth 1.5 marks. Within the Internal Medicine of TCM, Acupuncture and Moxibustion, and Medical Humanistic Spirit modules, questions numbered 37–81 were worth 2 marks each, while questions 98–105 were worth 1.5 marks each. Each multiple choice question was worth 2 points. Subsequently, LLM scores for the administered tests were computed.

We calculated the accuracy rates of LLMs' responses across various subjects in the test questions and conducted statistical analysis on them. SPSS 27.0 software was used for data processing, and Pearson's χ2 test was used for comparison of rates, with P < 0.05 indicating a statistically significant difference.

Qualitative Analysis

Three binary variables were employed to assess the responsiveness of LLMs towards each question:

Logical reasoning: The response demonstrated the logical connection between the provided information and the answer choices.

Ability to use internal information: The response incorporated information relevant to the question, including analyses derived from case content or essential explanations focused on the question.

Ability to use external information: The response incorporated external information beyond the scope of the question, including but not limited to additional analyses and expansions of the topic, provided answers, and distractors.

Each incorrect answer was categorized as one of three types based on the reason for the mistake:

1. Logical error: The parsing contained relevant information but did not correctly convert it into answers. The options in the parsing differed from those in the output. The parsing contained contradictory statements.

Example Question: "What can cause low-grade fever?" Model output (GPT-4): "D. Qi stagnation: Qi stagnation is not directly related to low-grade fever, but prolonged Qi stagnation can transform into Fire, which can cause a low-grade fever.” The analysis showed that Qi stagnation could lead to a low-grade fever, but option D was not selected in the response. This is a logical error.

2. Information error: The extraction, interpretation, and application of internal information or the presentation of external information is incorrect.

Example Question: According to “Huang Di Nei Jing Su Wen”, what are the manifestations of men at “five times eight”? Model output (Ernie Bot): According to "Huang Di Nei Jing Su Wen", men at “five times eight” exhibit strong tendons and bones. (A is correct)”. Due to this misinformation, the answer given is incorrect.

3. The presence of both types of errors concurrently.

This study involved three evaluators: two senior postgraduate students in TCM who successfully completed the Postgraduate Examination, and one senior clinical professional in TCM. To familiarize the evaluators with the scoring system, 30 questions from the 2022 Chinese Postgraduate Examination for TCM were used for training purposes. The process was independently assessed by the two masters based on the mentioned criteria and then cross-checked. In the event of disputes, another Chinese medicine professional would make the final judgment.

Competing interests

The authors declare no competing interests.

Author Contribution

B. Wang and M. Zhang performed the data acquisition. S. Peng performed the data analysis. M. Hao, M. Zhang, and J. Wang performed the evaluation. B. Wang, Zhe Wang, Y. Zhu, and S. Peng contributed to writing and drafting. K. Yao contributed to reviewing and proofreading. Y. Zhu was the collaborative lead.

Acknowledgments

This work was supported by grants from the National Natural Science Foundation of China (No. 82174534), Research on Evaluation Methods of Traditional Chinese Medicine Examination Based on Mainstream Large Language Models (TB2023008), China Academy of Chinese Medical Science and Technology Innovation Project (C12021A05306), Special Funds for Basic Research Operating Costs of Central-level Public Welfare Research Institutions (ZZ13-YQ-126), China Academy of Chinese Medical Science Basic Research Operating Expenses (ZZ170320), and China Academy of Chinese Medical Science and Technology Innovation Project (C12021A02314).

Data Availability

The study did not include human clinical trials/interventions or human tissue samples. It utilized publicly available medical examination question dataset. The test data in this study are included in the Supplementary Information file provided in this paper.

Hsieh, C. R. & Tang, C. The multi-tiered medical education system and its influence on the health care market-China's Flexner Report. Hum Resour Health 17, 50, doi:10.1186/s12960-019-0382-4 (2019).
Liu, X. et al. Medical Education Systems in China: Development, Status, and Evaluation. Acad Med 98, 43–49, doi:10.1097/acm.0000000000004919 (2023).
Anand, S. et al. China's human resources for health: quantity, quality, and distribution. Lancet 372, 1774–1781, doi:10.1016/s0140-6736(08)61363-x (2008).
2024 National Graduate Enrolment Survey Report, <https://www.eol.cn/e_ky/zt/report/2024/abstract.html> (2023).
Wang, W. Medical education in china: progress in the past 70 years and a vision for the future. BMC Med Educ 21, 453, doi:10.1186/s12909-021-02875-6 (2021).
OpenAI et al. GPT-4 Technical Report. arXiv:2303.08774 (2023). <https://ui.adsabs.harvard.edu/abs/2023arXiv230308774O>.
Berg, H. T. et al. ChatGPT and Generating a Differential Diagnosis Early in an Emergency Department Presentation. Ann Emerg Med, doi:10.1016/j.annemergmed.2023.08.003 (2023).
Srivastav, S. et al. ChatGPT in Radiology: The Advantages and Limitations of Artificial Intelligence for Medical Imaging Diagnosis. Cureus 15, e41435, doi:10.7759/cureus.41435 (2023).
Liu, H., Azam, M., Bin Naeem, S. & Faiola, A. An overview of the capabilities of ChatGPT for medical writing and its implications for academic integrity. Health Info Libr J, doi:10.1111/hir.12509 (2023).
Baxi, V., Edwards, R., Montalto, M. & Saha, S. Digital pathology and artificial intelligence in translational medicine and clinical practice. Mod Pathol 35, 23–32, doi:10.1038/s41379-021-00919-2 (2022).
Sharma, P. et al. Performance of ChatGPT on USMLE: Unlocking the Potential of Large Language Models for AI-Assisted Medical Education. arXiv:2307.00112 (2023). <https://ui.adsabs.harvard.edu/abs/2023arXiv230700112S>.
Gilson, A. et al. How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ 9, e45312, doi:10.2196/45312 (2023).
Kung, T. H. et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health 2, e0000198, doi:10.1371/journal.pdig.0000198 (2023).
Takagi, S., Watari, T., Erabi, A. & Sakaguchi, K. Performance of GPT-3.5 and GPT-4 on the Japanese Medical Licensing Examination: Comparison Study. JMIR Med Educ 9, e48002, doi:10.2196/48002 (2023).
Aljindan, F. K. et al. ChatGPT Conquers the Saudi Medical Licensing Exam: Exploring the Accuracy of Artificial Intelligence in Medical Knowledge Assessment and Implications for Modern Medical Education. Cureus 15, e45043, doi:10.7759/cureus.45043 (2023).
Wójcik, S. et al. Reshaping medical education: Performance of ChatGPT on a PES medical examination. Cardiol J, doi:10.5603/cj.97517 (2023).
Lin, S. Y., Chan, P. K., Hsu, W. H. & Kao, C. H. Exploring the proficiency of ChatGPT-4: An evaluation of its performance in the Taiwan advanced medical licensing examination. Digit Health 10, 20552076241237678, doi:10.1177/20552076241237678 (2024).
Wang, X. et al. ChatGPT Performs on the Chinese National Medical Licensing Examination. J Med Syst 47, 86, doi:10.1007/s10916-023-01961-0 (2023).
Zong, H. et al. Performance of ChatGPT on Chinese national medical licensing examinations: a five-year examination evaluation study for physicians, pharmacists and nurses. BMC Med Educ 24, 143, doi:10.1186/s12909-024-05125-7 (2024).
Cai, Y. et al. MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models. arXiv:2312.12806 (2023). <https://ui.adsabs.harvard.edu/abs/2023arXiv231212806C>.
Nicholas, G. & Bhatia, A. Lost in Translation: Large Language Models in Non-English Content Analysis. arXiv:2306.07377 (2023). <https://ui.adsabs.harvard.edu/abs/2023arXiv230607377N>.
Meyer, A., Riese, J. & Streichert, T. Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study. JMIR Med Educ 10, e50965, doi:10.2196/50965 (2024).
Yu, P. et al. Performance of ChatGPT on the Chinese Postgraduate Examination for Clinical Medicine: Survey Study. JMIR Med Educ 10, e48514, doi:10.2196/48514 (2024).
Farhat, F., Chaudhry, B. M., Nadeem, M., Sohail, S. S. & Madsen, D. Ø. Evaluating Large Language Models for the National Premedical Exam in India: Comparative Analysis of GPT-3.5, GPT-4, and Bard. JMIR Med Educ 10, e51523, doi:10.2196/51523 (2024).
Zhao, W. X. et al. A Survey of Large Language Models. arXiv:2303.18223 (2023). <https://ui.adsabs.harvard.edu/abs/2023arXiv230318223Z>.
Huang, L. et al. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv:2311.05232 (2023). <https://ui.adsabs.harvard.edu/abs/2023arXiv231105232H>.
Wójcik, S. et al. Beyond ChatGPT: What does GPT-4 add to healthcare? The dawn of a new era. Cardiol J, doi:10.5603/cj.97515 (2023).
Fernandes, A. C. & Souto, M. Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. N Engl J Med 388, 2399–2400, doi:10.1056/NEJMc2305286 (2023).
Li, Y. et al. ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge. arXiv:2303.14070 (2023). <https://ui.adsabs.harvard.edu/abs/2023arXiv230314070L>.
Marr, B. Revolutionizing Healthcare: The Top 14 Uses Of ChatGPT., <https://bernardmarr.com/revolutionizing-healthcare-the-top-14-uses-of-chatgpt-in-medicine-and-wellness/> (2023).
Mese, I., Taslicay, C. A. & Sivrioglu, A. K. Improving radiology workflow using ChatGPT and artificial intelligence. Clin Imaging 103, 109993, doi:10.1016/j.clinimag.2023.109993 (2023).
Franco D'Souza, R., Amanullah, S., Mathew, M. & Surapaneni, K. M. Appraising the performance of ChatGPT in psychiatry using 100 clinical case vignettes. Asian J Psychiatr 89, 103770, doi:10.1016/j.ajp.2023.103770 (2023).
Lee, H. The rise of ChatGPT: Exploring its potential in medical education. Anat Sci Educ, doi:10.1002/ase.2270 (2023).
Epstein, R. M. Assessment in medical education. N Engl J Med 356, 387–396, doi:10.1056/NEJMra054784 (2007).
Jeblick, K. et al. ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports. arXiv:2212.14882 (2022). <https://ui.adsabs.harvard.edu/abs/2022arXiv221214882J>.
OpenAI. GPT-4 Technical Report. arXiv:2303.08774 (2023). <https://ui.adsabs.harvard.edu/abs/2023arXiv230308774O>.
Baidu. Ernie Bot, <https://yiyan.baidu.com/> (2023).
ZhipuAI. ChatGLM, <https://chatglm.cn/> (2023).
iFLYTEK. SparkDesk, <https://xinghuo.xfyun.cn/> (2023).
White, J. et al. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 (2023). <https://ui.adsabs.harvard.edu/abs/2023arXiv230211382W>.

No competing interests reported.

PerformanceofErnieBotChatGLMSparkDeskandGPT4ontheChinesePostgraduateExaminationdatasetenglishversion.xlsx

Download PDF

Version 1

posted

You are reading this latest preprint version

Performance of GPT-4 and mainstream Chinese Large Language Models on the Chinese Postgraduate Examination dataset: Potential for AI-assisted Traditional Chinese Medicine

Status:

Version 1

Abstract

Introduction

Results

Discussion

Methods

Data Analysis

Declarations

Competing interests

Author Contribution

Acknowledgments

Data Availability

References

Additional Declarations

Supplementary Files

Status:

Version 1