Assessing the feasibility of using GPT models for clinical decision support in patients suspected of prostate cancer: a comparative study

doi:10.21203/rs.3.rs-4885411/v1

Download PDF

Article

Assessing the feasibility of using GPT models for clinical decision support in patients suspected of prostate cancer: a comparative study

https://doi.org/10.21203/rs.3.rs-4885411/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

Large Language Models (LLMs), such as the GPT model, leverage supervised learning and reinforcement learning with human labels for fine-tuning. Despite showing promise in various medical fields, the feasibility and safety of using GPT models for clinical decision support (CDS) in prostate cancer remain unverified. This study aims to evaluate the feasibility of GPT models in providing CDS for patients suspected of prostate cancer by comparing the recommendations generated by GPT models with those provided by real-world urologists.

Methods

Patient data were collected from March 2022 to December 2023 from Tianjin Medical University Cancer Institute and Hospital and Tianjin Medical University Second Hospital. A total of 113 cases with comprehensive clinical and imaging data were selected. Clinical recommendations were generated by GPT models (ChatGPT and GPT-3.5) and compared with those provided by a non-oncology specialized urologist. The recommendations were evaluated by three prostate cancer experts for coherence, factual consistency, comprehensiveness, and potential medical harm using a 5-point Likert scale. Mann-Whitney U tests were employed to determine significant differences.

Results

The GPT models demonstrated high factual consistency (98.1% in high consistency group) and coherence in generating clinical recommendations. In terms of medical harm, no significant difference was observed overall between GPT models and the non-oncology urologist (p ≥ 0.05). However, in cases rated neutral (score = 3), the non-oncology urologist showed higher rates of ambiguous recommendations (10.5%) compared to GPT models (2.8%, p < 0.05). The GPT models' response time was significantly faster, averaging 5–15 seconds per case versus approximately 1 minute for the urologist.

Conclusion

GPT models show promise in providing clinical decision support for patients suspected of prostate cancer, with high factual consistency and efficient response times. However, challenges such as comprehensiveness and potential medical harm need to be addressed before widespread clinical application. Further research is warranted to validate the empowering effect of GPT models on non-specialist clinicians in clinical decision-making.

Health sciences/Diseases/Cancer/Urological cancer/Prostate cancer

Health sciences/Urology/Prostate

Large Language Models (LLMs), such as the GPT model, utilize supervised learning and reinforcement learning with human labels for fine-tuning. These AI models rely on a vast number of parameters for text prediction. Unlike past machine learning or deep learning models, the GPT model, trained with massive parameters, possesses powerful text recognition and processing capabilities. Many studies have confirmed the ability of the GPT model to provide clinical decision support (CDS) using patient histories and imaging examinations supplied by clinicians. It is widely believed to have a clear supportive role and tremendous potential in various medical-related scenarios^[1–5]. A recent study indicated that ChatGPT achieved an overall accuracy rate of 71.7% in decision support for 36 common complex clinical scenarios^[6]. This helps reduce the workload of clinicians to some extent and improves patient involvement during diagnosis and treatment^[7]. However, it is crucial for clinicians to refine the final clinical decisions.

Prostate cancer is a relatively indolent tumor, and clinical decisions for patients with different risks, ages, and disease courses vary. Urologists often need to read and analyze multiple test results and combine them with clinical guidelines to provide personalized diagnosis and treatment recommendations that maximize patient benefits. This data summarization and clinical information extraction task consumes a considerable amount of clinicians' time. GPT models excel in processing and integrating text information. Rational use of GPT models can greatly save clinicians' time and improve work efficiency. Moreover, GPT has also demonstrated an empowering effect on junior and less experienced clinicians in some studies^[8]. However, the rationality and alignment of clinical recommendations provided by GPT models with actual situations need to be verified.

In this study, we explore and assess a novel application of the GPT model for providing CDS to patients with prostate cancer. By utilizing initial patient histories, serum examinations, imaging examinations, etc., we provide clinical advice fosr suspected prostate cancer patients. Furthermore, we compare and analyze the differences between the clinical recommendations provided by the GPT model and those provided by real-world urologists, exploring the feasibility and risks of providing CDS.

2.1 patients

This study involved the collection of patient data from March 2022 to December 2023 from Tianjin Medical University Cancer Institute & Hospital and Tianjin Medical University Second Hospital by two researchers. Patients included in the study were initial presentation with prostate diseases or referred from other hospitals (all patients had not undergone biopsy or other invasive examinations or treatments before their visit). The medical history, serum PSA examination, MRI, PET-CT, transrectal ultrasound, and other results of the patients were collected and organized. We selected 113 cases with comprehensive clinical and imaging data, including at least two imaging examinations. All clinical data of the included patients were compiled into brief medical records according to a fixed format. All patient data were anonymized, and written informed consent was obtained from the patients. Ethical approval of all methods in this study was obtained from the Ethics Committee of Tianjin Medical University Cancer Institute & Hospital (Approval Number Ey2023021) and we confirmed that all experiments were performed in accordance with relevant guidelines and regulations.

2.2 LLM

One of the researchers created prompts (Supplementary materials table 1) and input them into both ChatGPT (using the version from September 25, 2023) and the GPT-3.5 model (GPT 3.5-turbo-16k-0613). Since the output of the GPT model has some randomness, each researcher repeated the input three times. The outputs were independently evaluated by three prostate cancer experts Dr. Zhang Zhenting, Dr. Liao Wenfeng and Dr. Liang Xuan (with more than 10 years of clinical and research experience in prostate diseases). The evaluation criteria were based on the standards used in previous studies by Liyan, T^[9], and Zhaoyi, S^[10], which included "coherence," "factual consistency," "comprehensiveness," and "medical harm" ( Fig. 1). Coherence refers to the logical structure and organization of the recommendations. Factual consistency assesses whether the recommendations align with the facts and whether there are contradictions or discrepancies with the patient's condition. Comprehensiveness considers whether the recommendations take into account the patient's general condition, medical history, and all relevant examination data, providing personalized and appropriate clinical advice. Medical harm is defined as the likelihood that the audience of the recommendation may draw incorrect conclusions from the text, potentially causing physical or mental harm or affecting treatment or compliance. Before the evaluation, the three reviewers received training on the content and evaluation criteria to ensure consistency in their evaluations. Each reviewer evaluated each output of the GPT model on multiple dimensions using a 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree). If reviewers identified issues with factual consistency and medical harm, they were required to select the reasons for the issues. Mann-Whitney U tests were used to determine whether there were significant differences between multiple output contents and whether there were differences between different reviewers. It should be noted that this study does not include any AI authors. GPT was only used to generate content related to clinical decision-making suggestions, while all other parts of the research were completed by the authors collectively

2.3 NOS urologist

We engaged a seasoned urologist from a non-oncology specialized hospital (NOS urologist) with over 10 years of experience in urology. Given the low rate of patient referrals from external institutions to their hospital, we focused exclusively on primary cases to simulate real-world patient scenarios. The NOS urologist was provided with the same medical records used for GPT and asked to provide the most suitable clinical recommendations, including at least two options ranked by preference (Supplementary materials table 2). These recommendations were then evaluated by reviewers solely based on factual consistency and potential medical harm. The evaluation results were compared with those obtained from GPT using the Mann-Whitney U test to assess differences. All statistical analyses were performed by a single researcher using SPSS 22.0 software, with a significance level set at p < 0.05.

3.1 GPT vs NOS urologist

We compared the clinical recommendations provided by the GPT model and NOS urologist (Fig. 2) and categorized all cases into three groups based on scores: those with scores greater than 3 (Group 3A), less than 3 (Group 3B), and a score of 3 (Group 3C) for analysis (Table 1). In terms of medical harm, there was no significant statistical difference observed overall between the two (p ≥ 0.05). Specifically, in Group 3B (low medical harm risk) and Group 3A (high medical harm risk), both GPT and NOS urologist did not exhibit significant statistical differences (p ≥ 0.05). However, in Group 3C, cases where NOS urologist's recommendations were significantly more than GPT's (p < 0.05).

Table 1

Expert assessment results of GPT compared to NOS urologist in medical risk and factual consistency.
Distribution of cases (%)	GPT	NOS urologist	p-value
Medical harm			0.069
3A (> 3, high risk)	12.4%	10.0%	0.439
3B (< 3, low risk)	84.8%	79.5%	0.161
3C ( 3, neutral)	2.8%	10.5%	0.002
Factual consistency			0.005
3A (> 3, good consistency)	98.1%	93.8%	0.026
3B (< 3, poor consistency)	1.9%	3.3%	0.359
3C ( 3, neutral)	0	2.9%	0.014

Factual consistency assessment indicated that the GPT model performed better overall than clinical doctors, with statistically significant differences observed (p < 0.05). Particularly, in Group 3A (good consistency), GPT's performance was significantly better than NOS urologist and had statistical differences (p < 0.05), while in Group 3B (poor consistency), its performance was similar to NOS urologist (p ≥ 0.05). This suggests that the clinical recommendations provided by the GPT model may better align with the actual patient conditions and clinical guidelines compared to non-oncology specialized clinical doctors. Additionally, in Group 3C, both showed significant statistical differences (p < 0.05).

Furthermore, the response time of the GPT model was significantly more efficient compared to the NOS urologist. The GPT model took approximately 5–15 seconds to complete clinical recommendations, whereas the NOS urologist averaged around 1 minute per case to provide clinical advice (including the time needed to consult clinical guidelines and relevant materials for certain cases). Particularly, in the context where clinical data was already organized and important information was extracted, reading imaging reports and various test results would require more time in real world clinical scenario.

3.2 GPT models

The results of the evaluation of GPT clinical recommendations are shown in Fig. 3. Firstly, based on the review results, we found that both ChatGPT and GPT-3.5 models were recognized for their coherence and consistency, demonstrating the completeness and accuracy of the GPT model's output. The occasional issues with consistency were minor deviations from prostate diagnosis and treatment guidelines, but not serious errors, and were mainly due to the conservative nature of the clinical recommendations provided by GPT, which all reviewers generally agreed upon. No reviewers reported instances of contradictions or biases, common shortcomings of GPT^[11]. Secondly, in terms of comprehensiveness, some reviewers noted that the GPT model did not provide a sufficient range of options in its clinical recommendations for a few cases (less 3 points, 5.7%). For example, for patients with PSA levels in the gray zone and unclear results on imaging examinations, the recommendation was for patients to actively monitor, but did not suggest that prostate biopsy could also be an option. Conversely, there were no instances of difficulty in understanding due to complex content.Some reviewers suggested that the clinical recommendations provided by ChatGPT and GPT-3.5 models might pose potential medical harm risks (over 3 points, 12.4%), primarily categorized as being "too conservative" or "too aggressive," with a few recommendations (< 1%) possibly leading to decreased patient compliance.

In all the evaluation criteria mentioned above, including comparisons between three sets of output contents and between different reviewers, we did not find any statistically significant differences (p ≥ 0.05). Similarly, there were no statistically significant differences between ChatGPT and GPT-3.5 models (p ≥ 0.05).

Through a multicenter study involving multiple reviewers and iterations of output, we found that the GPT model is capable of generating coherent and understandable content while preserving the integrity of clinical data. The CDS provided by GPT also reached a level that was generally satisfactory to physicians, and its reliability was not compromised by the randomness inherent in multiple outputs. This indicates that the GPT model can assist urologists in completing a significant portion of repetitive tasks such as reading, organizing, and summarizing medical records, thereby notably improving clinical work efficiency.

In comparing the clinical recommendations of the GPT model with those of the NOS urologist, we found that the recommendations from GPT may better align with individual patient needs and prostate cancer-related diagnostic and treatment guidelines without significantly increasing the risk of patient harm. This result suggests that the GPT model may empower non-specialist urologists in clinical decision-making for prostate diseases. Furthermore, the CDS provided by GPT not only includes recommendations but also involves analysis and summarization of clinical data, which substantially reduces the difficulty patients face in understanding their disease status and personal condition, thus enhancing patient engagement and compliance during diagnosis and treatment. Meanwhile, NOS urologists showed significantly higher rates compared to the GPT model in cases rated 3 points for medical harm and factual consistency assessment (GPT vs NOS urologist, medical harm: 2.8% vs 10.5%; factual consistency: 0 vs 2.9%, p < 0.05). Feedback from the three reviewers indicated that NOS urologists provided more ambiguous clinical recommendations, requiring clearer communication with patients to arrive at definitive clinical advice. This aligns more closely with the realities of medical practice. Reviewers noted the difficulty in assessing the presence of medical harm risk and factual consistency in such cases. Suggestions provided by GPT were relatively clearer and more precise. Consequently, cases rated 3 points exhibited significant differences between GPT and NOS urologists.

In a few cases, reviewers found that the clinical recommendations provided by GPT lacked comprehensiveness (below 3 points, 5.7%), as they only offered the most reasonable clinical advice without considering other options. However, the final recommendations made by clinicians may align with those suggested by the GPT model. Reviewers also reported occasional issues with the factual consistency of GPT's clinical recommendations, where the preferred and alternative clinical advice suggested by GPT were sometimes reversed. While medical experts can easily identify such biases, they may confuse patients and affect their judgment of the condition. Although these discrepancies are likely to be resolved through subsequent communication between clinicians and patients, they may require clinicians to spend more time addressing patient concerns.

Moreover, some reviewers expressed concerns about potential medical harm (above 3 points, 12.4%) associated with the clinical recommendations provided by GPT. This was mainly observed in cases where the clinical advice was overly conservative, potentially overlooking the need for further investigation or treatment in patients. We speculate that this may be related to regional differences in the epidemiology of prostate cancer. Statistical data show that unlike in Western countries, where a large number of patients are diagnosed with localized prostate cancer or even clinically insignificant prostate cancer during initial consultations, in China, the incidence of prostate cancer is relatively low, but the mortality rate is high, with a high proportion of patients diagnosed at an advanced stage during the initial consultation^[12]. Therefore, Chinese urologists may be more inclined to perform biopsies on patients with PSA levels in the gray zone and unclear MRI results. Furthermore, the significant disparity in published studies between European and American countries and Asian regions may have contributed to the deviation of GPT's clinical recommendations in the review process, as GPT models were trained using these data during pre-training and fine-tuning. All potential risks of medical harm can be mitigated to the greatest extent through the final clinical decisions made by urology specialists.

Of course, the clinical application of GPT models still faces many challenges and risks. First is data privacy. In this study, we only accessed ChatGPT and GPT-3.5 models for conversations, and these operations did not retain the content of the conversations. Additionally, they did not involve tasks such as fine-tuning that require uploading data. Furthermore, all conversation contents were anonymized, protecting patient privacy. Second is model hallucination, which refers to the generation of inaccurate or unrealistic information during text generation, which may affect the quality of the model's output. However, no such cases were observed in our study. Finally, there are ethical issues. As repeatedly emphasized, the clinical recommendations provided by GPT models need to be open and optional, and the final decision should be made by clinicians after thorough communication with patients.

Lastly, the limitations of this study mainly lie in several aspects: 1. This study did not use the GPT-4 model for research. The main reason is that GPT-4 is not a completely open and free platform. If patients need to obtain clinical recommendations through the GPT model, it may pose difficulties. Although GPT-4 has demonstrated powerful content output capabilities, considering usability and accessibility, we chose to conduct the study based on the GPT-3.5 model. GPT-4 will be included in our research scope in future studies. 2. All evaluation indicators were subjectively evaluated by reviewers. Although the introduction of multiple reviewers and multiple outputs reduces the impact of subjective evaluation, it may lead to an incomplete evaluation of the clinical practicality of GPT models, requiring more objective indicators and larger sample sizes for research. 3. Only one NOS urologist participated in this study, and whether GPT models empower non-specialist clinical physicians requires confirmation through more extensive research.

In summary, the GPT model demonstrates a clear supportive role and tremendous potential in CDS for patients with prostate diseases. The application of GPT in clinical settings may alleviate the workload of clinicians, but its empowering effect on urologists remains uncertain. This study also identified some potential issues and limitations of the GPT model, indicating that further research is needed before its widespread clinical application. However, it is imperative that human medical experts continue to safeguard the ultimate line of defense for health.

Ethics approval and consent to participate

Ethical approval was obtained from the Ethics Committee of Tianjin Medical University Cancer Institute & Hospital (Approval Number Ey2023021).

Consent for publication

All patient data were anonymized, and written informed consent was obtained from the patients.

Data availability statement

The datasets generated and analyzed during the current study are not publicly available due individual privacy, but are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Fundings

Funded by Tianjin Key Medical Discipline (Specialty) Construction Project（TJYXZDXK-009A）s

Authors' contributions

XW conceived and designed the entire experiment, supervised the research process, and reviewed the manuscript. XL and XW jointly formulated the experimental content and procedures, organized the experimental data, and wrote the manuscript. XW, YL, and XL collectively compiled the clinical data of the patients. YL completed the methodology section and was responsible for the theoretical foundation and practical application of the GPT model. XL, WL, and ZZ jointly evaluated the clinical decision support provided by GPT. GZ, from the perspective of a NOS urologist, provided content for clinical decision support.

Acknowledgements

Not applicable

Cheng SW, Chang CW, Chang WJ, et al. The now and future of ChatGPT and GPT in psychiatry. Psychiatry Clin Neurosci. 2023;77(11):592-596. doi:10.1111/pcn.13588
Crimì F, Quaia E. GPT-4 versus Radiologists in Chest Radiography: Is It Time to Further Improve Radiological Reporting? Radiology. 2023;308(2):e231701. doi:10.1148/radiol.231701
Shea YF, Lee CMY, Ip WCT, Luk DWA, Wong SSW. Use of GPT-4 to Analyze Medical Records of Patients With Extensive Investigations and Delayed Diagnosis. JAMA Netw Open. 2023;6(8):e2325000. doi:10.1001/jamanetworkopen.2023.25000
Ueda D, Mitsuyama Y, Takita H, et al. ChatGPT’s Diagnostic Performance from Patient History and Imaging Findings on the Diagnosis Please Quizzes. Radiology. 2023;308(1):e231040. doi:10.1148/radiol.231040
Lim DYZ, Tan YB, Koh JTE, et al. ChatGPT on guidelines: Providing contextual knowledge to GPT allows it to provide advice on appropriate colonoscopy intervals. J Gastroenterol Hepatol. Published online October 19, 2023. doi:10.1111/jgh.16375
Rao A, Pang M, Kim J, et al. Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study. J Med Internet Res. 2023;25:e48659. doi:10.2196/48659
Xue VW, Lei P, Cho WC. The potential impact of ChatGPT in clinical and translational medicine. Clin Transl Med. 2023;13(3):e1216. doi:10.1002/ctm2.1216
Guerra GA, Hofmann H, Sobhani S, et al. GPT-4 Artificial Intelligence Model Outperforms ChatGPT, Medical Students, and Neurosurgery Residents on Neurosurgery Written Board-Like Questions. World Neurosurg. Published online August 18, 2023:S1878-8750(23)01144-0. doi:10.1016/j.wneu.2023.08.042
Tang L, Sun Z, Idnay B, et al. Evaluating large language models on medical evidence summarization. NPJ Digit Med. 2023;6(1):158. doi:10.1038/s41746-023-00896-7
Sun Z, Ong H, Kennedy P, et al. Evaluating GPT4 on Impressions Generation in Radiology Reports. Radiology. 2023;307(5):e231259. doi:10.1148/radiol.231259
ChatGPT and clinical decision support: Scope, Application, and Limitations - PubMed. Accessed November 20, 2023. https://pubmed.ncbi.nlm.nih.gov/37516680/
Wang F, Wang C, Xia H, et al. Burden of Prostate Cancer in China, 1990-2019: Findings From the 2019 Global Burden of Disease Study. Front Endocrinol. 2022;13:853623. doi:10.3389/fendo.2022.853623

No competing interests reported.

Supplementarymaterials.docx

Download PDF

Editor invited by journal
30 Aug, 2024
Submission checks completed at journal
30 Aug, 2024
First submitted to journal
09 Aug, 2024

You are reading this latest preprint version

Assessing the feasibility of using GPT models for clinical decision support in patients suspected of prostate cancer: a comparative study

Status:

Version 1

Abstract

Background

Methods

Results

Conclusion

Figures

1. Introduction

2.Methods and Materials

2.1 patients

2.2 LLM

2.3 NOS urologist

3.Results

3.1 GPT vs NOS urologist

3.2 GPT models

4. Discussion

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1