Applications of GPT-4 for Accurate Diagnosis of Retinal Diseases Through Optical coherence tomography Image Recognition

doi:10.21203/rs.3.rs-3644163/v1

Download PDF

Research Article

Applications of GPT-4 for Accurate Diagnosis of Retinal Diseases Through Optical coherence tomography Image Recognition

https://doi.org/10.21203/rs.3.rs-3644163/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Since the emergence of pre-trained language models, such as the GPT series including GPT4, their affiliated products have demonstrated remarkable capabilities across various fields. In the field of medicine, the application of the GPT series has gained widespread adoption. Ophthalmology, characterized by its high level of specialization and complexity, stands to benefit from the utilization of artificial intelligence technology, as it has the potential to assist healthcare professionals in making more comprehensive, accurate, and personalized diagnoses and treatment plans. This amalgamation of artificial intelligence and ophthalmology holds substantial promise in significantly contributing to the improvement of ocular health conditions for patients. However, it is crucial to ensure the reliability and safety of employing artificial intelligence in medical practice while addressing pertinent concerns surrounding privacy and ethical considerations.

Background: Since the advent of pre-training generators, the application of intelligent chat robots such as ChatGpt in medicine has become a research hotspot.

Implementation: Investigating the potential of GPT-4 in recognizing OCT images of retinal diseases. We selected OCT images from 80 patients who sought treatment at the Department of Ophthalmology, the Second Affiliated Hospital of Harbin Medical University. These OCT images exhibited typical disease characteristics. Before experimenting, we established disease diagnoses based on previous cases. The images were processed and uploaded to GPT-4 for diagnostic analysis using various approaches. The accuracy rates were recorded and subsequently compared with the diagnoses made by retinal disease specialists.

Results: In four distinct experiments, the diagnostic accuracy of GPT-4 was consistently around 26%, which showed a significant disparity compared to the diagnostic accuracy of retinal disease specialists (93.75%). Leveraging GPT-4's interactivity, users had the option to provide relevant information before posing questions, guiding GPT-4 to generate more contextually appropriate responses. This approach appeared feasible, mostly for addressing objective queries, and limited to textual information and simple images. However, when faced with complex or highly specialized images, its performance still requires improvement. Currently, GPT-4 is not a substitute for professional physicians.

Conclusions: The performance of GPT-4 in processing information may deteriorate when subjected to an excessive amount of input data. Optimizing the presentation of information could facilitate more efficient utilization of GPT-4 for problem-solving. This necessitates a substantial amount of domain expertise. Therefore, to leverage AI models for disease diagnosis, it is crucial to distill concise and representative disease features. Simultaneously, there is a pressing need for specialized AI models dedicated to healthcare to be developed and introduced as early as possible, as this would significantly enhance the diagnostic efficiency of healthcare professionals.

Fundus disease

Optical coherence tomography

ChatGPT

In the specialized field of ophthalmology, the diagnosis and therapeutic intervention of retinal diseases epitomize a nexus of deep-seated knowledge and intricate technical acumen, posing formidable challenges. The retina's anatomical composition is distinguished by its nuanced intricacies, amplifying the multifaceted diagnostic dilemmas. Pertinently, incipient retinal afflictions may not exhibit overt symptoms or might precipitate only marginal degradation in visual acuity. This subtle onset often culminates in patients inadvertently trivializing their conditions, thereby precipitating protracted delays in seeking timely medical intervention. The definitive diagnosis of retinal pathologies predominantly hinges on sophisticated imaging modalities, including Optical Coherence Tomography (OCT) and Scanning Laser Ophthalmoscopy (SLO). These avant-garde diagnostic tools give clinicians granular and holistic insights into retinal anomalies. However, given the expansive and intricate landscape of retinal pathologies—some of which may exhibit nuanced differences—pinpoint diagnosis remains an enduring conundrum within the medical fraternity. This underscores the imperative for astute expertise and a wealth of experiential knowledge among ophthalmologists.

In contemporary times, pre-trained generative computational models have catapulted to the forefront of scientific research, evincing transformative potential across a panoply of academic and pragmatic arenas. Parallel to burgeoning technological strides, these models have been at the epicenter of rigorous scholarly exploration and expansive real-world deployment across various sectors. Within the medical realm, iterations such as ChatGPT and its subsequent avatars (e.g., GPT-3.5, GPT-4) have been progressively spotlighted as nexus points of research^[1]. Specifically, within the ophthalmological purview, GPT-4 has manifested commendable efficacy in discerning and addressing salient ophthalmological quandaries. Empirical analyses underscore that GPT-4, when confronted with inquiries about ophthalmological assessments, consistently achieves an accuracy metric exceeding 80%. Nevertheless, extant literature intimates that in the specialized realm of vitreoretinal disorders, GPT-4's prowess remains amenable to augmentation, heralding a fertile ground for ensuing investigative endeavors^[2–3].

It is pivotal to elucidate that ChatGPT, emblematic of text-centric pre-trained generative models, has been honed through voluminous textual datasets, equipping it with the proficiency to interpret and articulate responses to niche medical inquiries, such as those encompassing cardiology. In disciplines based on imaging examination, GPT-4's foray remains fraught with complexities. A recent technological renaissance has witnessed the integration of image recognition faculties into GPT-4, enabling users to proffer retinal images, thereby engendering more nuanced interactions with the model. This paradigm shift augurs an expansive trajectory for GPT-4, propelling its potential in the realm of retinal image-based diagnostics.

For this investigation, we procured OCT images from a cohort of 80 patients presenting with a spectrum of ocular pathologies. Specifically, these conditions encompassed Central Retinal Vein Occlusion (CRVO), Diabetic Macular Edema (DME), Central Serous Chorioretinopathy (CSC), and Polypoidal Choroidal Vasculopathy (PCV). Each pathological condition was delineated by a set of 20 representative OCT images.

To ensure rigor in the initial phase of our study, these images were subjected to an independent evaluation by two seasoned researchers, Sun and Jiang. This meticulous assessment laid the groundwork for the formulation of specific diagnostic criteria that would be adhered to in subsequent phases of the research.

Experiment 1: Preliminary Diagnostic Assessment Using GPT-4's Image Recognition

Leveraging GPT-4's innate image recognition capabilities, we subjected the previously mentioned OCT images to interpretation. The model was explicitly directed to render the two most plausible diagnostic outcomes. Based on this, an initial diagnostic accuracy metric was established.

Experiment 2: Refining the Diagnostic Process Based on Initial Observations

Our introductory analysis indicated that GPT-4's responses, at times, lacked precision. Some diagnostic outcomes were proximate to the image content, notably in cases where PCV was erroneously discerned as AMD. Conversely, there were marked discrepancies in other instances, such as the misclassification of an OCT image of CSC as indicative of a macular hole. Notably, GPT-4 occasionally misconstrued symptomological manifestations, like retinal pigment epithelial detachment and macular edema, as conclusive diagnostic results. In an endeavor to recalibrate and attenuate the influence of these inaccuracies, we revisited the experimental design. In a subsequent iteration, we presented the same set of 80 OCT images to GPT-4 but narrowed its diagnostic purview. Using the four diseases in the experiment as the selection range, compelling it to yield solely one diagnostic conclusion.

Experiment 3: Pattern Recognition through Composite Image Integration

In this phase, an image set was meticulously curated, showcasing hallmark features of the quartet of diseases under investigation, with each condition represented by two prototypical images duly labeled. These representative images were then juxtaposed with the original batch of 80 OCT images to fabricate composite images. Subsequently, GPT-4 was entrusted with the dual tasks of image recognition and interpretation. The primary thrust was to discern whether GPT-4 could innately extrapolate patterns from the juxtaposed images.

Experiment 4: Augmenting Diagnoses with Annotated Textual Descriptors

For this experiment, we conducted an extensive literature review to identify key features of the four diseases in OCT images^[4–16]. We then carefully examined textual explanations supplemented with graphical markers, such as arrows, to highlight significant features. These enhanced images, along with the initial set of 80 primary OCT images, were combined to create a new collection of composite images. GPT-4, once again, was beckoned to undertake the diagnostic task. Concurrently, the model was probed to elucidate the underlying rationale shaping its diagnostic decisions. This approach aimed to harness GPT-4's interactive prowess, striving to instigate a semblance of an AI training effect. The overarching aspiration was to gauge whether GPT-4 could dynamically assimilate and manifest newfound knowledge in real time.

Validation: Benchmarking GPT-4's Performance against Expert Diagnosis

To contrast with the diagnostic outcomes generated by GPT, we sought the expertise of an experienced specialist in retinal diseases. The specialist performed a thorough examination of OCT images without access to the patient's medical history. Subsequently, we recorded and analyzed the diagnostic conclusions made by the specialist, comparing and evaluating them against the diagnostic accuracy of GPT-4.

Experiment 1: Initial Diagnostic Evaluation Using GPT-4

Upon preliminary assessment, when GPT-4 was tasked with producing two potential diagnoses, it accurately identified 26.25% (21 out of 80) images. The detailed diagnosis rates for specific diseases were as follows: CRVO at 20% (4 out of 20), DME at 50% (10 out of 20), CSC at 35% (7 out of 20), and a nil accuracy for PCV. Remarkably, limiting the evaluation to only GPT-4's primary diagnosis reduced the accuracy to 16.25% (13 out of 80). Specifically, DME and CSC garnered diagnostic accuracies of 40% (8 out of 20) and 25% (5 out of 20) respectively, while both PCV and CRVO recorded 0% precision.

Experiment 2: Modifying the Diagnostic Scope for GPT-4

Upon circumscribing GPT-4's diagnostic parameters, there were notable variations in the results; however, they remained less than optimal. The aggregate diagnostic accuracy was 27.5% (22 out of 80), with individual rates being 20%(4 out of 20) for both CRVO and DME, an elevated 65%(13 out of 20) for CSC, and a minimal 5%(1 out of 20) for PCV. There was a discernible decrease in diagnostic accuracy for DME contrasted with a marked increase for CSC.

Experiment 3: Diagnostic Approach Post Image Reference Integration

Incorporating a reference image set for the diseases into GPT-4 did not alter the overarching diagnostic accuracy, which stood at 26.25% (21 out of 80). An in-depth assessment revealed accuracies of 20% (4 out of 20)for CRVO, 40%(8 out of 20) for both DME and CSC, and a nominal 5%(1 out of 20) for PCV. This further emphasizes GPT-4’s inherent challenges in independently identifying patterns in visual data, consolidating the idea that its core competency lies in text analysis.

Experiment 4: Assimilation of Annotated Sample Images and Textual Descriptions

When furnished with annotated sample images complemented by their textual descriptions, GPT-4 displayed an ability to generate responses based on the prompts. However, the precision remained suboptimal. Out of 80 Optical Coherence Tomography (OCT) images, only 26.25%(21 out of 80) were correctly diagnosed, with the breakdown being: CRVO at 40% (8 out of 20), DME at 45% (9 out of 20), CSC at 20% (4 out of 20), and PCV registering no correct diagnoses.

Finally, we enlisted the expertise of a specialist in retinal diseases for image diagnosis, achieving an accuracy rate of 75/80 (94%) without providing any patient history or answer options. Specifically, there was one erroneous diagnosis in CRVO, one in DME, one in CSC, and two in PCV(Table 1).

Even for seasoned retinal disease specialists, making a 100% accurate diagnosis for the cases in this study is challenging without the assistance of medical history. However, their accuracy rate is exceptionally high, implying that the OCT images we used are representative of the disease characteristics. Yet, once the range of disease diagnosis is determined, choosing one out of the four diseases—even based purely on conjecture—should have an accuracy rate of about 25%, which is on par with GPT's accuracy. Previous research has highlighted significant room for improvement for GPT-4 in ophthalmology, especially in vitreous and retinal diseases, underscoring the limitations of GPT-4 in retinal disease applications. Even when provided with example images of the disease, GPT's diagnostic accuracy did not improve, indicating that GPT's capability in deducing image content remains subpar. However, for an AI model primarily designed for text processing, parsing image information is considerably more complex than parsing textual data. Thus, the high error rate of GPT-4 in retinal disease diagnosis is anticipated.

Distinctively different from other AI models, the GPT series excels in interactivity. In light of this, we provided it with illustrative images equipped with arrow annotations, accompanied by detailed text descriptions to guide the identification of key features. Despite these measures, the diagnostic accuracy of the trained GPT did not meet expectations. Delving into the diagnostic errors of GPT-4, we hoped for it to elucidate the rationale behind its diagnosis. GPT indeed can analyze based on the supplied textual information and identify these characteristics in the corresponding OCT images(Fig. 1), suggesting that the data provided to GPT-4 was processed effectively. This approach is somewhat viable. However, the root cause of its suboptimal accuracy is that GPT-4 often focuses on partial evidence as the basis for diagnosis, compromising its judgment accuracy. For instance, in diagnosing DME and CRVO, it merely noticed irregular cystic cavities and widespread retinal thickening but overlooked hard exudates between the retinal layers. At times, GPT fails to correctly interpret hard exudates, misjudging high reflective signals adjacent to RPE in CSC as hard exudates. This implies that regardless of how meticulously the text is edited, organized, and annotated, if the conveyed data volume is excessive or involves profound professional knowledge, GPT's processing capacity might still be constrained, hindering its effective integration of this information. Paradoxically, simplifying key points may result in diagnostic omissions by GPT, while comprehensive input may induce confusion due to diseases presenting only some symptoms clinically. To utilize GPT for image diagnosis, it may be necessary to clearly distinguish between essential and auxiliary diagnostic conditions for diseases. However, considering the intricacy and variability of retinal diseases, this presents a formidable challenge. This study didn't delve into subtypes of diseases, only using typical cases. A deeper knowledge reference would demand even more detailed textual descriptions. Thus, optimizing the information structure and presentation may be pivotal to enhancing its accuracy and efficiency.

It needs to be explicitly stated: Although the GPT-4 model can process user input information and use it as a reference for answering questions, it still appears imprecise for knowledge not within its pre-trained database. In the previously mentioned Experiment Four, we extensively discussed that hard exudates primarily concentrate around the outer plexiform layer's vicinity. However, during the assessment, GPT-4 still used this scenario as a benchmark, misdiagnosing CSC as DME. Such misjudgment may arise from the GPT model's interpretation of the high-reflective points to their position within the overall OCT image, rather than their actual anatomical location within the retinal structure.

Based on these observations, we have ample justification to assert that GPT-4 has significant potential for improvement in handling specialized tasks. Furthermore, in comparison to subjective questions, GPT-4 appears to demonstrate superior performance in addressing objective queries. It is capable of immediate acquisition and application of new knowledge, facilitating problem-solving. However, such questions must possess a certain degree of regularity or be based on textual descriptions, as opposed to being highly variable or presented in image format. In terms of recognizing ophthalmic OCT images, its performance is still unsatisfactory.

To achieve more precise judgments and diagnoses, future language models urgently need further training and optimization for handling such specialized questions. However, for a model of GPT-4's magnitude, mere integration or database updates often come with a duration of several weeks or even months and await OpenAI initiating a new round of pre-training for the model to be available to the general user base. Consequently, at this stage, GPT-4's image recognition capability still cannot operate entirely independently for retinal disease diagnosis. In clinical practice, we should always prioritize and rely on experienced clinicians to take the lead in decision-making.

In a recent study, we thoroughly evaluated the performance of OpenAI's GPT-4 model in the diagnosis of retinal diseases using OCT (Optical Coherence Tomography) imaging. Although GPT-4 has demonstrated outstanding performance in various text information processing tasks, its efficacy is suboptimal when applied to retinal disease image diagnosis. The analysis revealed that GPT-4 is prone to errors when interpreting images that involve detailed anatomical and specialized medical knowledge.

Based on these observations, we propose that future research directions should consider in-depth optimization and specialized training of the language models, or the development of models specifically tailored for the medical domain, to enhance their accuracy for such specialized issues. Such advancements have the potential to significantly improve the efficiency of clinical diagnostics. However, due to the scale and complexity of training these models, such updates and integrations might require an extended timeline. In this context, we also call upon medical experts to provide more precise and detailed data and feedback to support the continuous improvement of these models.

Nevertheless, at this juncture(2023.11.4), it is imperative to emphasize the irreplaceable role of clinical doctors' expertise and judgment in medical decision-making. AI should be considered as an auxiliary tool in medical decisions, not a substitute.

SLO: Scanning Laser Ophthalmoscopy;

OCT: Optical coherence tomography;

CRVO: Central retinal vein occlusion

DME： Diabetic Macular Edema

CSC： central serous chorioretinopathy

PCV： Polypoid choroidal vascular

Acknowledgments

Not applicable.

Authors’ contributions

H.Y. and Y.F. collaborated in conceptualizing the experimental methodology and authoring the article's subject. B.J. and D.S. provided the final diagnosis for these patients. P.F. and L.S. revised the manuscript, providing stylistic enhancements and refinements to the article. L.G., as a doctor, made a diagnosis of the disease and compared it with the diagnostic results of GPT. H.L., Yin.Z., and S.D. collected OCT images and classified them. J.L. and M.H. created annotated images for interaction with GPT. X.Z. and Yi.Z. counted and organized the accuracy of the diagnosis. All authors have read and approved the final manuscript.

Funding

Not applicable.

Availability of data and materials

Not applicable.

Ethics approval and consent to participate

This research involves a retrospective analysis utilizing pre-existing imaging reports of patients, where no direct intervention or treatment was implemented. In compliance with ethical standards, all patient data were anonymized and de-identified to ensure confidentiality and privacy. The study design, inherently based on historical data, aligns with the criteria for exemption from ethical review, as outlined by institutional guidelines. Furthermore, this study adheres to all applicable legal and regulatory requirements concerning medical data usage. Consent from subjects was deemed unnecessary due to the retrospective nature of the study and the use of de-identified data, which does not compromise patient autonomy or confidentiality. All methods were carried out in accordance with relevant guidelines and regulations. Due to the retrospective nature of the study ethics approval of the study and informed consent to participate was waived by the ethics committee of the Second Affiliated Hospital of Harbin Medical University

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Goodman RS, Patrinely JR, Stone CA Jr, et al. Accuracy and Reliability of Chatbot Responses to Physician Questions. JAMA Netw Open. 2023;6(10):e2336483. Published 2023 Oct 2. doi:10.1001/jamanetworkopen.2023.36483
Mihalache A, Popovic MM, Muni RH. Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment. JAMA Ophthalmol. 2023;141(6):589-597. doi:10.1001/jamaophthalmol.2023.1144
Mihalache A, Huang RS, Popovic MM, Muni RH. Performance of an Upgraded Artificial Intelligence Chatbot for Ophthalmic Knowledge Assessment. JAMA Ophthalmol. 2023;141(8):798-800. doi:10.1001/jamaophthalmol.2023.2754
Decroos FC, Stinnett SS, Heydary CS, Burns RE, Jaffe GJ. Reading Center Characterization of Central Retinal Vein Occlusion Using Optical Coherence Tomography During the COPERNICUS Trial. Transl Vis Sci Technol. 2013;2(7):7. doi:10.1167/tvst.2.7.7
Ogino K, Murakami T, Tsujikawa A, et al. Characteristics of optical coherence tomographic hyperreflective foci in retinal vein occlusion. Retina. 2012;32(1):77-85. doi:10.1097/IAE.0b013e318217ffc7
Tang W, Guo J, Zhuang X, et al. Wide-Field Swept-Source Optical Coherence Tomography Angiography Analysis of the Periarterial Capillary-Free Zone in Branch Retinal Vein Occlusion. Transl Vis Sci Technol. 2021;10(2):9. doi:10.1167/tvst.10.2.9
Lim HB, Kim MS, Jo YJ, Kim JY. Prediction of Retinal Ischemia in Branch Retinal Vein Occlusion: Spectral-Domain Optical Coherence Tomography Study. Invest Ophthalmol Vis Sci. 2015;56(11):6622-6629. doi:10.1167/iovs.15-17678
Dysli M, Rückert R, Munk MR. Differentiation of Underlying Pathologies of Macular Edema Using Spectral Domain Optical Coherence Tomography (SD-OCT). Ocul Immunol Inflamm. 2019;27(3):474-483. doi:10.1080/09273948.2019.1603313
Ruia S, Saxena S, Gemmy Cheung CM, Gilhotra JS, Lai TY. Spectral Domain Optical Coherence Tomography Features and Classification Systems for Diabetic Macular Edema: A Review. Asia Pac J Ophthalmol (Phila). 2016;5(5):360-367. doi:10.1097/APO.0000000000000218
Virgili G, Menchini F, Casazza G, et al. Optical coherence tomography (OCT) for detection of macular edema in patients with diabetic retinopathy. Cochrane Database Syst Rev. 2015;1:CD008081. Published 2015 Jan 7. doi:10.1002/14651858.CD008081.pub3
Li Y, Jin P, Lu J, Ma L, Qin X. Analysis of Optical Coherence Tomography in Acute versus Chronic Central Serous Chorioretinopathy. Optom Vis Sci. 2022;99(3):267-273. doi:10.1097/OPX.0000000000001846
Parameswarappa DC, Maltsev DS, Goud A, Singh SR, Chhablani J. Characteristics of Central Serous Chorioretinopathy without Leakage. J Curr Ophthalmol. 2021;33(2):152-157. Published 2021 Jul 5. doi:10.4103/joco.joco_35_20
Qu Y, Gong D, Yu W, Dong F. Characteristics of the Choriocapillaris Layer in Optical Coherence Tomography Angiography of Acute Central Serous Chorioretinopathy. Ophthalmic Surg Lasers Imaging Retina. 2017;48(12):1000-1005. doi:10.3928/23258160-20171130-07
Permadi AC, Djatikusumo A, Adriono GA. Optical coherence tomography in diagnosing polypoidal choroidal vasculopathy. Looking into the future: a systematic review and meta-analysis. Int J Retina Vitreous. 2022;8(1):14. Published 2022 Feb 28. doi:10.1186/s40942-022-00365-5
Yang L, Jonas JB, Wei W. Optical coherence tomography-assisted enhanced depth imaging of central serous chorioretinopathy. Invest Ophthalmol Vis Sci. 2013;54(7):4659-4665. Published 2013 Jul 12. doi:10.1167/iovs.12-10991
Yang L, Jonas JB, Wei W. Optical coherence tomography-assisted enhanced depth imaging of central serous chorioretinopathy. Invest Ophthalmol Vis Sci. 2013;54(7):4659-4665. Published 2013 Jul 12. doi:10.1167/iovs.12-10991

Table 1. Diagnostic accuracy of four Experiments and ophthalmologists

	CRVO(n=20)	DME(n=20)	CSC(n=20)	PCV(n=20)	Total(n=80)
Experiment 1	4(20%)	10(50%)	7(35%)	0(0%)	21(26%)
Experiment 2	4(20%)	4(20%)	13(65%)	1(5%)	22(28%)
Experiment 3	4(20%)	8(40%)	8(40%)	1(5%)	21(26%)
Experiment 4	8(40%)	9(45%)	4(20%)	0(0%)	21(26%)
Expert Evaluation	19(95%)	19(95%)	19(95%)	18(90%)	75(94%)

No competing interests reported.

Download PDF

Version 1

posted

You are reading this latest preprint version

Applications of GPT-4 for Accurate Diagnosis of Retinal Diseases Through Optical coherence tomography Image Recognition

Status:

Version 1

Abstract

Figures

INTRODUCTION

Methods

Experiment 1: Preliminary Diagnostic Assessment Using GPT-4's Image Recognition

Experiment 2: Refining the Diagnostic Process Based on Initial Observations

Experiment 3: Pattern Recognition through Composite Image Integration

Experiment 4: Augmenting Diagnoses with Annotated Textual Descriptors

Validation: Benchmarking GPT-4's Performance against Expert Diagnosis

Results

Experiment 1: Initial Diagnostic Evaluation Using GPT-4

Experiment 2: Modifying the Diagnostic Scope for GPT-4

Experiment 3: Diagnostic Approach Post Image Reference Integration

Experiment 4: Assimilation of Annotated Sample Images and Textual Descriptions

DISCUSSION

Conclusion

Abbreviations

Declarations

References

Table

Additional Declarations

Status:

Version 1