Rare eye diseases such as inherited retinal diseases (IRDs) are challenging to diagnose genetically. IRDs are typically monogenic disorders and represent a leading cause of blindness in children and working-age adults worldwide. A growing number are now being targeted in clinical trials, with approved treatments increasingly available. However, access requires a genetic diagnosis to be established sufficiently early. Critically, the timely identification of a genetic cause remains challenging. We demonstrate that a deep-learning algorithm, Eye2Gene, trained on the largest imaging dataset of patients with IRDs currently available, provides expert-level accuracy for genetic diagnosis for the 36 most common molecular causes (top-5 accuracy = 85.6%). This algorithm has been deployed online (app.eye2gene.com) and externally validated on data provided by four different clinical centers. Eye2Gene can facilitate access to diagnostic expertise, only currently available in a limited number of specialist centers globally, and thereby dramatically accelerate the genetic diagnostic odyssey.
Article
Eye2Gene: prediction of causal inherited retinal disease gene from multimodal imaging using deep-learning
https://doi.org/10.21203/rs.3.rs-2110140/v1
This work is licensed under a CC BY 4.0 License
Version 1
posted
You are reading this latest preprint version
Inherited retinal diseases (IRDs) are a group of rare monogenic conditions affecting 1 in 3000 people, with over 300 different IRD-associated genes identified to date. IRDs cause degeneration of the retina, the light-sensitive tissue at the back of the eye, responsible for vision. Some patients with IRDs may be profoundly visually impaired from birth, while others may find their peripheral and/or central vision progressively deteriorate over time. Cumulatively, IRDs are a leading cause of blindness in children and the commonest cause in the working-age population, with a significant psychological and socioeconomic impact 1,2.
Revealing the genetic cause of an IRD is a prerequisite to optimally determining prognosis, providing genetic counseling, and inclusion in gene-directed clinical trials. However, this genetic diagnosis remains elusive in more than 40% of cases on average according to studies conducted in the UK3–6, and the diagnostic yield is likely to be much lower in parts of the world where genetic testing is less widely available. This is mostly due to lack of resources to support genetic testing services, limited access to these, inefficiencies in their provision, or a shortage of specialists that can interpret the findings for molecular tests.
IRDs often have distinct phenotypic features, that clinicians learn to recognize using modern high-resolution retinal imaging technology that rapidly and non-invasively acquires images of the retina via a dilated pupil with minimal inconvenience to the patient.
These scans can be performed via a variety of imaging modalities such as fundus autofluorescence (FAF), infrared reflectance (IR) imaging, and spectral-domain optical coherence tomography (SD-OCT), each of which convey different information about retinal health and architecture (Supplementary Figure 1).
For example, FAF images can help identify patterns of both photoreceptor dysfunction and apoptosis by specifically detecting lipofuscin and related compounds based on their auto-fluorescent properties, which primarily accumulate in the retinal pigment epithelium (RPE) promoted by oxidative stress7. IR imaging helps identify melanin, a naturally occurring pigment that protects the eye, and melanin lipofuscin, a granule containing both melanin and lipofuscin. SD-OCT helps identify and characterize the multiple cellular layers of the retina such as the RPE, photoreceptors, outer nuclear and plexiform, inner nuclear and plexiform, ganglion cells, and nerve fiber layer. The cells most commonly affected by IRDs are the photoreceptors, with SD-OCT able to delineate the hyperreflective ellipsoid zone which is used to identify photoreceptor inner and outer segment layers and is often affected early in IRDs8.
This high-resolution in-depth multimodal information enables ophthalmologists to identify some gene-specific patterns of disease, allowing prediction of the disease-associated gene in some cases. However, given the sparsity of these diseases, the expertise and experience required to make accurate clinical diagnoses is not widely available and limited to a handful of specialists and hospitals who have developed this expertise in specialized clinics over several decades. Wider access to this expertise, could be deployed via an AI system trained to detect gene-specific patterns from retinal imaging scans.
Due to transformational improvements in imaging technology and a comprehensive genetic testing framework for IRDs embedded in specialist healthcare services over the last decade, there are now a sufficient number of molecularly characterized patients with detailed retinal phenotyping to build representative datasets for deep-learning. Moorfields Eye Hospital (UK) currently provides one of the most extensive datasets in the world9.
We have leveraged this resource to develop a deep-learning model, Eye2Gene, as an ensemble of deep convolutional neural networks, able to predict the causative IRD gene from the retinal scans of a patient with an IRD, acquired using the three aforementioned imaging modalities of FAF, IR and SD-OCT. Eye2Gene was trained on retinal scans acquired in patients with IRDs seen at Moorfields Eye Hospital (MEH) who had undergone genetic testing and where a confirmed genetic cause had been identified by an accredited diagnostic laboratory. Eye2Gene was internally validated on a held-out set of retinal scans from MEH, and externally validated on retinal scans from patients with IRDs from four different external clinical centers. Eye2Gene performance was also compared to that of expert clinicians.
Eye2Gene is an ensemble of a total of fifteen constituent Inception-v3 deep Convolutional Neural Networks, which takes one or more retinal scans of three different imaging modalities from a given patient and outputs a gene-level prediction score for 36 individual IRD genes. Collectively, these 36 genes cover over 80% of IRD cases in the European population10. The Eye2Gene model was trained on a total of 44,817 scans from 1,907 patients (3749 eyes, 6397 appointments) from Moorfields Eye Hospital 10, split into three different modalities: FAF (N=13,509), IR (N=20,098) and SD-OCT volumes (N=11,344). While SD-OCT is typically a volumetric imaging modality, we only used the central foveal b-scan, resulting in a single image per volume. For each of the three modalities, five distinct Inception V3 deep convolutional neural networks were trained using 5-fold cross validation on the development dataset, resulting in a total of fifteen neural networks with the same architecture but different network weights (Supplementary Figure 2). The five networks obtained for each of the three modalities were then combined into three modality-specific models via ensembling. The combination of these three models constitutes Eye2Gene. Given a single input scan of one of the three supported modalities, Eye2Gene ensembles the five networks corresponding to the modality of the scan to obtain a prediction. Given multiple scans from a single patient, the appropriate ensembles are applied to each scan in turn and the resulting predictions are combined to produce a single prediction for the patient by averaging over individual (post-softmax) predictions (Figure 1).
To evaluate Eye2Gene, we simulated the scenario of applying Eye2Gene to retinal scans acquired over one or multiple follow-up appointments per patient in our internal test set of 7605 retinal scans in 264 patients. When using only the scans acquired during the first appointment of each patient, Eye2Gene attained a top-1 accuracy of 66.7% (CI95%=61.0-72.3), top-5 accuracy of 85.6% (81.4-89.8), and an average per-gene receiver operator characteristic AUC of 0.935 (0.915-0.955). When all available scans per patient were used, Eye2Gene attained an accuracy of 66.7% (CI95%=61.0-72.3), top-5 accuracy of 89.4% (85.6-92.8), and mean per-gene AUROC of 0.950 (0.931-0.967).
Eye2Gene performance is comparable to experts
To contextualize Eye2Gene’s performance relative to human specialists, we asked nine ophthalmologists, specializing in IRDs, with varying years of experience, to predict the causative gene based on a single FAF image per patient across 50 different patients (chosen to get an approximately uniform representation of all genes in the dataset) from the internal held-out test set. In this task, ophthalmologists were asked to identify the correct diagnostic gene when shown an FAF scan of a patient with an IRD. On this task the ophthalmologists achieved an average accuracy of 17%, compared to 38% for Eye2Gene on the same images. We also asked a further four ophthalmologists to predict the causative genes for a selection of patients from the University Hospital Bonn external test set, asking them to choose from just the 11 genes in the dataset. On this task they achieved an average accuracy of 42%, compared to an accuracy of 49% for Eye2Gene (where Eye2Gene still had to choose from 36 genes).
The results of the human benchmarking by ophthalmologists are presented in Table 1. As expected, human performance tended to improve with the level of experience. We also applied Eye2Gene to the same datasets, restricted to single-image predictions (only using the 5 model FAF ensemble) for a fair comparison. Table 1 shows that the performance of Eye2Gene was generally better than any single human expert (except for ophthalmologist 10 who obtained 60.3% accuracy) and the frequency that the correct gene appeared in the top 5 predictions of Eye2Gene was comparable to the frequency that the correct gene appeared in all human guesses taken together.
Table 1: For comparison, ophthalmologists and Eye2Gene classified 50 FAF retinal scans of 50 patients results from Moorfields Eye Hospital internal held-out test dataset and also on a subset of Bonn external test datasets which included 73 FAF retinal scans from 37 patients. For the ophthalmologist majority vote, the most commonly predicted top guess by ophthalmologists was used. Ties were settled by selecting whatever guess was most favorable to the ophthalmologists.
|
Ophthalmologist |
Years of experience specializing in IRDs
|
Correct top guess |
Correct top-5 guesses |
|
Human benchmark internal Moorfields dataset (N=50) |
|||
|
1 |
< 10 |
6 (12%) |
13 (26%) |
|
2 |
< 10 |
10 (20%) |
15 (30%) |
|
3 |
< 10 |
12 (24%) |
16 (32%) |
|
4 |
< 10 |
9 (18%) |
15 (30%) |
|
5 |
> 10 |
6 (12%) |
11 (22%) |
|
6 |
> 10 |
8 (16%) |
15 (30%) |
|
7 |
> 10 |
10 (20%) |
15 (30%) |
|
Ophthalmologist majority vote |
16 (32%) |
31 (62%) |
|
|
Eye2Gene (FAF only) |
19 (38%) |
35 (70%) |
|
|
Human benchmark external Bonn dataset (N=73) |
|||
|
8 |
< 10 |
25 (34.3%) |
60 (82.2%) |
|
9 |
< 10 |
26 (35.6%) |
56 (76.7%) |
|
10 |
> 10 |
44 (60.3%) |
61 (83.6%) |
|
11 |
> 10 |
36 (49.3%) |
57 (78.1%) |
|
Ophthalmologist majority vote |
42 (57.5%) |
55 (75.3%) |
|
|
Eye2Gene (FAF only) |
36 (49.3%) |
60 (82.2%) |
|
Eye2Gene is generalizable across sites
To ensure that these results were repeatable across different sites, we also evaluated Eye2Gene on a further external test dataset of 1133 retinal scans from 236 patients from four different external IRD clinics, which included Oxford Eye Hospital (UK), Liverpool University Hospital (UK), University Hospital Bonn (Germany), and the Federal University of São Paulo (Brazil).
We ran Eye2Gene on scans from each patient and compared the prediction of Eye2Gene to the underlying gene diagnosis. Eye2Gene achieved an overall accuracy of 65.3% (CI95%=59.3-71.1), a top-5 accuracy of 86.4% (81.8-90.7), and an average per-gene receiver operator characteristic AUC of 0.930 (0.901-0.952). A breakdown of results per center is shown in Table 2 (with a full breakdown of results in Supplementary Table 8). This was slightly lower than on our internal test data. Since the distribution of genes in the external test data was different from the main MEH dataset (and hence our internal held-out test set), and the accuracy of Eye2Gene varies across genes (Supplementary Figure 5), we also recorded the per-gene prevalence (proportion of the dataset) and sensitivity for each gene at each of the external centers. To understand how much the difference in accuracy on the external data was due to differences in gene distribution, for each dataset we calculated the accuracy on our internal test data, reweighting contributions to match the gene distribution of the target external dataset. Doing this we found that, assuming consistent performance gene-for-gene, we would expect to see an overall accuracy of 67.2% on the external data, which was only slightly higher than the actual figure of 65.3%. This pattern was consistent across all four centers, where the actual accuracy was found to be a few percent lower than what the internal test data would predict.
Table 2: Overview of Eye2Gene results on the external data across the different centers. Gene-For-Gene Extrapolation from Internal Dataset is the sum of the per-gene sensitivity on our MEH internal test dataset multiplied by the gene frequency of the external dataset. It represents the accuracy we would expect to see based on the Moorfields results for a gene distribution identical to the target dataset.
|
Center |
Num Patients |
Num Unique Genes |
Accuracy |
Gene-For-Gene Extrapolation from Moorfields |
|
Oxford |
59 |
29 |
54.2% |
58.4% |
|
Liverpool |
35 |
15 |
57.1% |
59.9% |
|
Bonn |
127 |
11 |
70.1% |
70.3% |
|
São Paulo |
15 |
3 |
86.7% |
92.3% |
|
Combined (external only) |
236 |
30 |
65.3% |
67.2% |
|
Moorfields |
264 |
32 |
66.7% |
- |
|
Combined (all test data) |
500 |
36 |
66.0% |
- |
Eye2Gene’s data-driven approach identifies optimal image modality per gene
By breaking down Eye2Gene into its constituent networks we saw that, while the overall accuracy across each modality was relatively similar (Supplementary Figure 3), there were some notable differences for specific genes. For example, as shown in Figure 2, the FAF and SD-OCT networks were significantly better at detecting TIMP3 than the IR networks. This is expected as TIMP3-retinopathy is associated with an increase in signal in the peripheral macula in FAF imaging and drusen-like deposits in SD-OCT scans11. Similarly, the SD-OCT networks were better at detecting CRB1 than the IR and FAF networks. CRB1-retinopathy is often characterized by a thicker retina with less defined layers, something which cannot be distinguished in FAF or IR imaging12.
Including age and mode of inheritance improves Eye2Gene performance
In addition to retinal scans, the age of a patient at first presentation to the hospital and their family’s history of their IRD can be useful in determining the causative gene. Since this information is frequently available alongside patient scans, we looked at the effect of incorporating this additional information with the output of Eye2Gene. In particular, we tested the effect of combining patient age at first presentation (age), and mode of inheritance (MOI), which can be inferred from family history, with the output of Eye2Gene (Supplementary Figure 4). By training a linear classifier to predict the gene class based on the output of Eye2Gene, age, and MOI, we found this slightly improved the per patient accuracy from 68.1% to 71.6% (Supplementary Table 1).
Eye2Gene is accessible as an online web application
To demonstrate how Eye2Gene can be used to assist clinicians in diagnosing IRD patients, Eye2Gene is deployed as a web application online at app.eye2gene.com (currently invite-only). Users upload a series of scans of the supported modalities (SD-OCT, 55-degree FAF, and 30-degree IR) from a single patient and input basic case information such as age at presentation and MOI, if known. These images are then passed to the Eye2Gene model, which outputs a set of prediction scores for each of the 36 genes on each of the input scans. This information is aggregated into an overall prediction score for that case and presented to the user. An example of the information provided to the user is shown in Figure 3.
We present Eye2Gene, a deep learning model for classifying 36 causative genes in patients with inherited retinal diseases, using retinal scans acquired using three different imaging modalities. These scans can be obtained via non-invasive eye scans using widely available technology. We have comprehensively evaluated Eye2Gene on internal datasets, against human experts, and validated it on external datasets.
Only three previous studies have attempted to apply AI to IRDs but in significantly smaller datasets (N<400). Fujinami et al (2019) and Shah et al (2020) trained a classifier on SD-OCT and obtained an internal test AUC>90% for four classes (ABCA4, RP1L1, EYS and normal) (N=75) 13 and for two classes (Stargardt and normal) (N=93) 14, respectively. A more recent publication also obtained an AUC>90% for three broad IRD phenotypes (retinitis pigmentosa, Best disease and Stargardt disease) from FAF images in a small sample (N=389) 15 . By comparison, Eye2Gene, due to the benefit of having been trained on the world's largest dataset of IRD patients (N=1,907), has potential for much broader clinical utility, as is the first, to support as many as 36 gene diagnoses, multiple imaging modalities, and to have conducted external validation.
The top-1 accuracy of 66.7% of Eye2Gene should be put into the context of the challenging problem of identifying the causative variant from a selection of 36 different possible candidate genes, based on retinal scans alone. This is a task that typically requires years of training for clinicians to perform. Eye2Gene correctly identifies the correct gene on the first try for two thirds of patients, and within the top five guesses (likely to be a reasonable number of possibilities for a clinician to eliminate) for 85% of patients. While this is made easier by the skewed nature of the gene distribution (and hence favors performing well on the more common genes), we note that the naive baseline of always selecting the most common gene (i.e. ABCA4) achieves an accuracy of only 30.3%.
Additionally, given our human benchmark, we have shown that Eye2Gene is at least as good, and often significantly better, than IRD experts at this specific task. Although this task is a long way from capturing the full scope of standard clinical practice, since clinical decisions are seldom (if ever) based on imaging alone, it demonstrates the strength of Eye2Gene at assessing retinal scans and its potential in assisting even expert ophthalmologists.
Despite these promising results, we recognize that there remain a number of important limitations of Eye2Gene. These include performance linked to attributes of the training data such as gene class prevalence (Supplementary Figure 5), imbalanced patient distribution (Supplementary Figure 6), under-represented ethnicities (Supplementary Table 2), variation in image quality (Supplementary Figure 7), limitation on type of supported scan (currently restricted to Heidelberg Spectralis), as well as limitations linked to scope of clinical application (limited to 36 gene classes), and cannot be used for screening presence/absence of IRDs. Another well-known and important limitation of deep-learning classification networks is that their interpretability remains challenging and that approaches such as saliency maps are not, as of now, sufficiently reliable for medical decisions 16,17. This was confirmed by our own findings (Supplementary Figure 8). Hence Eye2Gene remains in active development to address and alleviate these limitations.
Although Eye2Gene is the first AI system to demonstrate significant potential for clinical utility in the challenging area of IRDs, Eye2Gene predictions will not replace the need for genetic testing or counseling at specialized IRD centers, especially when a treatment such as a gene therapy is to be administered based on a genetic diagnosis. Therefore, currently, the main clinical application of Eye2Gene is in accelerating the time to diagnosis since it can point to a specific gene to be investigated.
It is anticipated Eye2Gene could aid the vast majority of ophthalmologists and other vision healthcare workers who are not specialized in rare IRDs, to indicate when molecular testing would be worthy of consideration. Eye2Gene gene predictions can also be used for scoring and therefore prioritizing genetic variants where the gene matches the phenotype 18. In disorders with a highly distinct phenotype, the matching phenotype PP4 criterion in the ACMG guidelines provides a higher level of evidence in variant classification 19 20, which might be crucial for the transition from a variant of unknown significance to a likely pathogenic variant 21. This approach has been shown to improve diagnostic yield for other highly heterogeneous disorders such as syndromic intellectual disability 22.
In conclusion, Eye2Gene shows that AI is a promising approach to aid in accelerating genetic diagnosis for patients with IRDs, something which is not only important for improving patient experience, and reducing associated overheads, but is likely to become especially important due to the growing number of treatable IRDs where a rapid diagnosis can lead to an improved outcome for the patient 23–25.
Dataset quality control and preparation
The Moorfields Eye Hospital (MEH) inherited retinal disease (IRD) cohort was previously described by Pontikos et al. (2020) 10 and encompasses 4236 individuals with IRDs caused by variants in 135 distinct genes, of which 452 individuals (with variants in 66 genes) were younger than 18 years of age as of 2019-08-02. Patients with an IRD and a confirmed genetic diagnosis by an accredited genetic diagnosis laboratory were identified and information about the genetic diagnosis, the age at presentation and mode of inheritance was exported from the MEH electronic health record (OpenEyes) using a SQL query on the Microsoft SQL Server hospital data warehouse database.
Images were exported from the MEH Heidelberg Imaging (Heyex) database (Heidelberg Engineering, Heidelberg, Germany) for all patients with an IRD, based on their hospital number, for records between 2006-06-05 and 2018-04-05. This resulted in a dataset of 1,196,038 images from 2871 IRD patients with disease-causing variants in 132 distinct genes. The quality control (QC) process is explained in Supplementary Note 1 and Supplementary Figure 9 where the number of images rejected at each stage of the process is provided. To ensure sufficient data for training and testing, we restricted our datasets to only genes with at least 5 images per gene per fold in each of the three modalities (after QC), leaving 36 individual genes. The distribution of all 132 genes is presented in Supplementary Figure 10 and the selected 36 genes are given in Table 3. Following the QC and gene selection process, 15,692 FAF, 23,631 IR, and 13,099 SD-OCT images remained across 2171 patients in 36 distinct genes.
Post QC, these patients were split into a ‘development’ set of 1907 patients, and a held-out internal test set of 264 patients (Supplementary Figure 2). The development set was used to train our 15 constituent Eye2Gene networks, and to investigate network-level properties and ensembling approaches, while the internal test set was kept separate to enable testing of the final Eye2Gene model.
Table 3: Number of patients and images per gene and modality (FAF, IR and SD-OCT) for the 36 genes included in the Eye2Gene development dataset.
|
|
FAF |
IR |
SD-OCT |
Patients with images from all 3 modalities |
Age at first presentation range |
|||
|
Gene |
Patients |
Images |
Patients |
Images |
Patients |
Images |
||
|
ABCA4 |
521 |
4309 |
575 |
5708 |
535 |
3527 |
472 |
6-81 |
|
USH2A |
164 |
1525 |
186 |
1754 |
162 |
1175 |
142 |
9-80 |
|
PRPH2 |
118 |
907 |
133 |
1201 |
127 |
850 |
109 |
16-75 |
|
RPGR |
111 |
1002 |
133 |
1463 |
120 |
907 |
98 |
4-79 |
|
BEST1 |
89 |
990 |
106 |
1291 |
101 |
763 |
81 |
3-80 |
|
CHM |
79 |
786 |
87 |
972 |
83 |
599 |
75 |
5-71 |
|
RS1 |
71 |
718 |
88 |
1097 |
81 |
567 |
69 |
4-71 |
|
RP1 |
69 |
587 |
68 |
574 |
64 |
459 |
62 |
11-78 |
|
RHO |
58 |
444 |
70 |
426 |
62 |
311 |
49 |
7-89 |
|
PRPF31 |
38 |
362 |
43 |
337 |
36 |
248 |
32 |
7-73 |
|
MYO7A |
35 |
326 |
37 |
342 |
31 |
219 |
29 |
5-73 |
|
CRB1 |
30 |
328 |
34 |
454 |
25 |
202 |
29 |
7-57 |
|
EYS |
29 |
290 |
30 |
298 |
29 |
203 |
25 |
5-69 |
|
CNGB3 |
29 |
224 |
63 |
1829 |
54 |
602 |
24 |
3-74 |
|
PROM1 |
28 |
204 |
30 |
368 |
24 |
169 |
23 |
7-72 |
|
EFEMP1 |
26 |
172 |
30 |
382 |
23 |
147 |
18 |
21-70 |
|
TIMP3 |
22 |
153 |
21 |
246 |
17 |
131 |
17 |
3-46 |
|
RDH12 |
21 |
169 |
28 |
589 |
20 |
131 |
17 |
36-66 |
|
CNGA3 |
19 |
133 |
40 |
843 |
31 |
285 |
16 |
3-55 |
|
CACNA1F |
17 |
133 |
23 |
239 |
16 |
85 |
15 |
13-62 |
|
RP2 |
17 |
125 |
20 |
204 |
17 |
104 |
15 |
5-71 |
|
GUCY2D |
17 |
115 |
22 |
323 |
18 |
109 |
14 |
11-69 |
|
RPE65 |
17 |
93 |
31 |
510 |
26 |
158 |
14 |
4-74 |
|
BBS1 |
16 |
183 |
18 |
238 |
16 |
127 |
13 |
13-77 |
|
NR2E3 |
14 |
216 |
15 |
168 |
14 |
99 |
13 |
25-71 |
|
MERTK |
14 |
163 |
14 |
181 |
12 |
90 |
13 |
7-70 |
|
CRX |
14 |
113 |
17 |
242 |
14 |
95 |
12 |
3-73 |
|
CERKL |
13 |
137 |
13 |
162 |
9 |
71 |
12 |
5-52 |
|
MTTL1 |
13 |
71 |
14 |
67 |
14 |
57 |
12 |
6-60 |
|
OPA1 |
13 |
64 |
40 |
377 |
35 |
118 |
11 |
5-62 |
|
PDE6B |
11 |
157 |
13 |
140 |
12 |
106 |
10 |
26-65 |
|
RP1L1 |
11 |
82 |
14 |
184 |
12 |
86 |
9 |
7-57 |
|
CYP4V2 |
11 |
80 |
13 |
103 |
10 |
71 |
9 |
18-67 |
|
PRPF8 |
10 |
111 |
11 |
99 |
10 |
82 |
8 |
4-52 |
|
CDH23 |
9 |
175 |
10 |
92 |
9 |
76 |
8 |
14-67 |
|
KCNV2 |
8 |
45 |
13 |
137 |
13 |
70 |
8 |
6-76 |
In addition to the MEH data described above we also obtained images from a further four centers to enable external validation of Eye2Gene. Oxford Eye Hospital (UK) provided a sample of 346 scans from 59 patients with distinct gene diagnoses in 29 different genes. The University Eye Hospital of Liverpool (UK) provided a sample of 210 scans from 35 patients with distinct gene diagnoses in 15 different genes. The University Eye Hospital Bonn (Germany) provided a sample of 473 scans from 127 patients with distinct gene diagnoses in 11 different genes. The Federal University of Sao Paulo (Brazil) provided a sample of 104 scans from 15 patients with distinct gene diagnoses in 3 different genes. For reference, the MEH (UK) internal test dataset consisted of 1900 scans from 264 patients across 32 gene diagnoses. Further break-down by dataset is available in Table 4 (with further break-down by gene in Supplementary Table 7). Patients and scans were selected by clinicians based at these different clinics, who were instructed to select patients with a confirmed genetic diagnosis and who had a number of scans available for each modality. For each patient a selection of scans was shared, typically consisting of one scan per-patient per-eye per-modality, along with their genetic diagnosis.
Table 4: Number of patients, scans and genes in internal (Moorfields) and external (Oxford, Liverpool, Bonn and São Paulo) test datasets.
|
Centre |
Number Patients |
Number of Images |
Number of Genes |
|||
|
FAF |
IR |
SD-OCT |
Total |
|||
|
Moorfields |
264 |
733 |
692 |
475 |
1900 |
32 |
|
Oxford |
59 |
106 |
120 |
120 |
346 |
29 |
|
Liverpool |
35 |
70 |
70 |
70 |
210 |
15 |
|
Bonn |
127 |
241 |
0 |
232 |
473 |
11 |
|
São Paulo |
15 |
16 |
36 |
52 |
104 |
3 |
|
Total |
500 |
1166 |
918 |
949 |
3033 |
36 |
Model training
Patients in the development set were divided into 5 approximately equal-sized patient subsets, taking care to ensure an approximately even split of images across each of the different genes in each of the three different modalities. For each of the three modalities, images from each of the five patient subsets were used as the folds in a 5-fold cross-validation set-up. On each combination of four patient subsets a 36-class Convolutional Neural Network (CNN) was trained for 100 epochs (passes over the entire dataset), which was found to be sufficient for the network to converge in most settings (Supplementary Figure 11). An Inception-v3 architecture26 initialized with pre-trained ImageNet weights 27 was used, where the final output layer was replaced by a linear layer with 36 outputs, including drop-out, followed by softmax normalization. The Inception-v3 architecture was chosen as it was the smallest model size amongst a selection of similarly performing architectures applied in other ophthalmology contexts 28. Cross entropy loss was used for the loss function, using additional class-weighting inversely proportional to gene frequency in the dataset, where the labels were given by the gene diagnosis of the underlying patients. For training, the Adam optimizer was used with the default parameters used in the Keras library (β1=0.9, β2=0.999). To obtain values for other settings, hyper-parameter tuning was conducted via random sampling over 10 trials on an 80/20 train/validation split on the first cross-validation training fold (Supplementary Table 3). Following hyper-parameter tuning, a batch size of 128, learning rate of 0.0001 and dropout probability of 0% were found to be optimal for all three modalities.
These hyper-parameters were used for the training of the networks across all experiments for all three modalities. To avoid overfitting to the training data, data augmentation techniques were applied: A number of realistic image transformations were applied automatically to the input data during training (Supplementary Figure 12).
Evaluation
The output prediction of Eye2Gene is obtained by combining the predictions of its fifteen constituent networks using an ensemble approach. For each single retinal scan of a specific modality, a 5-model ensemble is applied by averaging the output probabilities per gene (Supplementary Figure 13.B). Given a collection of retinal scans from a single patient, the appropriate ensemble model corresponding to each scan’s modality is applied, then the average predictions across all scans in the collection is taken and used as the final prediction for the patient (Supplementary Figure 13.D). This approach was applied across all available scans per patient-appointment (defined as all scans from a given patient at a given date), per first-appointment (all scans from a given patient only from the first date they appear within the dataset), and on a per-patient level (using all the available scans per patient across all appointments), as well as on the individual images.
The model predictions were then compared against the underlying gene diagnoses for each patient to compute the overall accuracy of the model on the test data, the top-k accuracy (the proportion of images where the correct gene was within the highest k predictions of the network) for k=2,3,5,10, and the average per-class Area Under the Receiver Operator Characteristic curve (AUROC). This was repeated over the different datasets and combinations of datasets as appropriate (Supplementary Note 2).
Confidence intervals for accuracies and AUCs on the development data were obtained by taking the standard deviations of the accuracy/AUC for each of the three modalities across the five networks, and estimating a 95% confidence interval via the expression 
while confidence intervals for accuracies and AUCs on the test data were obtained by bootstrapping over 10,000 resamplings and taking the 2.5th and 97.5th percentiles.
Incorporating age and mode of inheritance into the prediction
Age and mode of inheritance (MOI) are important factors to consider in the genetic diagnosis of IRDs as these provide strong prior evidence towards a specific gene. For example, if the patient is young and the MOI X-linked this may suggest RP2 as an associated gene (Supplementary Figure 14 and 15). To incorporate these non-imaging features to the output of Eye2Gene, a model stacking approach was taken 29 where a linear classifier was trained on the combined predictions from Eye2Gene and the additional input features, which then produced a single prediction based on the original model prediction and the additional features.
Age was treated as a numeric variable and MOI as a one-hot vector with five classes: recessive, dominant, X-linked, mitochondrial, and unknown. For each scan/appointment from our five patient sets, we concatenated these features with the final output of the corresponding network (where each gene prediction was treated as an individual feature) and used these, along with the true gene labels, to fit a logistic regression classifier (without regularization). To test the resulting combined model we applied this trained model to our internal test set, using the ensemble predictions from the full Eye2Gene model concatenated with age and MOI, and use the resulting predictions as our new gene-level predictions, performing the same top-k and mean per-gene AUC analysis as for the other experiments. We also performed a cross-validation analysis based on the five folds in our development set where we used the model outputs on 4 folds as train data, and tested on the remaining fold, which we repeated for each fold.
Human benchmarking
In order to contextualize the performance of Eye2Gene compared to human experts, a challenge dataset of 50 FAF images from patients sampled from the MEH held-out set was created, with an approximately uniform distribution across genes. FAF was selected since it is the modality that is the most commonly used in IRD clinics. We asked seven ophthalmologists from Bonn and Oxford, specializing in IRDs, with varying levels of experience to predict the causative gene based only on the images provided. For each image, each ophthalmologist was asked to name the gene they thought was most likely out of the Eye2Gene list of 36, and were allowed to suggest a further four genes. In some cases, ophthalmologists felt unable to assess certain images based on poor image quality, or did not find it necessary to identify a further four genes. We also asked a further 4 ophthalmologists specializing in IRDs from MEH to perform a similar grading on a selection of images from 37 patients from the Bonn external test dataset. However, in this case, doctors were asked to choose from only the 11 genes within the external test data.
Ethics
This research was approved by the IRB and the UK Health Research Authority Research Ethics Committee (REC) reference (22/WA/0049) “Eye2Gene: accelerating the diagnosis of inherited retinal diseases” Integrated Research Application System (IRAS) (project ID: 242050). All research adhered to the tenets of the Declaration of Helsinki.
Acknowledgement
The research was supported by a grant from the National Institute for Health Research (NIHR) Biomedical Research Centre (BRC) at Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology.
NP and WAW are currently funded by a NIHR AI Award (AI_AWARD02488).
NP was also previously funded by Moorfields Eye Charity Career Development Award (R190031A).
This project was also supported by a generous donation by Stephen and Elizabeth Archer in memory of Marion Woods. The hardware used for analysis was supported by the BRC Challenge Fund (BRC3_027). We also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Author Contributions
NP, WAW and AV devised the experiment, analyzed the data and wrote the manuscript. BJ, MAI, AH, IM, KH, MG, MP, MS, JY, SKW, MDV, TACG, NK, JF, QN, BL, CT, FB, GA, KF, JS, SM, SD, FH, KB, ARW, OM, and PW analyzed the data. All co-authors critically reviewed the manuscript.
Competing Interests statement
NP and IM are co-founders and directors of Phenopolis Ltd.
1. Galvin, O. et al. The Impact of Inherited Retinal Diseases in the Republic of Ireland (ROI) and the United Kingdom (UK) from a Cost-of-Illness Perspective. Clin. Ophthalmol. 14, 707–719 (2020).
2. Hanany, M., Rivolta, C. & Sharon, D. Worldwide carrier frequency and genetic prevalence of autosomal recessive inherited retinal diseases. Proc. Natl. Acad. Sci. U. S. A. 117, 2710–2716 (2020).
3. Yohe, S. et al. Prevalence of mutations in inherited retinal diseases: A comparison between the United States and India. Mol Genet Genomic Med 8, e1081 (2020).
4. Jiman, O. A. et al. Diagnostic yield of panel-based genetic testing in syndromic inherited retinal disease. Eur. J. Hum. Genet. 28, 576–586 (2020).
5. Sheck, L. H. N. et al. Panel-based genetic testing for inherited retinal disease screening 176 genes. Mol Genet Genomic Med 9, e1663 (2021).
6. 100,000 Genomes Project Pilot Investigators et al. 100,000 Genomes Pilot on Rare-Disease Diagnosis in Health Care - Preliminary Report. N. Engl. J. Med. 385, 1868–1880 (2021).
7. Heiferman, M. J. & Fawzi, A. A. DISCORDANCE BETWEEN BLUE-LIGHT AUTOFLUORESCENCE AND NEAR-INFRARED AUTOFLUORESCENCE IN AGE-RELATED MACULAR DEGENERATION. Retina 36 Suppl 1, S137–S146 (2016).
8. Tanna, P. et al. Cross-Sectional and Longitudinal Assessment of the Ellipsoid Zone in Childhood-Onset Stargardt Disease. Transl. Vis. Sci. Technol. 8, 1 (2019).
9. Méjécase, C. et al. Practical guide to genetic screening for inherited eye diseases. Ther Adv Ophthalmol 12, 2515841420954592 (2020).
10. Pontikos, N. et al. Genetic basis of inherited retinal disease in a molecularly characterised cohort of over 3000 families from the United Kingdom. Ophthalmology (2020).
11. Rahman, N., Georgiou, M., Khan, K. N. & Michaelides, M. Macular dystrophies: clinical and imaging features, molecular genetics and therapeutic options. Br. J. Ophthalmol. 104, 451–460 (2020).
12. Kousal, B. et al. Phenotypic features of CRB1-associated early-onset severe retinal dystrophy and the different molecular approaches to identifying the disease-causing variants. Graefes Arch. Clin. Exp. Ophthalmol. 254, 1833–1839 (2016).
13. Fujinami, Y. Y. et al. Prediction of Causative Genes in Inherited Retinal Disorders From Spectral-domain Optical Coherent Tomography Utilizing Deep Learning Techniques. Invest. Ophthalmol. Vis. Sci. 60, 2950–2950 (2019).
14. Shah, M., Roomans Ledo, A. & Rittscher, J. Automated classification of normal and Stargardt disease optical coherence tomography images using deep learning. Acta Ophthalmol. 98, e715–e721 (2020).
15. Miere, A. et al. Deep Learning-Based Classification of Inherited Retinal Diseases Using Fundus Autofluorescence. J. Clin. Med. Res. 9, (2020).
16. Arun, N. et al. Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging. Radiol Artif Intell 3, e200267 (2021).
17. Salahuddin, Z., Woodruff, H. C., Chatterjee, A. & Lambin, P. Transparency of deep neural networks for medical image analysis: A review of interpretability methods. Comput. Biol. Med. 140, 105111 (2021).
18. Cipriani, V. et al. An Improved Phenotype-Driven Tool for Rare Mendelian Variant Prioritization: Benchmarking Exomiser on Real Patient Whole-Exome Data. Genes 11, (2020).
19. Gattorno, M. et al. Classification criteria for autoinflammatory recurrent fevers. Ann. Rheum. Dis. 78, 1025–1032 (2019).
20. Tavtigian, S. V. et al. Modeling the ACMG/AMP variant classification guidelines as a Bayesian classification framework. Genet. Med. (2018) doi:10.1038/gim.2017.210.
21. Richards, S. et al. Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genet. Med. 17, 405–424 (2015).
22. Hsieh, T.-C. et al. PEDIA: prioritization of exome data by image analysis. Genet. Med. 21, 2807–2814 (2019).
23. Maguire, A. M. et al. Age-dependent effects of RPE65 gene therapy for Leber’s congenital amaurosis: a phase 1 dose-escalation trial. Lancet 374, 1597–1605 (2009).
24. Maguire, A. M. et al. Safety and efficacy of gene transfer for Leber’s congenital amaurosis. N. Engl. J. Med. 358, 2240–2248 (2008).
25. Acland, G. M. et al. Gene therapy restores vision in a canine model of childhood blindness. Nat. Genet. 28, 92–95 (2001).
26. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the Inception Architecture for Computer Vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016) doi:10.1109/cvpr.2016.308.
27. Deng, J. et al. ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition (2009) doi:10.1109/cvpr.2009.5206848.
28. Ting, D. S. W. et al. Artificial intelligence and deep learning in ophthalmology. Br. J. Ophthalmol. 103, 167–175 (2019).
29. Wolpert, D. H. Stacked generalization. Neural Netw. 5, 241–259 (1992).
30. Yang, L. et al. Genetic Spectrum of EYS-associated Retinal Disease in a Large Japanese Cohort: Identification of Disease-associated Variants with Relatively High Allele Frequency. Sci. Rep. 10, 5497 (2020).
31. Gupta, K., Walia, G. S. & Sharma, K. Quality based adaptive score fusion approach for multimodal biometric system. Applied Intelligence 50, 1086–1099 (2020).
32. Mittal, A., Moorthy, A. K. & Bovik, A. C. No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21, 4695–4708 (2012).
Yes there is potential Competing Interest.
Nikolas Pontikos, William Woof, Mital Shah and Jennifer Furman are funded by a National Institute for Health Research AI Award (AI_AWARD02488) to develop Eye2Gene.
- flatPontikosspc.pdf
Software Checklist
- flatPontikosrs.pdf
Reporting Summary
- flatPontikosepc.pdf
Editorial Policy Checklist
- SupplementaryMaterial.docx