A self-administered, artificial intelligence (AI) platform for cognitive assessment in multiple sclerosis (MS)

doi:10.21203/rs.2.10768/v2

Download PDF

Research article

A self-administered, artificial intelligence (AI) platform for cognitive assessment in multiple sclerosis (MS)

https://doi.org/10.21203/rs.2.10768/v2

This work is licensed under a CC BY 4.0 License

Journal Publication

published 18 May, 2020

Read the published version in BMC Neurology →

You are reading this older preprint version

Read the latest preprint version →

Background Cognitive impairment is common in patients with MS. Accurate and repeatable measures of cognition have the potential to be used as a marker of disease activity. Methods We developed a 5-minute computerized test to measure cognitive dysfunction in patients with MS. The proposed test –named Integrated Cognitive Assessment (ICA)– is self-administered and language-independent. 91 MS patients and 83 healthy controls (HC) took part in substudy 1, in which each participant took the ICA test and the Brief International Cognitive Assessment for MS (BICAMS). We assessed ICA’s test-retest reliability, its correlation with BICAMS, its sensitivity to discriminate patients with MS from the HC group, and its accuracy in detecting cognitive dysfunction. In substudy 2, we recruited 48 MS patients, and examined the association between the level of serum neurofilament light (NfL) in these patients and their ICA scores. Results ICA demonstrated excellent test-retest reliability (r=0.94), with no learning bias (i.e. no significant practice effect); and had high level of convergent validity with BICAMS. ICA was sensitive in discriminating the MS patients from the HC group, and demonstrated a high accuracy (AUC = 95%) in discriminating cognitively normal from cognitively impaired participants. Additionally, we found a strong association (r=-0.79) between ICA score and the level of NfL in MS patients. Conclusions ICA has the potential to be used as a digital biomarker for assessment and monitoring of cognitive performance in MS patients. In comparison to standard cognitive tools for MS (e.g. BICAMS), ICA is shorter in duration, does not show a learning bias, is independent of language, and takes advantage of artificial intelligence (AI) to identify cognitive status of patients more accurately. Being a digital test, it further has the potential for easier electronic health record or research database integration.

Neurology

Multiple sclerosis

BICAMS

digital biomarkers

Integrated Cognitive Assessment (ICA)

language-independent

Artificial Intelligence (AI)

Multiple sclerosis (MS) can cause demyelination and neurodegeneration in patients ¹. Therefore, cognitive dysfunction is common in MS patients (40-70% of these patients are reported to have cognitive impairment²), and is associated with a higher risk of disease progression in the subsequent years². Cognitive impairment can have significant negative impacts on several domains of activities of daily living, such as social functioning, employment ³ and driving ⁴. Furthermore, early detection of cognitive impairment in MS might be helpful in the identification of patients at high risk of disability progression and poor clinical outcome⁵.

The Brief International Cognitive Assessment for MS (BICAMS)^6,7 is a pen-and-paper based cognitive assessment battery for detecting cognitive dysfunction in MS patients. The BICAMS battery includes tests of mental processing speed and memory, and takes about 15 to 20 minutes to administer and score. We used BICAMS in this study as a standard reference test to measure the efficacy of the proposed ICA test in detecting cognitive impairment in MS patients. Despite the prevalence of cognitive impairment and its negative impact on patients’ lives, cognitive assessment is not routinely done for MS patients in clinical sites⁸. This is in part due to the extra time and resources needed to administer pen-and-paper tests, and the manual nature of scoring and integrating the test results with medical health records.

Blood neurofilament light chain (NfL) has been shown to be a valuable fluid biomarker of MS disease activity and treatment response⁹. And is associated with clinical and MRI-related measures of disease activity and neuroaxonal damage¹⁰. Therefore, to establish the validity and utility of the ICA test as a potential digital biomarker, we compared ICA test results against participants’ level of serum NfL.

Cognition has the potential to be used as a marker of disease progression or treatment efficacy in MS ^11,12. When patients report a cognitive problem, they are describing a change in function from a previous level; however, the majority of cognitive tests, due to a learning bias ^13,14 , cannot be used for frequent monitoring of cognitive performance. On the other hand, neuroimaging and fluid biomarkers of disease activity ^15–17, while more accurate, are less suitable for frequent monitoring of disease progression, and more difficult to integrate into routine clinical practice. Here, we propose an AI-assisted digital biomarker of cognitive function, appropriate for monitoring the disease activity.

It is documented that the afferent visual system is highly vulnerable to MS¹⁸. Furthermore, deficit in information processing speed (IPS) is the most prevalent cognitive impairment in MS, and can affect speed of sensory, motor and cognitive processes ¹⁹. We therefore designed an iPad-based rapid visual categorization task^20–22, called the Integrated Cognitive Assessment (ICA), that primarily assesses IPS in visuo-motor domains. The task is designed to give a sensitive, repeatable measure of IPS, and is additionally shown to be correlated with other cognitive domains, such as visual memory and visuospatial²³. The test is software-based, self-administered and is shown to have little dependency on participant’s language, and is not confounded by participants’ different level of education ²³.

In this study, we investigate ICA’s validity as a digital biomarker for assessing cognitive performance in MS. We report results for convergent validity between BICAMS and ICA, test-retest reliability, ICA correlation with serum NfL, effect of repeated exposure to the tests (i.e. learning effect), sensitivity to detecting cognitive impairment, and the accuracy of the ICA test in discriminating MS patients from healthy controls (HC).

2.1 ICA test description and the scientific rationale behind the test

The ICA test is a rapid visual categorization task with backward masking ^20,21,24. The test takes advantage of the human brain’s strong reaction to animal stimuli ^25,26. One hundred natural images (50 animal and 50 non-animal) are carefully selected, with various levels of difficulty, and are presented to the participants in rapid succession. Images are presented at the center of the screen at 7° visual angle. In some images the head or body of the animal is clearly visible to the participants, which makes it easier to detect. In other images the animals are further away or otherwise presented in cluttered environments, making them more difficult to detect. Few sample images are shown in Figure 1. We used grayscale images to remove the possibility of some typical color blindness affecting participants’ results. Furthermore, color images can facilitate animal detection solely based on color^27,28, without fully processing the shape of the stimulus. This could have made the task easier and less suitable for detecting less severe cognitive dysfunctions.

The strongest categorical division represented in the human higher level visual cortex appears to be that between animates and inanimates ^29,30. Studies also show that on average it takes about 100ms to 120ms for the human brain to differentiate animate from inanimate stimuli ^26,31,32. Following this rationale, each image is presented for 100 ms followed by a 20 millisecond inter-stimulus interval (ISI), followed by a dynamic noisy mask (for 250 ms), followed by subject’s categorization into animal vs non-animal (Figure 1). Shorter periods of ISI can make the animal detection task more difficult and longer periods reduce the potential use for testing purposes as it may not allow for the detection of less severe cognitive impairments. The dynamic mask is used to remove (or at least reduce) the effect of recurrent processes in the brain ^33,34. This makes the task more challenging by reducing the ongoing recurrent neural activity that could artificially boost subject’s performance; it further reduces the chances of learning the stimuli. For more information about rapid visual categorization tasks refer to Mirzaei et al., (2013) ²¹.

The ICA test starts with a different set of 10 test images (5 animal, 5 non-animal) to familiarize participants with the task. These images are later removed from further analysis. If participants perform above chance (>50%) on these 10 images, they will continue to the main task. If they perform at chance level (or below), the test instructions will be presented again, and a new set of 10 introductory images will follow. If they perform above chance in this second attempt, they will progress to the main task. If they perform below chance for the second time the test is aborted

Backward masking: To construct the dynamic mask, following the procedure in (Bacon-Macé and colleagues, 2005) ^20,21 , a white noise image was filtered at four different spatial scales, and the resulting images were thresholded to generate high contrast binary patterns. For each spatial scale, four new images were generated by rotating and mirroring the original image. This leaves us with a pool of 16 images. The noisy mask used in the ICA test was a sequence of 8 images, chosen randomly from the pool, with each of the spatial scales to appear twice in the dynamic mask.

2.2 Brief International Cognitive Assessment for MS (BICAMS)

The BICAMS battery consists of three standard pen-and-paper tests, measuring speed of information processing, visuo-spatial memory and verbal learning.

Symbol Digit Modalities Test (SDMT): The SDMT is designed to assess speed of information processing, and takes about 5 minutes to administer ³⁵.

California Verbal Learning Test -2^nd edition (CVLT-II): The CVLT-II test ^36,37 begins with the examiner reading a list of 16 words. Participants listen to the list and then report as many of the items as they can recall. Five learning trials of the CVLT-II are used in BICAMS⁶, which takes about 10 minutes to administer.

Brief Visual Memory Test–Revised (BVMT-R): The BVMT-R test assesses visuo-spatial memory ^38,39. In this test, in three consecutive trials, six abstract shapes are presented to the participant for 10 seconds. After each trial, the display is removed from view and patients are asked to draw the stimuli via pencil on paper manual responses. The test takes about 5 minutes to administer. 6 shapes are presented for 10 sec over 3 consecutive trials; after each trial participants are asked to draw the stimuli.

2.3 Participants

In total, 174 volunteers took part in substudy1 (Table 1): 91 patients diagnosed with multiple sclerosis (MS), and 83 age, gender and education matched healthy controls. 48 MS patients took part in substudy2 (Table 2). Of all participants 25 attended both substudies. Participants’ age varied between 18 and 65. The study was conducted according to the Declaration of Helsinki and approved by the local ethics committee at Royan Institute. Informed written consent was obtained from all participants. Patient participants were consecutively recruited from the outpatient clinic of the Aria Medical Complex for MS in Tehran, Iran. Patients were diagnosed by a consultant neurologist according to the McDonald diagnostic criteria (2010 revision)⁴⁰. Healthy controls (HC) were recruited through local advertisement.

Participants’ exclusion criteria included: Severe depression and other major psychiatric comorbidities, presence of neurological disorders and medical illness that independently affect brain function and cognition (other than MS for the patient group), visual problems that cannot be corrected with eye-glasses such that the problem prevents participant from reading, upper limb motor dysfunction, history of epileptic seizures, history of illicit substance and/or alcohol dependence.

For each participant, clinical characteristics of MS subtype, information on age, education and gender were also collected. We quantified participant disability and disability progression over time by utilising the Expanded Disability Status Scale (EDSS).

For the purposes of this study, patients with sever abnormality in at least one of the BICAMS sub-tests (defined as 2SD below the norm) or with mild abnormality (defined as 1SD below the norm) in at least two sub-tests of BICAMS were identified as cognitively impaired.

2.4 Study procedures

Substudy 1: 174 participants (Table 1) took the iPad-based ICA test and the pen-and-paper BICAMS test, administered in random order. The same researchers who administered the BICAMS, directed participants on how to take the iPad ICA test. In this substudy we investigated convergent validity of the ICA test with BICAMS, ICA’s test-retest reliability and the sensitivity and specificity of the ICA platform in detecting cognitive impairment in MS.

To measure test-retest reliability for the ICA test, a subset of 21 MS and 22 HC participants were called back after five weeks (± 15 days) to take the ICA test as well as the SDMT. The subset’s characteristics were similar to the primary set in terms of age, education and gender ratio. For both SDMT, and ICA, the same forms of the tests were used in the re-test session. Note that in the ICA test, while the images were the same, their presentation order randomly changes in every administration.

Substudy 2: In this substudy, we investigated ICA’s correlation with the level of serum NfL in 48 MS patients (Table 2). Participants took the iPad-based ICA test and the pen-and-paper SDMT test, administered in random order. ICA and SDMT were administered in the same session, but blood samples were collected in another visit with a gap of 2-3 days in between.

Blood samples were collected in tube for serum isolation, then centrifuged at 3000 rpm for 20 minutes of blood draw, and finally placed on ice. Serum samples were measured at 1:4 dilution. NfL concentrations in serum were measured using a commercial ELISA (NF-light® ELISA, Uman Diagnostics, Umeå, Sweden). We used Anti NF-L monoclonal antibody (mAB) as a capture antibody and a biotin-labeled Anti NF-L mAB as the detection antibody. All samples measured blinded. ELISA readings were converted to units per milliliter by using a standard curve constructed by calibrators (Bovine lyophilized NfL obtained from UmanDiagnostics).

Participants in substudy 2 also attended an 8-week physical/cognitive rehabilitation program, details of which are reported in separate studies (^41,42). For the purposes of this study, to show ICA’s ability as a digital biomarker to track changes in cognition, we report pre and post rehabilitation ICA results for these group of participants, and the ICA correlation with NfL pre and post rehabilitation. Participants were divided into a rehabilitation group of 38 individuals and a control group of ten; the control group only took the tests pre and post these 8 weeks without attending the rehabilitation program. The rehabilitation group attended three sessions in each week, each of them lasting about 70 minutes.

Physical rehabilitation program included a combination of endurance and resistance exercises, with gradually increasing intensities over the 8-week period.

Cognitive rehabilitation program included playing newly developed games in virtual reality (VR) environment, targeting sensorimotor integration, memory-based navigation, and visual search.

2.5 Accuracy, speed, and ICA summary score calculations

Participants’ responses to each image and their reaction times (i.e. time between image onset and response) are recorded and used to calculate their overall accuracy and speed. Speed and accuracy are then used to calculate an overall summary score, we refer to as the ICA score.

Accuracy is simply defined as the number of correct categorisations divided by the total number of images, multiplied by a 100.

(see Equation 1 in the Supplementary Files)

Speed is defined based on participant’s response reaction times in trials they responded correctly.

(see Equation 2 in the Supplementary Files)

Speed is inversely related with participants’ reaction times; the higher the speed, the lower the reaction time.

Preprocessing: We used boxplot to remove outlier reaction times, before computing the ICA score. Boxplot is a non-parametric method for describing groups of numerical data through their quartiles; and allows for detection of outliers in the data. Following the boxplot approach, reaction times greater than q3 + w * (q3 - q1) or less than q1 - w * (q3 - q1) are considered outliers. q1 is the lower quartile, and q3 is the upper quartile of the reaction times. Where “w” is a 'whisker' ; w = 1.5. The number of reaction-time data-points removed by the boxplot can vary case by case; if this number exceeds 40% of the observed images, the results are deems invalid, and a warning is shown to the clinician to repeat the test. In this study none of the participants faced such a warning. The maximum number of outliers was 15%, which happened in one of the MS patients.

The ICA summary score is a combination of accuracy and speed, defined as follows:

(see Equation 3 in the Supplementary Files)

2.6 ICA’s artificial intelligence (AI) engine

ICA’s AI engine (Figure 2) used in this study was a multinomial logistic regression (MLR) classifier trained based on the set of ICA features extracted from the ICA test for each participant. These features included, the ICA score, and the trend of speed and accuracy during the test (i.e. whether the speed and/or accuracy were increasing or decreasing during the time-course of the test). The classifier also took subject’s age, gender and education in order to match subjects with similar demographics.

Multinomial logistic regression classifier (MLR)⁴³ is a supervised regression-based learning algorithm. The learning algorithm’s task is to learn a set of weights for a regression model that maps participants’ ICA test output to classification labels.

The basic difference between ICA’s classification of patients (using the AI engine) and the conventional way of defining an optimal cut-off value for classification is the dimensionality (or the number of features) used to make the classification. For example, in a conventional assessment tool, an optimal cut-off value is defined based on the test score. This is a one-dimensional classification problem, and there is only one free parameter to optimize, therefore less flexibility to learn from more data. In ICA, however, the test returns a rich set of features (we have one reaction-time and accuracy per each image). ICA score is the most informative summary score, but on top of this, we used a classifier to find the optimum classification boundary in the higher dimensional space. There are more free parameters here to optimize and therefore, the classifier can benefit from more data to best set these parameters for achieving a higher accuracy. Furthermore, ICA’s performance can be further improved over time if it is exposed to more labelled data. This can be done by providing new batches of training to update the current AI model available on the cloud.

3.1 Convergent validity with BICAMS, and sensitivity to MS

In substudy 1, we assessed convergent validity by examining the correlation between scores on the ICA test and the BICAMS battery (i.e. SDMT, BVMT-R and CVLT-II). Figure 3 presents scatterplots examining the relationship between BICAMS and ICA test performance. A high level of convergent validity is demonstrated between ICA and BICAMS. Within the BICAMS battery, SDMT had the highest correlation with the ICA test for HC (Pearson’s r = 0.81, p< 10^-14), MS (r = 0.71, p< 10^-13), and combined (r = 0.82, p< 10^-11) groups. Scatterplots show ICA vs. BICAMS correlation separately for MS and HC; combining results from both groups (n = 174 total), we find a correlation of 0.82 with SDMT (p< 10^-15), 0.71 with CVLT-II (p< 10^-10), and 0.60 with BVMT-R (p< 10^-8). The correlation results between BICAMS and ICA are largely similar when including only RRMS patients [r (SDMT) =0.71 (p< 10-13) ; r (BVMT-R) = 0.51 (p< 10-6); r (CVLT-II) = 0.56 (p< 10-7)]. Correlations between ICA’s speed and accuracy components with the BICAMS battery are also reported in Table 3. Furthermore, we calculated BICAMS composite score by averaging the z-scores of the CVLT-II, the BVMT-R, and the SDMT. ICA had a correlation of r=0.82 (p< 10^-11) with BICAMS composite score.

To compare sensitivity of BICAMS and ICA in detecting MS dysfunctions, we compared mean test scores in MS and HC groups separately for BICAMS battery of tests and the ICA test (Table 4). Within the BICAMS battery, SDMT and CVLT-II could differentiate between HC and MS patients (Table 4). The scores on both SDMT and CVLT-II were significantly lower for the MS patients compared to the HC group. However, there was no significant difference between BVMT-R scores of the HC and MS groups. These results are consistent with previous findings showing that SDMT has a better sensitivity in detecting MS compared to other tests within the BICAMS battery ^6,44. We repeated these analyses for the subset of RRMS patients (Supplementary Table 1), the results of which were similar to when we include all the patients.

As shown in Table 4, ICA could well discriminate between HC group and MS patients, at least as good as the SDMT; however ICA, as a digital test, has the advantages described in Figure 8.

Given that the ICA test involves tapping left or right on an iPad, we investigated the relation between subject’s handedness and their ICA score. The correlation between ICA score and handedness was (r=-0.13, p =0.07 -not significant), which is comparable (and lower) than the correlation between handedness and SDMT’s score (r=-0.17).

3.2 ICA accuracy in detecting cognitive impairment

45% of MS patients were identified to have cognitive impairment. Using an ROC curve (Figure 4), we then assessed the accuracy of the ICA’s AI engine (i.e. MLR classifier) in discriminating cognitively healthy from cognitively impaired individuals (Figure 4, area under curve (AUC) = 95.1 %, sensitivity = 82.9 %, and Specificity = 96.1%.)

3.3 ICA and SDMT correlations with Neurofilament light (NfL)

Neurofilament light (NfL) is a promising fluid biomarker of disease progression for various brain disorders, such as Alzheimer’s Disease and Multiple Sclerosis ^45,46 . In substudy 2, we demonstrated that there is a strong correlation between ICA score and the level of serum NfL (r=-0.79, p<10^-10) (Figure 5A). For comparison, on the same set of MS participants, SDMT correlations with NfL is also reported (r=-0.67, p<10^-6) (Figure 5B). SDMT and ICA were both administered in the same session.

3.4 ICA test-retest reliability and absence of a learning bias

Test-retest reliability was measured by computing the Pearson correlation between the two ICA scores. R values for test-retest correlation are considered adequate if >0.70 and good if >0.80 ⁴⁷.

Figure 6 presents scatterplots of ICA performance comparing 1st administration versus 2nd administration of the test for the HC, MS, and combined groups. Test–retest reliability was high, with correlation values in the range between 0.91 and 0.94.

In the subgroup of participants (21 MS, and 22 HC) who took the ICA and SDMT for a second time, we studied whether they could systematically get a better performance due to a previous exposure to either of the tests. This is called a learning bias (also referred to as practice effect). As shown in Table 5, comparing the first and second administration of the ICA and SDMT tests, ICA showed no learning bias. However, we see an improvement in participant’s average SDMT score. This improvement in SDMT score (i.e. learning bias) was statistically significant in the HC group, but not in the MS group.

3.5 ICA correlation with EDSS, age and education

To further characterize the ICA score and its relationship with other measures from the MS patients, we calculated the correlation between ICA score and patients’ EDSS, age and education (Table 6). Both BICAMS and ICA scores were negatively correlated with patients’ EDSS, demonstrating an inverse relation between disability scale and cognitive performance. For all the tests, we also observed a decrease in performance as the age increases, showing the effect of aging on cognitive performance. All tests were correlated with participant’s level of education, with ICA having the lowest correlation.

3.6 ICA’s performance pre- and post rehabilitation

We examined the level of serum NfL pre and post rehabilitation, as well as patients´ EDSS and ICA score (Figure 7). In the rehabilitation group, after the 8-week rehabilitation program, we observed a significant increase in subject's ICA score (Cohen's d=0.8, p< 0.0001), and a significant decrease in their level of serum-NfL (d=-0.4, p< 0.01) and EDSS (d=-0.4, p< 0.01). Whereas in the control group, we found the opposite pattern after 8 weeks, that is a decrease in ICA score (d=-0.4, p>0.05) and a significant increase in the level of serum-NfL (d=0.9,p< 0.001) and EDSS (d=1.0, p< 0.03).

In this validation study, we demonstrate that the ICA test has convergent validity with BICAMS, with an excellent test-retest reliability comparable to that reported for SDMT ¹⁴. ICA is a visuo-motor test and primarily tests information processing speed (IPS) and engages higher visual areas in the brain for semantic processing (i.e. animal vs. non-animal). Comparing speed versus accuracy in the ICA test (Table 4), speed seems to play a more significant role in discriminating MS patients from HC participants. This corroborates findings from other studies suggesting slower speed of information processing as a key deficit in multiple sclerosis ⁴⁸. IPS impairment underlies other areas of cognitive dysfunction ^19,49. This is because the speed with which an individual performs a cognitive task is not simply an isolated function of the processes required in that task, but also a reflection of their ability to rapidly carry out many different types of processing operations. In the case of ICA, these operations include transferring visual information through retina to higher level visual areas (i.e. sensory speed), processing the image representation in the visual system to categorize it into animal or non-animal (i.e. cognitive speed), and then translating this into a motor response (i.e. motor speed).

By measuring EDSS, we also explored the link between disability and cognitive impairment in the MS patients. Patients with cognitive impairment are typically found to be at higher risk of developing further disability ^5,12. While we did not have a long-term monitoring of disability progression in our patients in this study, the negative correlation between ICA score and EDSS corroborates previous findings that lower cognitive score is linked with higher disability (i.e. higher EDSS).

In contrast to most of the currently standard cognitive tests, whereby stimuli are language-dependent, the presented stimuli in the ICA test are natural images that contain universally recognizable images of animals or objects, thus making the test intrinsically language-independent. Furthermore, participants’ responses only involve tapping on the left or right side of an iPad, making it totally independent of participants’ knowledge of Arabic numerals or alphabets and words, or the drawing ability of a participant when drawing shapes (as in BVMT-R). This makes the ICA test more suitable for wider international use, and less dependent on lingual, educational, and demographic differences.

Computerized tests have several advantages over pen and paper tests, such as a) efficient administration that can save expensive clinical time, b) automatic scoring, which reduces errors in calculating and transferring scores, and c) easier integration with electronic medical records or research databases. The use of digital technology in this context can reduce barriers for both clinicians and patients to deliver or receive the assessments that would benefit their treatment and health throughout the course of the disease. With ICA, we aimed to develop a test that can close the current gap in clinical practice between patients’ needs and what clinicians can offer in terms of the much needed routine cognitive assessment and disease monitoring. Such a test must have a certain set of attributes, in addition to being sensitive and accurate. Figure 8 summarizes some of the key attributes of the ICA test as computerized test that makes it more scalable, accessible and cost effective compared to the standard pen and paper tests. Plus some of the unique features of the ICA test, which is absent from the computerized versions of the standard tests, such as the ability to benefit from more data and improve its performance when trained with new datasets over time.

Comparing ICA with the electronic implementations of SDMT ^50,51, we would like to highlight two main differences: a) The ICA test takes advantage of an AI platform, thus the capacity to learn from big data and further fine tune its multidimensional classification boundaries when it comes to see new training cases. b) ICA did not show a learning bias in this study and a previous study ²³, as opposed to the learning bias reported for the iPad-based SDMT (i.e. PST) ⁵⁰. This is because, in ICA, the images are shown in random order and they are presented only for 100 ms, making it very difficult for participants to learn the test.

For making an early diagnosis of MS and monitoring the disease progression, we need reliable biomarkers. NfL has been shown to be a promising fluid biomarker of disease progression for various brain disorders, including MS ^9,15 . Increased levels of NfL are shown to correlate with the severity of neural damage ¹⁰. In substudy 2, we demonstrated a strong association between ICA score and NfL in MS patients. This is particularly of interest given the totally non-invasive nature of the ICA test; and suggests the use of ICA as a digital biomarker of cognition in MS. The 8-week follow-up of the rehabilitation group, compared to the control group, further shows ICA’s sensitivity to track changes in cognition. For more frequent cognitive assessments, digital biomarkers have an advantage over fluid biomarkers, given their lower cost, accessibility, the possibility of remote administration and easier integration into routine clinical practice.

Some limitations encountered in this study include the lack of NfL data from the healthy control group, and the absence of neuroimaging markers of disease activity. Future studies are needed to investigate the link between ICA test results and other measures of brain atrophy, in particular, given the strong link between ICA and NfL—which reflects neural damage—would be informative to investigate the ICA relation with cortical thickness in MS patients.

Our results provide evidence for using ICA in neurology outpatient clinics as an accurate tool for assessing cognitive impairment in MS. Digital biomarkers of cognition (such as ICA) can be easily used to measure changes in cognitive performance relative to a baseline, which subsequently paves the way for using cognition as a marker of disease progression and treatment efficacy in MS. Given the absence of a learning bias in ICA, the test can be considered suitable for frequent monitoring of cognitive performance, allowing clinicians to measure cognitive decline, as well as potential treatment efficacy. The test further has the potential to be used for remote (i.e. home-based) monitoring of cognitive performance. Future longitudinal studies need to test this on large populations.

ICA has the potential to be used as a digital biomarker for assessment and monitoring of cognitive performance in MS patients. In comparison to standard cognitive tools for MS (e.g. BICAMS), ICA is shorter in duration, does not show a learning bias, is independent of language, and takes advantage of AI to identify cognitive status of patients more accurately. Being a digital test, it further has the potential for easier electronic health record or research database integration.

MS: Multiple Sclerosis AI: Artificial Intelligence ICA: Integrated Cognitive Assessment BICAMS: Brief International Cognitive Assessment for Multiple Sclerosis HC: Healthy Controls NfL: Neurofilament Light IPS: Information Processing Speed SDMT: Symbol Digit Modalities Test PST: Processing Speed Test CVLT-II: California Verbal Learning Test -2nd edition. BVMT-R: Brief Visual Memory Test–Revised ISI: Inter-Stimulus Interval

Ethics approval and consent to participate

The study was conducted according to the Declaration of Helsinki and approved by the local ethics committee at Royan Institute. Informed written consent was obtained from all participants.

Funding

SMKR was funded by a return home fellowship grant from the Iranian National Elite Foundation. Cognetivity ltd covered the costs for purchasing BICAMS. Other costs were covered by the Royan Institute internal funds to the investigators SMN, and SMKR.

Acknowledgements

The authors wish to acknowledge the MS patients, their family and the controls for participating in this study.

Consent for publication

Not applicable.

Authors Contributions

SMN and SMKR and CK conceived and designed the study. SMN, MS, and MK did the data collection and patient recruitment. The data were analyzed by MS and MK, under the supervision of MSN and SMKR. All authors were involved in writing the manuscript. All authors have read and approved the manuscript

Competing Interests

SMKR serves as the chief science officer at Cognetivity ltd; CK serves as Chief Medical Officer at Cognetivity Ltd. Other authors declared no potential conflicts of interest.

Availability of data and materials

The data generated during this study are included in this article. De-identified raw data are available to qualified investigators upon reasonable request from the corresponding author for the purposes of replicating procedures and results.

Trapp BD, Nave K-A. Multiple sclerosis: an immune or neurodegenerative disorder? Annu Rev Neurosci. 2008;31:247–269.
Filippo MD, Portaccio E, Mancini A, Calabresi P. Multiple sclerosis and cognition: synaptic failure and network dysfunction. Nature Reviews Neuroscience. 2018;19:599–609.
Rao SM, Leo GJ, Ellington L, Nauertz T, Bernardin L, Unverzagt F. Cognitive dysfunction in multiple sclerosis. II. Impact on employment and social functioning. Neurology. 1991;41:692–696.
Schultheis MT, Weisser V, Ang J, et al. Examining the relationship between cognition and driving performance in multiple sclerosis. Archives of physical medicine and rehabilitation. 2010;91:465–473.
Pitteri M, Romualdi C, Magliozzi R, Monaco S, Calabrese M. Cognitive impairment predicts disability progression and cortical thinning in MS: An 8-year study. Mult Scler. 2017;23:848–854.
Benedict RH, Amato MP, Boringa J, et al. Brief International Cognitive Assessment for MS (BICAMS): international standards for validation. BMC neurology. 2012;12:55.
Langdon DW, Amato MP, Boringa J, et al. Recommendations for a brief international cognitive assessment for multiple sclerosis (BICAMS). Multiple Sclerosis Journal. 2012;18:891–898.
Foley FW, Benedict RH, Gromisch ES, DeLuca J. The need for screening, assessment, and treatment for cognitive dysfunction in multiple sclerosis: results of a multidisciplinary CMSC consensus conference, September 24, 2010. International journal of MS care. 2012;14:58–64.
Berger T, Stüve O. Neurofilament light chain: An important step toward a disease biomarker in multiple sclerosis. AAN Enterprises; 2019.
Kuhle J, Kropshofer H, Haering DA, et al. Blood neurofilament light chain as a biomarker of MS disease activity and treatment response. Neurology. 2019;92:e1007–e1015.
Sumowski JF, Benedict R, Enzinger C, et al. Cognition in multiple sclerosis: State of the field and priorities for the future. Neurology. Epub 2018.:10–1212.
Motyl J, Kadrnozkova L, Dusankova JB, et al. Cognition as a Disability Progression Marker: Two-Years Follow-Up of People with Multiple Sclerosis (P5. 2-015). AAN Enterprises; 2019.
Benedict RH. Effects of using same-versus alternate-form memory tests during short-interval repeated assessments in multiple sclerosis. Journal of the International Neuropsychological Society. 2005;11:727–736.
Benedict R, Duquin J, Jurgensen S, et al. Repeated assessment of neuropsychological deficits in multiple sclerosis using the Symbol Digit Modalities Test and the MS Neuropsychological Screening Questionnaire. Mult Scler. 2008;14:940–946.
Calabresi PA, Arnold DL, Kinkel RP, et al. Serum Neurofilament Light (NfL): Towards a Blood Test for Prognosis and Disease/Treatment Monitoring in Multiple Sclerosis Patients (S24. 003). AAN Enterprises; 2018.
Eshaghi A, Marinescu RV, Young AL, et al. Progression of regional grey matter atrophy in multiple sclerosis. Brain. 2018;141:1665–1677.
Comabella M, Montalban X. Body fluid biomarkers in multiple sclerosis. The Lancet Neurology. 2014;13:113–126.
Qureshi SS, Beh SC, Frohman TC, Frohman EM. An update on neuro-ophthalmology of multiple sclerosis: the visual system as a model to study multiple sclerosis. Curr Opin Neurol. 2014;27:300–308.
Costa SL, Genova HM, DeLuca J, Chiaravalloti ND. Information processing speed in multiple sclerosis: Past, present, and future. Mult Scler. 2017;23:772–789.
Bacon-Macé N, Macé MJM, Fabre-Thorpe M, Thorpe SJ. The time course of visual processing: Backward masking and natural scene categorisation. Vision Research. 2005;45:1459–1469.
Mirzaei A, Khaligh-Razavi S-M, Ghodrati M, Zabbah S, Ebrahimpour R. Predicting the human reaction time based on natural image statistics in a rapid categorization task. Vision Research. 2013;81:36–44.
Khaligh-Razavi S-M, Habibi S. System for assessing a mental health disorder [online]. 2016. Accessed at: https://patents.google.com/patent/US20160278682A1/en. Accessed July 30, 2018.
Khaligh-Razavi S-M, Habibi S, Sadeghi M, et al. Integrated Cognitive Assessment: Speed and Accuracy of Visual Processing as a Reliable Proxy to Cognitive Performance. Scientific Reports. 2019;9:1102.
Vanrullen R, Thorpe SJ. The time course of visual processing: from early perception to decision-making. Journal of Cognitive Neuroscience. 2001;13:454–461.
Kiani R, Esteky H, Mirpour K, Tanaka K. Object Category Structure in Response Patterns of Neuronal Population in Monkey Inferior Temporal Cortex. J Neurophysiol. 2007;97:4296–4309.
Cichy RM, Pantazis D, Oliva A. Resolving human object recognition in space and time. Nat Neurosci. 2014;17:455–462.
Marx S, Hansen-Goos O, Thrun M, Einhäuser W. Rapid serial processing of natural scenes: Color modulates detection but neither recognition nor the attentional blink. Journal of Vision. 2014;14:4–4.
Zhu W, Drewes J, Gegenfurtner KR. Animal Detection in Natural Images: Effects of Color and Image Database. PLoS One [online serial]. 2013;8. Accessed at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3794973/. Accessed November 12, 2019.
Kriegeskorte N, Mur M, Ruff DA, et al. Matching Categorical Object Representations in Inferior Temporal Cortex of Man and Monkey. Neuron. 2008;60:1126–1141.
Naselaris T, Stansbury DE, Gallant JL. Cortical representation of animate and inanimate objects in complex natural scenes. Journal of Physiology-Paris. 2012;106:239–249.
Liu H, Agam Y, Madsen JR, Kreiman G. Timing, timing, timing: fast decoding of object information from intracranial field potentials in human visual cortex. Neuron. 2009;62:281–290.
Khaligh-Razavi S-M, Cichy RM, Pantazis D, Oliva A. Tracking the Spatiotemporal Neural Dynamics of Real-world Object Size and Animacy in the Human Brain. Journal of cognitive neuroscience. Epub 2018.:1–18.
Fahrenfort JJ, Scholte HS, Lamme VAF. Masking Disrupts Reentrant Processing in Human Visual Cortex. Journal of Cognitive Neuroscience. 2007;19:1488–1497.
Rajaei K, Mohsenzadeh Y, Ebrahimpour R, Khaligh-Razavi S-M. Beyond Core Object Recognition: Recurrent processes account for object recognition under occlusion. bioRxiv. Epub 2018.:302034.
Smith A. Symbol digit modalities test. Western Psychological Services Los Angeles, CA; 1982.
Delis DC, Kramer JH, Kaplan E, Ober BA. CVLT-II: California verbal learning test: adult version. Psychological Corporation; 2000.
Stegen S, Stepanov I, Cookfair D, et al. Validity of the California Verbal Learning Test–II in multiple sclerosis. The Clinical Neuropsychologist. 2010;24:189–202.
Benedict RH, Schretlen D, Groninger L, Dobraski M, Shpritz B. Revision of the Brief Visuospatial Memory Test: Studies of normal performance, reliability, and validity. Psychological Assessment. 1996;8:145.
Benedict RH. Brief visuospatial memory test–revised: professional manual. PAR; 1997.
Polman CH, Reingold SC, Banwell B, et al. Diagnostic criteria for multiple sclerosis: 2010 Revisions to the McDonald criteria. Ann Neurol. 2011;69:292–302.
Khaligh-Razavi S-M, Sadeghi M, Khanbagi M, Kalafatis C, Nabavi SM. Using ICA-an artificial intelligence (AI)-assisted technology-as a digital biomarker of MS disease progression and treatment efficacy. MULTIPLE SCLEROSIS JOURNAL. SAGE PUBLICATIONS LTD 1 OLIVERS YARD, 55 CITY ROAD, LONDON EC1Y 1SP, ENGLAND; 2019. p. 722–723.
Sadeghi M, Daemi M, Khaligh-Razavi S-M, et al. Virtual reality (VR)-based cognitive rehabilitation: cognitive games are complementary to physical training for an optimum rehabilitation strategy in patients with Multiple Sclerosis (MS). MULTIPLE SCLEROSIS JOURNAL. SAGE PUBLICATIONS LTD 1 OLIVERS YARD, 55 CITY ROAD, LONDON EC1Y 1SP, ENGLAND; 2019. p. 66–66.
Hastie T, Tibshirani R, Friedman J. The elements of statistical learning New York. NY: Springer. Epub 2009.
Eshaghi A, Riyahi-Alam S, Roostaei T, et al. Validity and reliability of a Persian translation of the Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS). The Clinical Neuropsychologist. 2012;26:975–984.
Lewczuk P, Ermann N, Andreasson U, et al. Plasma neurofilament light as a potential biomarker of neurodegeneration in Alzheimer’s disease. Alzheimer’s Research & Therapy. 2018;10:71.
Cai L, Huang J. Neurofilament light chain as a biological marker for multiple sclerosis: a meta-analysis study. Neuropsychiatr Dis Treat. 2018;14:2241–2254.
Anastasi A. Psychological testing. 6th edition. Macmillan Publishing Company; 1988.
Demaree HA, DeLuca J, Gaudino EA, Diamond BJ. Speed of information processing as a key deficit in multiple sclerosis: implications for rehabilitation. Journal of Neurology, Neurosurgery & Psychiatry. 1999;67:661–663.
DeLuca J, Chelune GJ, Tulsky DS, Lengenfelder J, Chiaravalloti ND. Is speed of processing or working memory the primary information processing deficit in multiple sclerosis? J Clin Exp Neuropsychol. 2004;26:550–562.
Rao SM, Losinski G, Mourany L, et al. Processing speed test: Validation of a self-administered, iPad®-based tool for screening cognitive dysfunction in a clinic setting. Multiple Sclerosis Journal. 2017;23:1929–1937.
Patel VP, Shen L, Rose J, Feinstein A. Taking the tester out of the SDMT: A proof of concept fully automated approach to assessing processing speed in people with MS. Mult Scler. Epub 2018 Aug 6.:1352458518792772.

Table 1. Demographic and disease related information for participants in substudy 1

Characteristic	*MS (n=91)*	*HC (n=83)*	p-value
Age –mean years ±SD	37.24 ±10.2	36 ±10	0.42
Education in years –mean ±SD	14.21 ±3.16	14.81 ±2.5	0.16
Gender (%female)	75 (82%)	58 (70%)	0.052
Disease Duration (in years)	6.8
Disease course
Relapsing remitting	83 (91%)
Secondary progressive	6 (7%)
Primary progressive	2 (2%)
EDSS –mean ±SD	1.27 ±1.8

P-values come from a two-sample t-test. MS: Multiple Sclerosis. HC: Healthy Controls. SD: standard deviation.

Table 2. Demographic and disease related information for participants in substudy 2

Number of Participants	Disease Course	Age- mean years ±SD	Gender (%female)	EDSS- mean ± SD	Education	Disease Duration
48 MS patients	Relapsing remitting	34.12 ±8.5	31 (64%)	2.95 ±0.92	14.79 ±1.41	8.27 ±5.20

Table 3. Speed and accuracy correlations with BICAMS

	SDMT	BVMT-R	CVLT-II
Speed	r=0.66 *	r=0.42 *	r=0.52 *
Accuracy	r=0.55 *	r=0.46 *	r=0.52 *

Pearson correlations between the BICAMS battery and speed, accuracy components of the ICA test across all participants. (* shows statistical significance at p < 10^-6)

Table 4. Mean ICA and BICAMS scores per group

	MS (n=91)		HC (n=83)
BICAMS	mean	SD	mean	SD	Difference	Cohen’s d	p-value
SDMT	41.04	11.02	54.73	9.77	13.69	1.31	<10^-14
BVMT-R	21.89	6.95	23.69	5.17	1.80	0.29	=0.0565
CVLT-II	48.96	11.13	58.28	6.59	9.32	1.00	<10^-9
ICA
ICA score	63.67	13.30	78.43	9.86	17.76	1.26	<10^-13
Accuracy	84.97	11.66	89.57	5.79	4.60	0.50	=0.0014
Speed	74.54	12.26	87.76	10.11	13.22	1.16	<10^-12

Mean and standard deviations (SD) for BICAMS test scores and the ICA test scores are compared for MS patients versus healthy controls (HC). The ICA score is a composite score made of both speed and accuracy of participants in ICA’s rapid visual categorization task. P-values come from a two-sample t-test.

Table 5. Learning bias (practice effect) for ICA and SDMT

Group	Test Name	Test 1 [mean ± SD]	Test 2 [mean ± SD]	Paired t-test
Group	Test Name	Test 1 [mean ± SD]	Test 2 [mean ± SD]	difference	Cohen’s d	p-value
*MS (n=21)*	ICA	64.76 ±12.7	64.66 ±12.2	-0.09	-0.02	0.91
*MS (n=21)*	SDMT	43.6 ±10	44.2 ±11	0.66	0.25	0.25
*HC (n=22)*	ICA	77.04 ±11	76.86 ±11	-0.18	-0.07	0.73
*HC (n=22)*	SDMT	55.13 ±12	56.36 ±12.3	1.22	0.50	0.02*

Mean and standard deviations (SD) of SDMT and ICA scores are compared between the two administrations of the tests for each group of participants. Only SDMT in healthy subjects showed a significant practice effect.

Table 6: Age/EDSS/Education vs. BICAMS/ICA

		BICAMS
		SDMT	BVMT-R	CVLT-II	ICA
Correlation with	EDSS	-0.41 (p<10^-4)	-0.26 (p<0.05)	-0.33 (p<0.001)	-0.58 (p<10^-8)
	Education	0.50 (p<10^-6)	0.34 (p<0.001)	0.31 (p<0.01)	0.25 (p<0.05)
	Age	-0.38 (p<10^-4)	-0.50 (p<10^-5)	-0.42 (p<10^-4)	-0.49 (p<10^-6)

The table shows Pearson Correlations of the ICA score and the BICAMS battery of tests with MS patients’ EDSS score, education in years, and their age. EDSS: Expanded Disability Status Scale; BICAMS: Brief International Cognitive Assessment for MS; SDMT: Symbol Digit Modalities Test; BVMT-R: Brief Visual Memory Test–Revised; CVLT-II: California Verbal Learning Test -2nd edition; ICA: Integrated Cognitive Assessment.

Download PDF

Journal Publication

published 18 May, 2020

Read the published version in BMC Neurology →

Editorial decision: Major revision
16 Dec, 2019
Review #1 received at journal
15 Dec, 2019
Reviewer #2 agreed at journal
03 Dec, 2019
Review #2 received at journal
03 Dec, 2019
Reviewer #1 agreed at journal
02 Dec, 2019
Reviewers invited by journal
28 Nov, 2019
Editor assigned by journal
27 Nov, 2019
Submission checks completed at journal
26 Nov, 2019
Editor invited by journal
26 Nov, 2019

You are reading this older preprint version

Read the latest preprint version →

A self-administered, artificial intelligence (AI) platform for cognitive assessment in multiple sclerosis (MS)

Status:

Journal Publication

Version 2

Abstract

Figures

Background

Methods

Results

Discussion

Conclusions

Abbreviations

Declarations

References

Tables

Supplementary Files

Status:

Journal Publication

Version 2