Participants
Twenty-six physically and mentally healthy Japanese adults without any history of neurological diseases participated in the study. They were recruited via senior citizen clubs and an employment service center for independently living older people in Sapporo, Japan from October 2019 to February 2020. The participants consisted of 11 women and 15 men between 65 and 84 years of age (mean: 74.2, standard deviation (SD): 5.2). Their average years of education were 12.8 (SD, 2.2). We judged them to be in good physical and mental health based on interviews about their medical and life histories and that they routinely work at the client sites through the employment service center or participate in the citizen clubs. The total number of entrants was 27, but we excluded one participant because of a history of cerebrovascular disease.
Materials
We selected the following tests for remote assessment: Raven's Colored Progressive Matrices (RCPM) [15], Story recall, 10/36 spatial recall, selective reminding test, Symbol Digit Modalities Test (SDMT), Paced Auditory Serial Addition Test (PASAT), Frontal Assessment Battery (FAB), Trail Making Test-A(TMT-A), Trail Making Test-B (TMT-B), visual cancellation task, digit span, and tapping span.
We obtained research use permission for the RCPM from Pearson Asia Inc. under the condition that we would not digitize the stimulus booklet. Story recall was excerpted from the Japanese version of the Rivermead Behavioral Memory Test (RBMT) [16] and obtained research use permission from the author and publisher. 10/36 spatial recall, selective reminding test, SDMT, and PASAT (3-sec version) were excerpted from the Japanese version of the Brief Repeatable Battery of Neuropsychological Tests (BRB-N) [17] and obtained research use permission from the author. We also obtained research use permission for FAB [18] from the author of the Japanese version.
Regarding TMT-A, TMT-B, visual cancellation task, digit span, and tapping span, we created them ourselves as original tasks because we could not obtain permission from the authors and publishers of these Japanese standardized tests. We created new tasks of TMT for VTC assessment: TMT-A for connecting numbers (1 to 20) with lines, and TMT-B for connecting numbers and Japanese kana letters (あ to こ) alternately with lines. We adjusted the size of the numbers and letters so that the size of the stimuli on the monitor matched the stimuli on an A4 size paper (placed vertically), and matched the trail lengths of A and B. For the visual cancellation task, we placed the Arabic numerals in 10 rows and 20 columns on the A4 size paper randomly; each row had 5–7 targets (number 5 was assigned as the target), for a total of 60 targets. For the digit span and tapping span, we created a random set of numbers with no duplicate numbers and no more than three consecutive numbers in a sequence, following a previous study [19]. We adopted the better performance of either of the two trials of each number in the digit span and tapping span tasks. To avoid adding alternate form differences, we repeated the same exact tests in all tasks.
Reliability of the self-created tests
The reliability of the self-created test was examined in 39 healthy older participants (16 men and 23 women) from senior citizen clubs and an employment service center in Sapporo, Japan. Because five of the 39 participants had also entered in the VTC validation study, these five received these tests one year later to rule out learning effects. Their average age was 76.7 years (SD, 7.6), with a mean RCPM score of 27.8 (SD, 3.8). We examined the correlations between the self-created TMT and standardized TMT-J [20], and the self-created visual cancellation task and standardized visual cancellation task from CAT [21]. The same examiner administered both tests on the same day, and the order of administration was counterbalanced among the participants. Pearson’s correlation coefficient was calculated. The result of TMT-A was r = .74 (p < .0001), TMT-B was r = .55 (p < .0003), time of visual cancellation task was r = .85 (p < .0001), score of visual cancellation task was r = .36 (p < .025), each with a significant correlation.
Procedures
The experimental design was an open-label, counterbalanced, crossover randomized controlled trial. We randomized participants into two groups by order of assessment administration (in a 1:1 ratio): VTC-first and FTF-first. We used stratified block randomization with age (< 65 years or ≧ 65 years) and gender as factors, and the block size was randomly varied between 2 and 4. However, there were no entries under the age of 65 years. An assistant not involved in the evaluation generated the random allocation sequence using web software [22] assignment, and informed the examiner after recruitment.
A nationally certified speech-language pathologist with at least 10 years of clinical experience conducted both, the VTC and FTF assessments, on the same day with an interval of approximately 60 minutes. The FTF assessment took approximately 50 minutes, and the VTC assessment took approximately 60 minutes. The examiner conducted the VTC assessment remotely by controlling the participant's PC from a separate room in the same building (central campus at Hokkaido University). No assistant was present during the assessment. We used Webex Meetings (Cisco Systems, Inc.) as a VTC system. We confirmed that the Internet speed was maintained at more than 70 Megabits per second for both downloads and uploads on all connected devices. We adapted the default value of Webex Meetings, 360×180, for image resolution because the goal of this study was simple and low-cost implementation.
Experimental equipment and setup
1. Tasks using only voice and video communication
We used voice and video calls for story recall, selective reminding test, PASAT, digit span, similarities (FAB), and word fluency (FAB). The examiner used a laptop PC (LIFEBOOK WU2/D2, 13.3-inch display manufactured by FUJITSU, Japan) with a unidirectional microphone (AT9933USB, manufactured by audio-technica) and a webcam (UCAM-C980FBBK, manufactured by ELECOM) connected to it. The examiner also used Microsoft Groove music and the audio sharing function in Webex Meetings for the presentation of PASAT. Participants used a desktop PC (Gemini X45 manufactured by Beelink) with a 25-inch color IPS monitor (manufactured by Philips), unidirectional microphone (same as the examiner), webcam (same as the examiner), and headphone (ATH-250M manufactured by audio-technica). The distance between the participant and monitor was set at approximately 50 cm. The examiner checked the volume and adjusted to a level that allowed each participant to hear the conversation comfortably before the experiment began.
2. Motor-dependent tasks that require writing or drawing operations
For the 10/36 spatial recall, SDMT, TMT-A, TMT-B, and visual cancellation tasks, which require writing or drawing operations, we used Microsoft PowerPoint and the screen sharing function of Webex Meetings. PowerPoint ran on the examiner's laptop PC and projected on the participant's PC (ENVY x360 13-ay1000 manufactured by HP) connected to a 23.8-inch touch monitor (P2418HT manufactured by Dell) using the screen sharing function of Webex Meetings. This touch monitor is equivalent to commonly available touch panel displays and is not a special device. To practice the use of the touch monitor and stylus pen before the test started, we asked participants to write numbers, draw circles, and lines. Participants wore gloves when malfunctions occurred because of palms touching the monitor.
In the VTC task, the testing seat was presented on the monitor of the same size as the FTF testing paper, and the examiner checked the handwriting locus in real time on the examiner's monitor. In the 10/36 spatial recall, since participants could not use the checkers for answering, we asked participants to draw circles on the monitor projected checkerboard. In 10/36 spatial recall using VTC, there is no limit to the number of checkers that can be placed, so filling the checkerboards would result in a perfect score. Therefore, we defined the score as the number of correct answers minus the number of incorrect answers.
3. Motor-dependent tasks that require confirmation of hand movements by the examiner
For the RCPM, tapping span, motor series “Luria” test (FAB), conflicting instructions (FAB), and go–no go (FAB), we used the webcam (Fig. 1a) built into the tablet (Surface Go manufactured by Microsoft) to show the examiner's hand movement and illustrations on the participants' monitor. The examiner checked the participant's hand movement and pointing place through the webcam (Fig. 1b) built into the smartphone (iPhone 8 manufactured by Apple) in real time on the examiner's monitor (Fig. 1c). Regarding the RCPM, unlike the original, we adopted the time required for the answer as the evaluation point as well as the total score.
Statistical analysis
A previous study recommended that a pilot study sample size should be 10% of a sample size of a project study [23]. The largest study of web-based neuropsychological assessment enrolled 202 participants [10]. Therefore, we estimated that 26 participants would be sufficient. Intraclass correlations (ICCs) and paired-samples t-tests were performed to compare the scores between the VTC and FTF conditions. We regarded ICC values as follows based on Koo et al [24]: < 0.50, poor; between 0.50 and 0.75, fair; between 0.75 and 0.90, good; above 0.90, excellent. Kappa coefficients and Wilcoxon signed-rank test were used for digit span and tapping span because the range of scores was narrow. We regarded κ as follows based on Ladies et al [25]: between 0.00 and 0.20, slight; between 0.21 and 0.40, moderate; between 0.41 and 0.60, substantial; between 0.61 and 0.80, almost perfect; above 0.81, excellent. Analyses were performed using IBM SPSS Statistics 26, and the Holm method was used for multiple comparison correction (α level was .05).
Incidents during VTC assessment
During the VTC assessment, there was one incident each of non-response of the touch monitor and failure to turn on the PowerPoint screen. Since assistants were not present, the examiner moved to the participant's room and performed troubleshooting. In each case, the problem was resolved within approximately two–three minutes, and the assessment was not stopped.
An Institutional Review Board approval statement and statement of patient consent
The Ethics Committee of the Faculty of Health Sciences, Hokkaido University approved this study (approval number:20-21-1), and all participants provided written informed consent. Furthermore, the Ethics Committee determined that this study was not recognized as an intervention study and that trial registration was not required.