Wearable Multimodal - Serious Game System For Hand and Cognitive Rehabilitation After Stroke

doi:10.21203/rs.3.rs-917425/v1

Download PDF

Research

Wearable Multimodal - Serious Game System For Hand and Cognitive Rehabilitation After Stroke

https://doi.org/10.21203/rs.3.rs-917425/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Objective: Stroke often leads to both motor control and cognitive dysfunction, and effective rehabilitation requires keeping patients engaged and motivated. We introduce a wearable multimodal system based on force myography, electromyography, and inertial sensing with two associated serious games for stroke rehabilitation of twelve hand movements related to activities of daily living and the Fugl Meyer Assessment.

Methods: In the ‘Find the Sheep’ serious game, patients performed corresponding hand movements to select the correct sheep card, and in the ‘Best Salesman’ serious game, patients performed corresponding hand movements to grab specific food and drink items in a store. A multi-sensor fusion model was developed for movement classification via linear discriminant analysis. Ten stroke patients with mild to moderate motor impairments (Brunnstrom Stage for Hand II-VI) performed validation testing, and effectiveness was evaluated by movement classification accuracy and qualitative patient questionnaires.

Results: Classification accuracy for twelve movements using combined force myography, electromyography, and inertial sensing was 81.0%, and accuracies for using electromyography, force myography, or inertial sensing alone were 69.6%, 63.2%, and 47.8%, respectively. All patients reported that they were more enthusiastic about rehabilitation while playing serious games than conventional rehabilitation, and a majority reported the wearable multimodal-based system was easier to wear than a sensorized data glove.

Significance: Results showed that multi-sensor fusion could improve hand gesture classification accuracy for stroke patients and demonstrated that the proposed wearable multimodal-serious game system could potentially facilitate upper extremity rehabilitation and cognitive training after stroke.

Physical Medicine & Rehab

Upper limb rehabilitation

cognitive function rehabilitation

EMG

FMG

IMU

serious games

Stroke is one of the most common causes of serious, long term disability, which affects fifteen million people each year worldwide [1]. Up to 60% of stroke survivors suffer from upper extremity impairments [2]. The functional improvement of upper extremities mainly depends on the improvement of hand function [3]. However, in the process of recovery from upper extremity hemiplegia, the restoration of distal motor function comes later and is more strenuous than the restoration of proximal motor function [4]. Intensive, repetitive, goal-oriented, and feedback-oriented movement training is critical to restore neural organization [5] and reduce hand motor function impairment [6]. In addition, depression [7] and cognitive dysfunction [8] are also common symptoms after a stroke. These patients suffer from a decreased mental state, accompanied by a decline in attention, execution, and memory [9]. Thus, recovery of the mental state and cognitive function also play an important role in rehabilitation [10].

For hospital-based rehabilitation of low hand motor function, artificial assistance or exoskeleton gloves are commonly used for passive training [11]. Patients with moderate to high hand function normally perform goal-oriented and activities-of-daily living (ADLs)-related movements repeatedly under the guidance of therapists, such as pinching a pen or using a spoon to hold beans. However, high costs and the shortage of medical resources make therapy-assisted training difficult to achieve. The most common home-based rehabilitation approach is to follow the rehabilitation exercise plan formulated by clinicians, which has low compliance and a high dropout rate due to boredom and a lack of motivation [12]. In addition, studies have shown intense goal-oriented training has little value unless the stroke patient is engaged and motivated [13]. Therefore, a number of technologies have been developed to address these problems.

Robot-assisted systems have been investigated to provide patients with intensive and effective therapy [14]. Although these systems can improve hand function, an unassisted system could provide a more effective rehabilitation method [15] for patients with mild to moderate hand dysfunction. Based on different motion capturing techniques, an unassisted rehabilitation system can be divided into three categories: camera-based, tangible-interaction-objects-based, and wearable-sensor-based. Rehabilitation systems that use cameras for motion tracking [16], [17] can be quite accurate, however, this method involves privacy issues [18] and has specific environment requirements [19]: specifically, the environment cannot be too cluttered, and other people should not appear in the camera’s view to avoid skeleton merging [20]. Tangible-interaction-objects-based rehabilitation systems have been developed [21], [22], which enable users to interact with the systems through manipulating tangible digital devices. These systems are friendly to the elderly and can reduce a user’s learning curve [23]. However, these systems are limited in a relatively single training mode and have poor scalability. Another approach is using a wearable-sensor-based system, such as those based on a data glove [24], [25], surface electromyography (sEMG) [26] or inertial measurement units (IMU) [27]. Data gloves were commonly integrated with flex sensors, accelerometers, or magnetic sensors [28]. Although data gloves can detect finger movements precisely, it’s hard for stroke patients to wear gloves due to the occurrence of spasticity after a stroke [29].

sEMG sensors have been widely utilized for hand movement estimation [30] and played an important role in rehabilitation systems [31]. However, the classification accuracy for stroke patients was much lower than that of healthy people because of neural damage [32]. The combination of sEMG and IMUs has produced better results in training systems [33], [34]. Barometric sensors have been employed to estimate hand gestures by measuring the tendon slide of the wrist [35]. In addition, the EMG-based recognition strategies are usually hyposensitive to low-strength gestures, and the strain signals are more sensitive to low strength gestures [36]. Therefore, multi-sensor fusion could be a promising method to improve sEMG-based recognition accuracy for stroke patients. In addition, to optimize the engagement of patients [37], a large amount of games have been designed and utilized in rehabilitation systems [38], [39], [40]. The effectiveness of serious games has proven to be better than conventional rehabilitation in upper limb motor function rehabilitation [41].

The purpose of this paper is to present a new rehabilitation system and newly developed serious games for motor function training. We hypothesized that the wearable multimodal-based system would be able to recognize a variety of stroke patient ADLs-related hand movements, and the stroke patients would be enthusiastic about rehabilitation using the proposed serious game system. Validation experiments were conducted on stroke patients with hand motor dysfunction in a hospital clinic to test our hypothesis.

A. System overview

The general structure of the wearable multimodal-based rehabilitation system consists of five elements: human fine movements, multi-sensor model, feature extraction, classification algorithms, and serious games (Fig. 1). It can provide users with movement-estimation-based serious games for rehabilitation. Stroke patients first perform fine movements selected from the Fugl Meyer Assessment (FMA) [42]. The physiological signal and kinematic signal of the user’s affected upper extremity are then collected by the multi-sensor model: the EMG data is collected by wireless electrodes on the forearm, the contact pressure profile is measured by barometric pressure sensors around the wrist, and kinematic data (including acceleration, angular velocity, magnetic field strength, and Euler angle) is collected by an IMU on the wrist, the barometric pressure sensors and IMU are connected by a USB cable. After preprocessing, features extracted from sEMG data, barometric sensor data, and IMU data are put into the movement classification algorithms. Finally, the estimated fine movements are sent to the serious games related to rehabilitation. More details about the system are presented below.

B. Upper extremity movement selection

The FMA is an effective and detailed evaluation tool for assessing motor function after stroke [43]. It is the most widely used clinical assessment scale, and the test items in it are highly correlated to ADLs. Eleven upper extremity fine movements (Fig. 2) were selected from the FMA, all suitable for motor function rehabilitation via a wearable multimodal-based system. The movements include hand movements: mass flexion (MF), mass extension (ME), hook-like grasp (HG), thumb adduction (TA), opposition (O), cylinder grip (CG) and spherical grip (SG); wrist movements: wrist volar flexion (WF) and wrist dorsiflexion (WE); forearm movements: forearm pronation (FP) and forearm supination (FS). A no-motion (NM) movement was also included. Some movements are also included in other practical scales, such as Wolf Motor Function Test (WMFT) [44]. All these movements are highly related to ADLs, which is very meaningful to stroke patients.

C. Prototype design

A sleeve for upper limb fine movement estimation was developed, containing 6 EMG sensors around the forearm and 8 barometric pressure sensors plus one IMU around the wrist (Fig. 1). There are multiple major superficial muscles around the forearm: the extensor carpi ulnaris, extensor digitorum, extensor carpi radialis longus and brevis, brachioradialis, pronator teres, flexor carpi radialis, flexor carpi ulnaris, and palmaris longus. Instead of applying muscle-targeted layout, low-density surface electrode layout was selected to detect the electromyographic signal of these muscles for practical use. Thus, 6 EMG wireless sensors from the Trigno Wireless EMG System (MAN-012-2-6, Delsys Inc., Natick, MA, USA) were selected and placed evenly around the forearm, about 10cm away from the elbow, covered and kept in place by an elastic band.

During wrist and hand movements, tendons of the wrist are shortened and lengthened, and muscles are deformed, resulting in wrist contour change. Thus, a flex wristband containing 8 barometric pressure sensors was developed to obtain contact pressure profiles around the wrist. Barometric sensors (MPL115A2, Freescale Semiconductor Inc., Austin, TX, USA) were covered by VytaFlex rubber to estimate the force myography (FMG) around the wrist. A 9-axis IMU (BNO055, BOSCH Inc, Stuttgart, Baden-Württemberg, German) was mounted on the back of the flex wristband to detect kinematic information. The output data of the IMU included 3-dimensional accelerations, 3-dimensional angular velocities, 3-dimensional magnetic field strengths, and 3-dimensional Euler angles. FMG data and IMU data were transmitted to a microcontroller (STM32F401, STMicroelectronics N.V., Geneva, Switzerland) for processing and analysis.

The EMG data, FMG data and inertial data from the IMU were collected by data collection software written in MATLAB (MathWorks, Natick, MA, USA) at 1926 Hz, 36 Hz, and 36 Hz, respectively. At the end of data collection, all data for each user was automatically saved into .csv files. For data synchronization, a user-friendly instruction program was developed. Users were asked to perform movements corresponding to the text and pictures shown on the software interface for training data collection. While the movements were being performed, corresponding triggers were transmitted to the data collection software in real-time via a virtual serial port.

D. Testing protocol

A clinical experiment was conducted to validate the estimation accuracy and practicality of the proposed system (Fig. 3). The experiment was pre-approved by the Huashan Hospital Institutional Review Board (CHiCTR1800017568) and was performed in accordance with the Declaration of Helsinki. Ten stroke patients (Brunnstrom stage for Hand II-VI) (TABLE 1) were recruited in this experiment (Supplementary File IV. Patient inclusion criteria). An experienced clinician was recruited from the Huashan hospital to assist in conducting the experiment with all patients and record the special circumstances during the experiment.

TABLE 1 STROKE PARTICIPANT CHARACTERISTICS

Sex (M/F)	8/2
Age (mean ± standard deviation)	58.3 ±18.09
Diagnosis (ischemic/hemorrhagic)	7/3
Hemiplegic side (left/right)	4/6
MMSE¹ (mean ± standard deviation)	27.7 ± 1.25
Brunnstrom stage for hand² (mean ± standard deviation)	4.5 ± 1.58
FMA upper extremity score² (mean ± standard deviation)	44.2 ± 13.9

¹Mini–Mental State Examination, used to measure the cognitive impairment,

ranging from 0 to 30.

²Brunnstrom stages range from 1 to 6.

³ Includes 33 FMA test items for upper extremity. Score range is 0 to 66.

Higher scores in MMSE, Brunnstrom stage, and FMA indicate better cognitive/motor function.

The experiment was conducted in the Rehabilitation Medicine Department of Hospital. First, an experienced clinician explained the experimental process and precautions to the patient and asked the patient to stop and report if there was any discomfort during the experiment. Then, the patient was asked to sit on a chair without armrest, so that the affected upper extremity naturally dangled to the side of the body. Patients put on the device with the assistance of the clinician: EMG sensors were placed evenly around the forearm of the patient’s affected side, where most muscular activity occurs. During hand movements, large contour changes occur on the underside of the wrist due to the shortening and extension of tendons. Therefore, the fourth and fifth pressure sensors of the 8 sensors-flex-wristband were aligned to the center of the underside of the patient’s wrist, with the other sensors wrapping around to the upper side of the wrist.

The experiment was divided into two phases: the training phase and the game phase. At the start of the training phase, the clinician explained all the movements to the patients in detail and showed instructional pictures to them. Next, patients were asked to perform movements following the instruction software we developed (Supplementary File V. Interface of the instructional software) to get familiar with the movements and the system. The software shows the text and pictures of the current movement and the movement that comes next. Then, patients were asked to perform 5 formal trials in the training phase, with 1-minute breaks in between. Each trial consisted of the data collection of 12 movements, and each movement lasted 6 seconds, with a 4-second break between movements.

After finishing the training, patients rested for ten minutes while watching a game demo video to get familiar with two serious games. Then, patients started to play 2 movement-estimation-based serious games. Each game session consisted of 5 trials. Each trial lasted 60 seconds, with a 60-second break between trials. Six patients finished all the 10 trials, two subjects lack of 1 trial, and two subjects quitted due to fatigue when 2 trials were left. After completing the serious games, patients were asked to fill out a questionnaire (TABLE 2) about the experience of using this serious-games rehabilitation system. There were ten questions, each of which could be answered with ‘strongly agree,’ ‘agree,’ ‘neutral,’ ‘disagree,’ and ‘strongly disagree.’ Besides, we also solicited opinions from patients on the improvement of this system.

TABLE 2 Questionnaire for Serious-Games Rehabilitation system

Symbol	Questions
1	Does the game make you more enthusiastic about rehabilitation?
2	Were you relaxed and happy while playing the games?
3	Did you feel frustrated while playing the games?
4	Are the serious games challenging?
5	Was your body uncomfortable while playing the game?
6	Was the training part before the games a burden to you?
7	Do you think the game is suitable for home-based rehabilitation?
8	Is this sleeved sensor setup more practical than a glove setup?
9	Do you think the game is beneficial for improving your cognitive function?
10	Do you think the game is beneficial for improving your upper limb motor function?

E. Serious games design

Two serious games (Fig. 4) were newly developed based on movement estimation. The first 2 seconds and the last 0.5 seconds of data collected from the training phase were removed and the rest of the data were used to train models for the serious games. Features were extracted from real-time data of sensors on patients in MATLAB, after which they were transferred to the game in real-time via TCP/IP communication. Then, the trained models were loaded and used to classify movements. Finally, the estimated movement was used as input to the game, allowing patients to choose targets by performing the corresponding movement. The games provide visual and audio feedback to the patients. When patients perform the correct movement, the text “excellent” appears on the screen, and the money on the screen increases by 1. Patients also hear a positive audio cue. In addition, all game materials are from online open-source materials (see Supplementary File I. Game materials). The games were written in python based on the pygame library.

The game “Find the Sheep” was designed for both motor and cognitive function training. Patients need to concentrate during the whole game and do the movements as instructed. Three cards appear in the game interface, with a sheep and two wolves on the front accordingly. Then, all three cards are flipped over and randomly swap positions with each other. After swapping, the patient needs to find which card is the one with the sheep and perform the corresponding movement shown below that card. The 12 movements are divided into 4 groups for “Find the Sheep”, which are displayed in different game rounds (Supplementary File II. Grouping of the different movements). Many of the movements we selected are similar, such as the spherical grasp and the cylinder grasp. By dividing the movements into several groups, the real-time recognition accuracy of the system is improved. During the game, the system loads the classification model trained for the current movement group. The game has multiple difficulty levels. The higher the difficulty level, the more times the cards will be rearranged. Although the moving speed of cards was set to be the same throughout this experiment, users have the option to set the moving speed according to their ability in daily use.

The game “Best Salesman” was designed to train patient motor function and improve their performance in ADLs by restoring movements to life scenes. In this game, the user owns a grocery store that sells seven types of food. Customers keep coming to the store to buy one to three types of food. Users need to pass the right food to the customers by performing the correct corresponding movement: hold a cup, take a cup with a handle, cover top burger bread, hold bottom burger bread, pinch a piece of watermelon, lateral pinch a popsicle and hold a tomato. Only seven ADLs-related movements which can be easily connected to objects were selected in “Best Salesman”, so that patients can intuitively know what hand gesture they should do when they see the object pictures just like they normally do in daily activities. Like the “Find the Sheep” game, different movements are divided into groups to increase the accuracy of the classification model for “Best Salesman” (Supplementary File II. Grouping of the different movements).

We assumed 1 second would be enough for patients to react and perform the corresponding movement. Therefore, at the end of each round, when the cards in “Find the Sheep” stopped moving and the products in “Best Salesman” were shown, the game waited 1 second before collecting input from the patients. In addition, only the predictions from the first ten windows starting from the 1st second were used. The most predicted movement was then regarded as the patient’s actual movement and sent to the game. The comprehensive results of ten windows were acceptable in real-time use and stable in prediction.

F. Feature extraction

Data of different movements were segmented automatically based on triggers in the data collection code, which correspond to different movements. During the transition period between movements, related muscle activities erupt and cause a larger EMG amplitude. Besides, stroke patients are generally older and slower to respond. Thus, for the data collected in the training phase, the first 2 seconds and the last 0.5 second of each movement are removed to reduce interference.

For EMG segmentation, overlapped segmentation with a window length of 200ms and an increment of 50ms has a short response time while ensuring accuracy, which is suitable for real-time movement classification [45]. Considering the performance of the algorithm and the synchronization of EMG, FMG, and IMU data, overlapped segmentation with a window length of 222ms and a step size of 55.6ms was adapted to divide the raw EMG data into windows. Disjoint segmentation with a window size of 55.6ms was used to segment both the FMG and IMU data. The FMG data that exceeded measuring range was deleted during the preprocessing phase”

Time domain features are very effective in EMG pattern recognition [46]. Four reliable time domain features (Supplementary File III. Feature formulas) were selected and extracted from EMG signals: Mean Absolute Value (MAV), Waveform Length (WL), Zero Crossings (ZC), Slope Sign Changes (SSC) [47]. MAV contains information about a signal’s strength and amplitude. WL reflects the signal’s complexity. ZC and SSC reflect the frequency information of the signal, both containing a threshold \(ϵ\) to reduce noise interference. A frequency domain feature was also selected from the EMG signal: Auto-Regressive Coefficients (AR), which describe each signal sample as a linear combination of previous samples plus white noise error terms \({e}_{k}\) [48]. The fourth-order AR was used (Supplementary File III. Feature formulas). Also, MAV was chosen as the feature of FMG and IMU signals.

In total, 6 channels of EMG data, 8 channels of FMG data, and 12 channels of IMU data were used. 8 features were extracted from each window of each EMG channel. Meanwhile, the MAV of each window of each FMG and IMU channel was calculated. Thus, 48 EMG features, 8 FMG features, and 12 IMU features result in 68-dimensional feature array. Each channel’s data was scaled and normalized via zero-mean normalization by using the mean value and standard deviation from each respective trial.

G. Classification algorithm

As an online classification system, this system uses linear discriminant analysis (LDA) as a pattern recognition classifier. This linear classifier can simplify the computational complexity, shorten the time, and still produce an accurate recognition result [47], and it has also proven to be very robust [49]. The LDA classification algorithm is based on Bayes decision theory and the Gaussian assumption. The discriminant function is defined as:

where x is the input vector, Σ is the covariance matrix, μ_k is the k class’s mean and π_k is the prior probability of class k.

In previous research, decision tree (DT) [50], k-nearest neighbor (KNN) [51], random forest (RF) [52], and support vector machine (SVM) [53] were also used for stroke rehabilitation. Apart from LDA, these four algorithms were also tested for movement classification in the study.

H. Statistical analysis

To validate the efficiency and accuracy of the proposed classification algorithm, the average accuracy of the LDA-based 12 movement classification was calculated. Five trials in the training phase were used to perform an offline test, using leave-one-out cross-validation. Training data and test data for offline testing were both taken from the 2 to 5.5 second. A confusion matrix was created to display the recognition rate of each gesture and the misclassification between gestures.

In order to analyze the contribution of different sensors to the classification, the accuracies of single, double, and triple sensor-based classification algorithms were calculated separately. Also, the confusion matrixes of EMG-alone-based hand gesture classification and FMG-alone-based hand gesture classification were created to show the contribution of different sensors on different gestures. In addition, Pearson correlation coefficients (PCCs) between EMG-based offline accuracies, FMG-based offline accuracies, and EMG-FMG-IMU-based offline accuracies for all subjects were calculated to study the correlation between the performances of different physiological information-based movement recognition.

The performances of DT, KNN, RF, and SVM were also analyzed as comparison of LDA.

Because of poor hand function in some patients, and some of the movements we choose having similarities, it is difficult to observe from the outside what movement the patient is performing in many cases. Therefore, in real-time classification games, when patient movement was determined to be incorrect, it was difficult to tell whether the system made classified incorrectly, or the patient did not successfully complete the right movement. Thus, to test the effectiveness of the proposed movement-detection-based serious games, the performance of real-time classification was simulated and validated. To validate the real-time performance, we applied cross-validation on 5 trials for each subject. Four trials were used as training data, all of which ranged from the 2nd to the 5.5th second (Supplementary File VI. Different cutoffs - Statistical analysis). The first ten samples starting from the 1st second of the leftover trial were used to test the model. One-way analysis of variance (ANOVA) was conducted to assess if there were differences between using different sensor configurations, different algorithms and different cutoffs. If there was a difference, LSD procedure was used for post hoc analysis. The statistical significance was set to p < 0.05.

To analyze the correlation between subjects’ upper limb motor function and their different information-based hand gesture classification accuracies, PCCs between the FMUE scores of stroke patients and their offline accuracies of EMG-based, FMG-based, and EMG-FMG-IMU-based hand gesture classification were calculated, respectively.

In addition, the correlation between subjects’ performance in playing serious games and their motor function, cognitive function, and movement recognition accuracies were analyzed by calculating PCCs of patients’ average scores in two serious games and their FMUE, MMSE, and EMG-FMG-IMU-based hand gesture classification offline accuracies, respectively. Besides, the correlation between subjects’ average scores in two serious games were analyzed by PCCs. The statistical significance for PCCs was set to p < 0.05.

The average scores of all the subjects for each trial in the serious games “Find the Sheep” and “Best Salesman” were calculated and analyzed to show the performance of the patients while playing serious games.

The results of the questionnaires were analyzed to define the patients’ subjective feelings about using the proposed rehabilitation system. Patients’ suggestions were also examined and serve as important references for future improvements to the proposed system.

The LDA-based offline classification accuracy of 12 movements for each subject ranged from 64.3–96.3%, with an average accuracy of 81.0% for all 10 patients. The predicted accuracies of each movement range from 70.5% (for TA) to 89.1% (for FP) (Fig. 5). CG and SG were most likely to be misclassified with each other. In addition, TA and O were commonly misclassified with each other, and TA was often misclassified as NM. ME was often mistakenly recognized as HG, and HG was often misrecognized as MF or NM. Also, WE and FP were also sometimes misclassified. The classification accuracy of using IMU alone, FMG alone, EMG alone, FMG and IMU, EMG and IMU, EMG, and FMG or all the sensors were 47.8%, 63.2%, 69.6%, 71.5%, 75.4%, 79.7%, and 81.0%, respectively (Fig. 6 ). There are significant improvements of using all three sensors, compared with using these sensors alone, using EMG and IMU together, or using FMG and IMU together (p < 0.05).

For both EMG-based and FMG-based hand gesture classification (Fig. 7), the performances of recognition on some gestures were the same. For example, CG and SG were easily misclassified with each other in both models. However, when recognizing other gestures, different models performed differently. For example, MF was often mistakenly recognized as HG when the EMG-based model was applied, and MF was easily misrecognized as O when the FMG-based model was applied. There’s a correlation between classification accuracies of the EMG-based model and classification accuracies of the FMG-based model (r = 0.69, p < 0.05) (Fig. 8), and there’s also a correlation between classification accuracies of the EMG-based model and classification accuracies of the EMG-FMG-IMU-based model (r = 0.73, p < 0.05) (Fig. 8). There’s a high correlation (r = 0.94) between classification accuracies of the FMG-based model and classification accuracies of the EMG-FMG-IMU-based model (p < 0.05) (Fig. 8). The average offline classification accuracies of applying DT, KNN, RF, and SVM were 62.7%, 72.9%, 78.4%, and 80.9%, which were lower than LDA’s 81.0% accuracy. However, LDA only had significant difference with DT and KNN (p < 0.05).

By splitting up the movements in groups, the accuracies of simulation real-time classification ranged from 80.7–84.7% for each movement group, and the simulated classification accuracies were 82.5% and 83.6% for the groups used in the “Find the Sheep” game and the “Best Salesman” game, respectively. Different combinations of cutoffs were applied, and the simulated real-time classification accuracies of two serious games were over 90% when applying ideal cutoffs (Supplementary File VII. Different cutoffs - Results).

There’s a correlation (r = 0.78) between FMUE and offline classification accuracies of the FMG-based model (p < 0.05). However, there’s no significant correlation between FMUE and the offline classification accuracies of the EMG-based model (r = 0.61, p = 0.065) or the EMG-FMG-IMU-based model (r = 0.61, p = 0.063), respectively (Fig. 8).

There’s a correlation (r = 0.78) between subjects’ average scores of playing “Find the Sheep” and patients’ average scores of playing “Best Salesman” (p < 0.05) (Fig. 9). However, there’s no correlation (r < 0.5) between scores of each serious game and MMSE, FMUE, and offline classification accuracies, respectively.

The average scores across all the subjects for each trial in the serious game “Find the Sheep” were 5.6, 4.9, 5.8, 5, and 5.3 (Fig. 10). In addition, the average scores of each trial in the serious game “Best Salesman” were 11, 13.5, 11.5, 11.4, and 13.4 (Fig. 10).

Ten stroke patients who participated in the serious games filled out the questionnaires (Fig. 11). The majority of patients strongly agreed that the serious games made them more enthusiastic about rehabilitation and that they felt relaxed and happy while playing the games. 60% of the patients didn’t feel frustrated at all during the games, while 20% of them felt a little frustrated. 60% of patients strongly agreed that the games were challenging. None of the patients experienced any upper limb discomfort during the games. 90% of patients strongly disagreed or disagreed that the training part before the games was a burden. 90% of patients strongly agreed or agreed that the proposed system is suitable for home-based rehabilitation and is more practical than wearing a glove. Most patients also strongly agreed that the proposed training is beneficial for improving both upper limb motor function and cognitive function. The patients expressed other thoughts and suggestions about the proposed system. Three patients thought the proposed system was beneficial for brain and neurological restoration. Two patients mentioned the proposed training was very entertaining and fun, which increased the attractiveness of rehabilitation. Also, two patients thought the selected movements were very meaningful, involving ADLs functional training such as grasping a cup. One patient expressed that the training strengthened his confidence. The patient with the lowest classification accuracy indicated that the sensing part needed to be improved, and the game time should be shortened. Two patients suggested the games should be harder, while two other patients thought it was too hard for them to complete the right movement within the prescribed time and that the games should be simpler and slower in the future.

We proposed a serious games rehabilitation system, to recognize the movement of patients’ affected sides via a multi-sensor fusion model and to provide patients with serious games and feedback for the restoration of upper limb motor function and cognitive function.

In previous research, most studies focused on pattern recognition for stroke rehabilitation systems [26], [31], but only healthy subjects were included, or the motion of each patient’s unaffected side was included to perform bilateral training [54], [55]. Few studies have worked on the affected-side-based motion recognition of stroke patients. Lee et al. [56] recruited 20 stroke patients with chronic hemiparesis, and selected six functional movements. Ten surface electrodes were applied to record EMG signals. The mean accuracy of moderate-function patients and low-function patients was 71.3% and 37.9%, respectively. Zhang et al. [57] applied high-density electrodes to classify 20 movements of 12 stroke patients, and resulted in a classification accuracy of 96%. Castiblanco et al. [58] proposed a study of hand motion recognition via EMG. Healthy subjects, stroke subjects without hand impairments, and stroke patients with impairments were included. In their research, hand gestures were separated into several groups, and each group contained 2 to 5 movements; in their study, the average recognition accuracy for stroke patients with impairments was 85%. The classification accuracy for 12 movements of the proposed system is 81.0%, which is lower than the system applying high-density EMG. However, we covered more movements related to ADLs, and obtained higher classification accuracy compared to other previous research.

Our results verify that the multi-sensor fusion method greatly improves accuracy over single sensor-based pattern recognition. The accuracy of applying both EMG and FMG was close to the accuracy of applying all sensors. EMG and FMG contribute the most to movement recognition, which may be due to the IMU being placed on the wrist and most of the movements we selected being finger movements. The EMG-based model and FMG-based model showed different performances on gesture recognition, and the information from these two models can be used to compensate for each other to increase the robustness of the system. In addition, there’s a significant high correlation between subjects’ FMG-based hand gesture classification accuracies and their EMG-FMG-IMU-based hand gesture classification accuracies, which indicate FMG information has the most influence on the multi-sensor fusion model. The multi-sensor fusion model we proposed contains more information, and improved robustness, which can improve the accuracy of motion recognition on the affected side, and expand the range of users.

The patients we recruited range from high motor function to low motor function. Results showed significant correlation between subjects’ upper limb motor function and the offline accuracies of FMG-based hand gesture recognition, which indicates wrist-tendon-slide-related information has the potential to be used to assess the upper limb motor function of stroke patients. Also, it's interesting that subjects showed consistency in two different serious games. However, the performaces of patients in either game were not correlated to either their upper limb motor function, their cognitive function, or their hand gesture classification accuracy, which may indicate that the performance of patients in the game may be caused by a combination of multiple factors. In addition, when subjects were playing the game, there was not an obvious improvement in their scores over time. We assumed the main reason is games were difficult for the subjects. Most subjects were elderly people and suffered from brain injuries, so their learning curve may be relatively long. The serious games’ settings need to be further considered in the future research.

To improve the effectiveness of the system and the patient’s experience of playing serious games, we grouped 12 movements during the real-time movement classification (Supplementary File II. Grouping of the different movements) based on experience and prior knowledge, so that three movements that are not easily confused for each other were grouped together. According to the results presented by the confusion matrix, it was verified that our grouping situation was almost ideal, avoiding the situation where movements that are easily misclassified appear in the same group. Spherical grasp and cylinder grasp are similar and easily misclassified with each other. In addition, the thumb adduction and opposition are also very similar; especially compared to other movements, it is more difficult for patients to exert force correctly and effectively complete these two movements, which also makes these two movements not only easy to mix up with each other but also easy to misclassify as no motion.

Sometimes, it was hard to tell what movement patients performed while they were playing games, which made it hard to distinguish whether the patients did not complete the right movement successfully or whether the system recognized the movement wrong. Therefore, we used training data to simulate real-time classification through cross-validation. This simulation process was the same as the process we used in the games. We found through this research that the real-time classification method we used in the experiment was not optimal: the simulated real-time accuracy is relatively low, and some patients reported that the games moved too fast to complete the movement. In addition, we’ll conduct a formal real-time experiment such as Motion Test and analyze the real-time accuracies of different algorithms and different sensor configurations to verify the real-time performance of the proposed multi-sensor fusion model on stroke patients. Besides, response time and execution time of patients will be analyzed, and new cutoff will be applied. In addition, it is necessary to consider reducing the number of gestures to improve the accuracy of recognition and performance of the system in future research and commercial product development.

There were some limitations of the proposed system which need to be improved in future work. The patients need to train before using the proposed system. Future work could be focused on increasing the robustness of the proposed system, addressing the problem of decreased recognition accuracy caused by sensor picking. In addition, we only included single hand gestures, and multi-DOF concurrent movements were missing. Users were asked to drop their hands naturally on both sides of the body to perform the corresponding hand gestures, which didn’t restore the ADLs completely. In order to make the system more practical, the experiment could be carried out in a semi-natural or total-natural environment in which patients could perform natural ADLs-related movements, such as drinking water, and a more practical model could be developed. Also, more ADLs-related hand gestures should be considered in future work, such as a tripod pinch [59]. In future work, a long-term, randomized control trial involving stroke patients with upper limb dysfunction should be conducted to validate whether doing training with the proposed system can improve the upper limb motor function and cognitive function of stroke patients. Also, the effectiveness of the proposed system should be compared with conventional therapies. Also, the questionnaire in the study was unstructured which is a potential limitation. A more structured questionnaire should be considered in the future work to strengthen these preliminary findings

The proposed system validates the advantages of multi-sensor fusion and its application prospects in the field of rehabilitation. The motion pattern recognition plays an important role in both exoskeleton-based rehabilitation training systems and daily life assistance systems. The multi-sensor fusion model we proposed has the potential to be applied on active assisted robotic rehabilitation systems or active ADLs-assisted orthoses to improve their motion recognition performance on the affected side. For the wider application of this system, a low-cost system will be developed and verified in future research.

This study proposes a wearable serious-game-based training system for the rehabilitation of both upper limb motor function and cognitive function. A multi-sensor fusion model was developed for the movement recognition of stroke patients with upper limb dysfunction, and two movement classification-based serious games were developed to train patients’ attention and memory. An experiment involving stroke patients with different levels of upper limb impairments was performed to validate the effectiveness of the proposed system. Results showed that the proposed system can classify a variety of ADLs related fine movements, and the proposed serious game system was able to stimulate patients’ enthusiasm for rehabilitation and guide them to actively perform repeated movements. The proposed training system has the potential to be used in both clinical-based and home-based environment by stroke patients to improve upper extremity motor function and cognitive function. The multi-sensor fusion method can improve the motion recognition performance with stroke patients. This effective model can be used both in unassisted serious-game-training systems and also in the active robotic-assisted rehabilitation system or ADLs-based orthosis.

ADLs: activities-of-daily living; sEMG: surface electromyography; IMU: inertial measurement units; FMG: force myography; FMA: Fugl Meyer Assessment; MF: mass flexion; ME: mass extension; HG: hook-like grasp; TA: thumb adduction; O: opposition; CG: cylinder grip; SG: spherical grip; WF: wrist volar flexion; WE: wrist dorsiflexion; FP: forearm pronation; FS: forearm supination; NM: no-motion; WMFT: Wolf Motor Function Test; LDA: linear discriminant analysis; DT: decision tree; KNN: k-nearest neighbor; RF: random forest; SVM: support vector machine; Pearson correltation coefficient (PCC).

Acknowledgment

The authors would like to thank all clinicians and patients for their participation in this study. The authors would also like to thank Dr. Kim Sunesen and Dr. Frank Wouda for their input on the study design and advice on the writing of this manuscript.

Authors’ contributions

XYS conceived study design, collected and analyzed the data, and drafted the manuscript. SSVDV involved in study design, data collection, and provided feedback on the manuscript. PQK involved in the study design. QHG involved in the study design. SGC recruited the subjects. JJ provided feedback on the manuscript. PBS involved in study design, interpretation of the results and provided feedback on the manuscript. All the authors read and approved the final manuscript.

Funding

This research supported by the National Natural Science Foundation of China (51950410602, 81672252, and 81702229), the National Key Research and Development Program Project of China (2018YFC2002300 and 2018YFC2002301), and Xsens Technologies B.V., Enschede, Netherlands.

Availability of data and materials

The data that was collected and/or analyzed are not publicly available but are available from the authors on reasonable request.

Ethics approval and consent to participate

The experiment was pre-approved by the Huashan Hospital Institutional Review Board (CHiCTR1800017568) and was performed in accordance with the Declaration of Helsinki.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

World Health Organization. “WHO - The top 10 causes of death,” 24 Maggio, 2018.
Van Der Lee JH, Wagenaar RC, Lankhorst GJ, Vogelaar TW, Devillé WL, Bouter LM. “Forced use of the upper extremity in chronic stroke patients: Results from a single-blind randomized clinical trial,” Stroke, 30, 11, 1999, doi:10.1161/01.STR.30.11.2369.
Kwakkel G, Boudewijn K. Predicting improvement in the upper paretic limb after stroke: A longitudinal prospective study. Restor Neurol Neurosci. 2007;25:no. 5–6., “,”, , pp. 453–460.
Twitchell TE. “The restoration of motor function following hemiplegia in man,” Brain, 74, 4, 1951, doi:10.1093/brain/74.4.443.
Song X, Chen S, Jia J, Shull PB. “Cellphone-Based Automated Fugl-Meyer Assessment to Evaluate Upper Extremity Motor Function after Stroke,” IEEE Trans Neural Syst Rehabil Eng, 27, 10, 2019, doi:10.1109/TNSRE.2019.2939587.
Ada L, Dorsch S, Canning CG. “Strengthening interventions increase strength and improve activity after stroke: A systematic review,” Aust J Physiother, 52, 4, 2006, doi:10.1016/S0004-9514(06)70003-4.
Robinson RG, Benson DF. Depression in aphasic patients: Frequency, severity, and clinical-pathological correlations. Brain Lang. 1981. doi:10.1016/0093-934X(81)90080-8.
Madureira S, Guerreiro M, Ferro JM. “Dementia and cognitive impairment three months after stroke,” Eur J Neurol, 8, 6, 2001, doi:10.1046/j.1468-1331.2001.00332.x.
Jaillard A, Naegele B, Trabucco-Miguel S, LeBas JF, Hommel M. Hidden dysfunctioning in subacute stroke. Stroke. 2009. doi:10.1161/STROKEAHA.108.541144.
Song X, Ding L, Zhao J, Jia J, Shull P, “Cellphone augmented reality game-based rehabilitation for improving motor function and mental state after stroke,” in 2019 IEEE 16th International Conference on Wearable and Implantable Body Sensor Networks, BSN 2019 - Proceedings, 2019, doi: 10.1109/BSN.2019.8771093.
Vanoglio F, et al. Feasibility and efficacy of a robotic device for hand rehabilitation in hemiplegic stroke patients: A randomized pilot controlled study. Clin Rehabil. 2017;31(3):351–60. doi:10.1177/0269215516642606.
Cox KL, Burke V, Gorely TJ, Beilin LJ, Puddey IB, “Controlled comparison of retention and adherence in home- vs center-initiated exercise interventions in women ages 40–65 years: The S.W.E.A.T. Study (Sedentary Women Exercise Adherence Trial),” Prev. Med. (Baltim)., vol. 36, no. 1, 2003, doi: 10.1006/pmed.2002.1134.
Winstein C, Varghese R. Been there, done that, so what’s next for arm and hand rehabilitation in stroke? NeuroRehabilitation. 2018;43(1):3–18. doi:10.3233/NRE-172412.
In H, Kang BB, Sin MK, Cho KJ. Exo-Glove: A wearable robot for the hand with a soft tendon routing system. IEEE Robot Autom Mag. 2015;22(1):97–105. doi:10.1109/MRA.2014.2362863.
Lum P, Reinkensmeyer D, Mahoney R, Rymer WZ, Burgar C. Robotic devices for movement therapy after stroke: Current status and challenges to clinical acceptance. Topics in Stroke Rehabilitation. 2002. doi:10.1310/9KFM-KF81-P9A4-5WW0.
Hondori HM, Khademi M, Dodakian L, Cramer SC, Lopes CV. A spatial augmented reality rehab system for post-stroke hand rehabilitation. MMVR. 2013;184:279–85. doi:10.3233/978-1-61499-209-7-279.
Liu J, Mei J, Zhang X, Lu X, Huang J. Augmented reality-based training system for hand rehabilitation. Multimed Tools Appl. 2017;76(13):14847–67. doi:10.1007/s11042-016-4067-x.
Alankus G, Kelleher C. Reducing compensatory motions in motion-based video games for stroke rehabilitation. Human-Computer Interact. 2015;30:no. 3–4. doi:10.1080/07370024.2014.985826., “,”, , pp. 232–262.
Cheung J, Maron M, Tatla S, Jarus T. “Virtual reality as balance rehabilitation for children with brain injury: A case study,” Technol Disabil, 25, 3, 2013, doi:10.3233/TAD-130383.
LaBelle K. “Evaluation of kinect joint tracking for clinical and in-home stroke rehabilitation tools,” Univ. Notre Dame, 2011.
Wessel MJ, et al., “Iterative Design of an Upper Limb Rehabilitation Game with Tangible Robots,” pp. 241–250, 2018, doi: 10.1145/3171221.3171262.
Delbressine F, et al., “Motivating arm-hand use for stroke patients by serious games.,” 34th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., vol. 2012, pp. 3564–3567, 2012, doi: 10.1109/EMBC.2012.6346736.
Apted T, Kay J, Quigley A, “Tabletop sharing of digital photographs for the elderly,” in Conference on Human Factors in Computing Systems - Proceedings, 2006, vol. 2, doi: 10.1145/1124772.1124887.
Sun Q, Gonzalez E, Abadines B, “A wearable sensor based hand movement rehabilitation and evaluation system,” Proc. Int. Conf. Sens. Technol. ICST, vol. 2017-Decem, pp. 1–4, 2018, doi: 10.1109/ICSensT.2017.8304471.
Zondervan DK, et al. Home-based hand rehabilitation after chronic stroke: Randomized, controlled single-blind trial comparing the music glove with a conventional exercise program. J Rehabil Res Dev. 2016;53(4):457–72. doi:10.1682/JRRD.2015.04.0057.
Yang G, et al., “An IoT-Enabled Stroke Rehabilitation System Based on Smart Wearable Armband and Machine Learning,” IEEE J. Transl. Eng. Heal. Med., vol. 6, no. February, pp. 1–10, 2018, doi: 10.1109/JTEHM.2018.2822681.
Friedman N, Rowe JB, Reinkensmeyer DJ, Bachman M. The manumeter: a wearable device for monitoring daily use of the wrist and fingers. IEEE J Biomed Heal informatics. 2014;18(6):1804–12. doi:10.1109/JBHI.2014.2329841.
Rashid A, Hasan O. “Wearable technologies for hand joints monitoring for rehabilitation: A survey,” Microelectronics J, 88, 2019, doi:10.1016/j.mejo.2018.01.014.
Thibaut A, Chatelle C, Ziegler E, Bruno MA, Laureys S, Gosseries O. “Spasticity after stroke: Physiology, assessment and treatment,” Brain Inj, 27, 10. 2013, doi:10.3109/02699052.2013.804202.
Jiang S, et al. Feasibility of wrist-worn, real-time hand, and surface gesture recognition via sEMG and IMU Sensing. IEEE Trans Ind Informatics. 2018;14(8):3376–85. doi:10.1109/TII.2017.2779814.
Chen M, Cheng L, Huang F, Yan Y, Hou ZG, “Towards Robot-Assisted Post-Stroke Hand Rehabilitation: Fugl-Meyer Gesture Recognition Using sEMG,” 2017 IEEE 7th Annu. Int. Conf. CYBER Technol. Autom. Control. Intell. Syst. CYBER 2017, pp. 1472–1477, 2018, doi: 10.1109/CYBER.2017.8446436.
Cesqui B, Tropea P, Micera S, Krebs HI. “EMG-based pattern recognition approach in post stroke robot-aided rehabilitation: A feasibility study,” J Neuroeng Rehabil, 10, 1, 2013, doi:10.1186/1743-0003-10-75.
Liu L, Chen X, Lu Z, Cao S, Wu D, Zhang X. Development of an EMG-ACC-based upper limb rehabilitation training system. IEEE Trans Neural Syst Rehabil Eng. 2017;25(3):244–53. doi:10.1109/TNSRE.2016.2560906.
Lu Z, Chen X, Li Q, Zhang X, Zhou P. “A hand gesture recognition framework and wearable gesture-based interaction prototype for mobile devices,” IEEE Trans Human-Machine Syst, 44, 2, 2014, doi:10.1109/THMS.2014.2302794.
Shull PB, Jiang S, Zhu Y, Zhu X. “Hand Gesture Recognition and Finger Angle Estimation via Wrist-Worn Modified Barometric Pressure Sensing,” IEEE Trans Neural Syst Rehabil Eng, 27, 4, 2019, doi:10.1109/TNSRE.2019.2905658.
Cai P, et al. Locally coupled electromechanical interfaces based on cytoadhesion-inspired hybrids to identify muscular excitation-contraction signatures. Nat Commun. 2020;11(1):1–12. doi:10.1038/s41467-020-15990-7.
Witmer BG, Singer MJ. Measuring presence in virtual environments: A presence questionnaire. Presence Teleoperators Virtual Environ. 1998. doi:10.1162/105474698565686.
Mohammadzadeh FF, Liu S, Bond KA, Nam CS, “Feasibility of a Wearable, Sensor-based Motion Tracking System,” Procedia Manuf., vol. 3, no. Ahfe, pp. 192–199, 2015, doi: 10.1016/j.promfg.2015.07.128.
Dehem S, et al., “Robotic-assisted serious game for motor and cognitive post-stroke rehabilitation,” 2017 IEEE 5th Int. Conf. Serious Games Appl. Heal. SeGAH 2017, no. i, 2017, doi: 10.1109/SeGAH.2017.7939262.
Nissler C, et al., “VITA - An everyday virtual reality setup for prosthetics and upper-limb rehabilitation,” J Neural Eng, 16, 2, 2019, doi:10.1088/1741-2552/aaf35f.
Di. Tǎut S, Pintea JPWR, Roovers MA, Mañanas, Bǎban A. “Play seriously: Effectiveness of serious games and their features in motor rehabilitation. A meta-analysis,” NeuroRehabilitation, 41, 1, 2017, doi:10.3233/NRE-171462.
Fugl-Meyer A, Jääskö L, I L, Olsson S, Steglind S. The post-stroke hemiplegic patient. 1. a method for evaluation of physical performance. ” Scand J Rehabil Med. 1975;7(1):13–31. doi:10.1038/35081184.
Gladstone DJ, Danells CJ, Black SE. The Fugl-Meyer Assessment of Motor Recovery after Stroke: A Critical Review of Its Measurement Properties. Neurorehabilitation Neural Repair. 2002. doi:10.1177/154596802401105171.
Wolf SL, Catlin PA, Ellis M, Archer AL, Morgan B, Piacentino A. “Assessing Wolf Motor Function Test as outcome measure for research in patients after stroke,” Stroke, 32, 7, 2001, doi:10.1161/01.STR.32.7.1635.
Oskoei MA, Hu H. Support vector machine-based classification scheme for myoelectric control applied to upper limb. IEEE Trans Biomed Eng. 2008. doi:10.1109/TBME.2008.919734.
Rechy-ramirez EJ, Hu H. “Stages for Developing Control Systems using EMG and EEG Signals: A survey,” Comput. Sci. Electron. Eng., p. 33, 2011.
Englehart K, Hudgins B, Robust “A. Real-Time Control Scheme for Multifunction Myoelectric Control. IEEE Trans Biomed Eng. 2003. doi:10.1109/TBME.2003.813539.
Proakis JG, Monolakis DG, Digital signal processing: principles, algorithms, and applications. 1996.
Kaufmann P, Englehart K, Platzner M, “Fluctuating EMG signals: Investigating long-term effects of pattern matching algorithms,” in 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC’10, 2010, doi: 10.1109/IEMBS.2010.5627288.
Zhang T, Fulk GD, Tang W, Sazonov ES, “Using decision trees to measure activities in people with stroke,” in Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 2013, doi: 10.1109/EMBC.2013.6611003.
Zhang Z, Fang Q, Gu X. Objective assessment of upper-limb mobility for poststroke rehabilitation. IEEE Trans Biomed Eng. 2016;63(4):859–68. doi:10.1109/TBME.2015.2477095.
Bochniewicz EM, Emmer G, McLeod A, Barth J, Dromerick AW, Lum P. “Measuring Functional Arm Movement after Stroke Using a Single Wrist-Worn Sensor and Machine Learning,” J Stroke Cerebrovasc Dis, 26, 12, 2017, doi:10.1016/j.jstrokecerebrovasdis.2017.07.004.
Cai S, et al., “SVM-based classification of sEMG signals for upper-limb self-rehabilitation training,” Front Neurorobot, 13, 2019, doi:10.3389/fnbot.2019.00031.
Leonardis D, et al., “An EMG-controlled robotic hand exoskeleton for bilateral rehabilitation,” IEEE Trans Haptics, 8, 2, 2015, doi:10.1109/TOH.2015.2417570.
Lipovsky R, Ferreira HA. “Hand therapist: A rehabilitation approach based on wearable technology and video gaming,” no. February, 1–2, 2015, doi:10.1109/enbeng.2015.7088817.
Lee SW, Wilson KM, Lock BA, Kamper DG. Subject-specific myoelectric pattern classification of functional hand movements for stroke survivors. IEEE Trans Neural Syst Rehabil Eng. 2011;19(5):558–66. doi:10.1109/TNSRE.2010.2079334.
Zhang X, Zhou P. High-density myoelectric pattern recognition toward improved stroke rehabilitation. IEEE Trans Biomed Eng. 2012;59(6):1649–57. doi:10.1109/TBME.2012.2191551.
Castiblanco JC, Ortmann S, Mondragon IF, Alvarado-Rojas C, Jöbges M, Colorado JD. “Myoelectric pattern recognition of hand motions for stroke rehabilitation,” Biomed Signal Process Control, 57, 2020, doi:10.1016/j.bspc.2019.101737.
Santello M, Flanders M, Soechting JF. “Postural hand synergies for tool use,” J Neurosci, 18, 23, 1998, doi:10.1523/jneurosci.18-23-10105.1998.

Download PDF

Version 1

posted

You are reading this latest preprint version

Wearable Multimodal - Serious Game System For Hand and Cognitive Rehabilitation After Stroke

Status:

Version 1

Abstract

Figures

Introduction

Methods

A. System overview

B. Upper extremity movement selection

C. Prototype design

D. Testing protocol

E. Serious games design

F. Feature extraction

G. Classification algorithm

H. Statistical analysis

Results

Discussion

Conclusion

Abbreviations

Declarations

Acknowledgment

Authors’ contributions

Funding

Availability of data and materials

Ethics approval and consent to participate

Consent for publication

Competing interests

References

Supplementary Files

Status:

Version 1