A Participants
Two participants with quadriplegia were recruited for the study after providing informed consent. The study protocols were approved by the Institutional Review Board of Northwell Health (Great Neck, NY). Participant 1 was a 32 year old male, injured 6 years prior, with a C4/C5 ASIA (American Spinal Injury Association) B injury. He participated in 10 sessions, out of which 7 sessions were used to record 2D and 3D arm movement trajectories. During the remaining 3 sessions, grasping intentions were decoded online (in real-time) and used to drive a custom neuromuscular stimulator with textile-based electrodes housed in a sleeve (12). This in turn allowed the participant to perform functional movements (e.g. eat a granola bar). Participant 2 was a 28 year old male, injured 10 years prior, with a C4/C5 ASIA A injury. He participated in 3 sessions, which involved 2 training and 1 online testing session.
B Experiment Setup and Data Collection
Participants were seated with their hands initially resting on a table. A wireless sensor module was attached to the wrist of their arm using a Velcro strap. While both participants were bilaterally impaired, each still possessed residual movement that allowed reaching with at least one of their arms and was eventually used for the study. The sensor module consisted of a 32-bit ARM microcontroller unit (MCU) from Adafruit (Feather Huzzah32) and a Bosch SensorTec BNO055 9-axis IMU. The IMU has a built-in processor and algorithms to estimate its orientation and perform gravity compensation in real-time to produce linear acceleration in three orthogonal directions. Linear acceleration along the X, Y, and Z axes was available externally via an I2C interface. A flexible printed circuit board was designed to interconnect the IMU with the MCU as shown in Fig. 1B. Data was continuously streamed from the MCU at 50 Hz via Bluetooth to MATLAB 2019a running on a desktop PC and stored for offline processing.
During the experiments, verbal cues associated with different 2D and 3D movement trajectories were randomly called out to the participant. The participants were instructed to perform the reaching trajectories starting from the edge or corner of the table and move towards the center, using smooth movements that were up to a second long. Three different 3D reaching trajectories: a sideways arc (e.g. reaching for a cup or bottle, Fig. 1A), a vertical arc (e.g. reaching for a pen or marker lying on a table), and a corkscrew motion were trained. Additionally four 2D trajectories (performed in the horizontal plane) corresponding to well-known English and Greek letters: S, ε (epsilon or E), γ (gamma), and M were trained. Experiments were conducted in blocks of 18–20 trials and sufficient breaks were given between blocks to minimize participant fatigue. Initially, the participants were asked to perform only S and ε trajectories because these were simple to learn and didn’t cause fatigue. Later, once the participants became comfortable with moving their arm, we included additional 2D and 3D trajectories. Thus, in our final datasets there was a higher percentage of 2D trajectories (especially, S and ε) than the remaining trajectories.
During online (real-time) testing of grasp inference, participants also wore a custom-built fabric sleeve with 128 textile-based electrodes over their forearm to receive neuromuscular stimulation, which activated the appropriate extensor and flexor muscles to open their hand and evoke different grasps (e.g. pinch, cylindrical, etc.). Neuromuscular stimulation was provided by a custom proprietary, battery-operated, 8-channel, voltage-controlled stimulator, with a stimulation pulse frequency of 20 Hz. The stimulation channels were mapped to individual or multiple electrodes on the fabric sleeve, in order to evoke various finger flexion and extension type movements. By grouping multiple stimulation channels and sequencing their activation profile, we could program different grasp types such as cylindrical and pinch grasps. Figure 1C shows still images of an SCI participant using a simple 2D trajectory (e.g. M) to grasp and eat a granola bar with his paralyzed hand.
C Data Processing and Machine Learning
The 3-axis linear acceleration obtained from the IMU was band-pass filtered (Butterworth, 8th order, 0.2–6 Hz) and processed offline for identifying training samples. The magnitude of the 3-axis acceleration vector was used to identify onset of movement by setting a threshold of 0.95 g. The movement onsets were then used to segment the acceleration data over time along the X, Y, and Z axes into windows ranging − 0.1 s to 0.9 s with respect to onset. Each trial was visually confirmed to be free from any noise artifacts or if it exceeded the 1 s window and such trials were excluded from further analysis. Next, two time series classifiers based on either a Dynamic Time Warping (DTW) distance measure or Long Short Term Memory (LSTM) network algorithms were trained separately for 2D and 3D trajectories.
The DTW algorithm optimally aligns a sample trajectory with respect to a previously determined template trajectory such that the Euclidean distance between the two trajectories is minimized. This is achieved by iteratively expanding or shrinking the time axis until an optimal match is obtained. For multivariate data such as acceleration, the algorithm simultaneously minimizes the distance along the different dimensions using dependent time warping (13). In our DTW-based classifier, this algorithm was used to compute the optimal distance between a test sample and pre-defined templates associated with the 2D and 3D trajectories. Ultimately, the template with the smallest optimal distance to the test sample, was selected as the classifier’s output. Since the classifier’s output is dependent on the quality of its templates, we used an internal optimization loop to select the best template trajectory from a set of training trajectories. Within this loop, the DTW scores of each training sample with every other training sample was computed. Then the training sample with the least aggregate DTW score, was chosen as the template for that trajectory.
To implement the LSTM network we used MATLAB R2019b Deep Learning Toolbox with default values for most parameters. Specifically, an LSTM network comprising of a single bidirectional layer with 10 hidden units was used. This transformed the 2D or 3D linear acceleration data into inputs for a fully connected layer whose outcome was binary, i.e. 0 or 1. Next, a softmax layer was used to determine the probability of multiple output classes. Finally, the network output mode was set at ‘last’, so as to generate a decision only after the final time step has passed. This allowed the LSTM classifier to behave similarly to DTW and classify trajectory windows. During training of the LSTM network weights, an adaptive moment estimation (ADAM) solver was used with a gradient threshold of 1 and maximum number of epochs of 200. Since all the training and validation data were 1 second long, zero padding was not used.
During real-time classification of arm trajectories, the linear acceleration signals were filtered and processed in real-time using a MATLAB script that looped at 50 Hz. Within the loop, the acceleration data was divided into 1 second long segments with 98% overlap. To demonstrate proof-of-concept, only the DTW-based classifier was implemented and was designed to compare the incoming acceleration windows with 2D trajectories. If the optimal distance between trajectories were below 10 units (empirically determined), then positive classification was issued, which then triggered our custom neuromuscular stimulator to perform a complete movement sequence of opening and closing of the hand.