System Description
The EPES system is a mechatronic device weighing 140 kg that provides 3-dimensional (3D) balance perturbations tilts during in-place walking (see Fig. 1). It is comprised of a stationary elliptical system that is mounted on a motion platform that consists of two DC motors and gears connected to them. The motion platform allows two degrees of freedom, roll and pitch (left-right and forward-backward tilts, respectively) during in-place walking in a safe environment. The EPES system provides a maximum 3D perturbation tilt angle of a maximum 8º (in all directions) with 3 options of rotational strength. The motor that performs the unannounced perturbation tilts is controlled by a motion control system which is controlled also by a camera system (ZED 2 from STEREOLABS), which are both controlled by a main computer software program. The computer program is on the host PC which also serves as a user interface. By a program command, the motion control system directs the motion platformer rotation based on the training plan. The computer software program allows the trainer to determine EPES perturbation parameters such as the tilt angle of perturbation, rotational strength, direction of rotation, and the interval time between perturbations.
Based on the ZED2 camera, the software is able to capture and identify effective balance reactive reactions after unannounced balance perturbation is given. In case an appropriate balance reactive response is detected by the software i.e., counter-rotation action of body segments, the tilt rotation (i.e., the perturbation) is stopped, and the motor returns the EPES system to its horizontal position (i.e., zero position) by motor counter-rotation. In this way, the trainee gets a real-time feedback. The immediate real-time balance reactive response feedback may help the trainee to learn implicitly how to react successfully to unexpected perturbation and provides the best possible motor learning implementation [49].
Main EPES system components
The motion platform
The motion platform is an off-the-shelf product purchased through the DOF reality company (dofreality.com) [50]. It is consisting of a platform (on which the elliptical is placed), two 24V DC brushed motors and gears connected to them (see Fig. 2). To link the rotation of the motors to the tilt of the platform, a four-bar linkage mechanism is attached to each output shaft of the gear. This mechanism allows control over the position of a certain point in a closed kinematic system [51]. Thus, through rotation combinations of the motors, it is possible to control the tilting direction of the platform. The two DC motors with a power rating of 200W, and the gear is NMRV gear type with a gear ratio of 1:50. Also, the motors have a maximum speed of 105 degrees per second and peak torque of 25 Nm. On the other side of the output shaft of the gear, there is a position sensor that monitors the rotation of each motor. The motion platform is controlled by an Arduino which is located inside an electrical cabinet (the motion platform controller) that came with the platform. The electrical cabinet connects with a USB cable to the PC for the purpose of connecting to an application programming interface (API) called Sim Raising Studio (SRS). This API enables programming of the motion platform using Python software in a convenient and simple way.
The stereo camera
The ZED 2 is a stereo camera that mounted at a horizontal plane at a height of 1.05 [m] and 1.8 [m] in behind the trainee’s standing position for the best motion capture of the trunk and arms’ reactions. The ZED 2 camera is a depth camera in which there is a ready-made and easy-to-use body frame model that maps the human body into a single kinematic chain (see Fig. 3). This way the ZED 2 camera captures the body posture in real time and allows implementation of the upper-body balance reactive responses to be monitored and increased. The camera sensors collect the trainee's body movements with respect to the EPES system state and analyzes their responses to ongoing events.
During training, the system calculates predefined angles (α1-α5) of the trainee's body. The angles are calculated using the information received from the camera about the position of the trainee's joints (Fig. 3). We used the upper-body joints skeleton stick figure that the camera provides because the balance, trunk, and arm movements are the training target. While training, the system calculates the desired angles (α1- α5) and saves them in a CSV file (see Fig. 4) to be able to observe the trainee's reactions and analyze them after training is over. Also, the system collects and saves the tilt angles (α6, α7) that are sent to the motion platform in a CSV file as well. When: α1) Shoulder angle: The angle of the line between the trainee's two shoulders and the ground on the left-right axis.; α2) L-R back angle: The angle of the line from the trainee’s center of mass (CoM) to the trainee's chest relative to the line vertical to the ground on the left-right axis; α3) F-B back angle: The angle of the line from the CoM to the chest relative to the line vertical to the ground on the back-front axis ; α4) Left elbow angle: The angle between the line from the left elbow to the left shoulder joints of the trainee and the line vertical to the ground on the left-right axis; α5) Right elbow angle: The angle between the line from the right elbow to the right shoulder joints of the trainee and the line vertical to the ground on the left-right axis; α6) System roll angle: The angle of the motion platform on the left-right axis; α7) System pitch angle: The angle of the motion platform on the front-back axis.
Each training consists of two phases: calibration and perturbation. First, by calibrating, we identify the effective balance response threshold during in-place walking on the elliptical personalized for each trainee. Secondly, during the perturbation phase, when the trainee responds well to perturbations, the system provides a customized intrinsic sensorimotor cue. This is done by stopping the perturbation immediately and returning the EPES system to its horizontal position. We used angles α1 and α2 in the real-time training process to detect the trainee's body position with respect to the calibration angles to check if an effective balance reactive response was performed when the system performs roll (left-right) perturbation. We used angle α3 for the same purpose when the system performs pitch (forward-backward) perturbation. Angles α4-α7 were not used during the real-time training process but are shown in post-training graphs for advanced post-training analysis. Monitoring the trainee's balance responses according to the system movement over time can indicate the implementation of skill acquisition and the motor learning progress of the balance upper-body reactive responses. The data of angles α1–α5 reveals to the trainer all the information about the balance reactions. For example, the arm reactions (α4 and α5), which are part of the training goal, are reflected in all types of perturbations. But the other angles that are measured represent more reliable information about the nature of the response according to the perturbation; Therefore, the calculation is not based on these angles. However, it can clarify the entire response to perturbations.
The control system (PC)
The computer program serves as the control system and runs on the host PC. Using Python software, it is possible to control both systems, the ZED 2 camera, and the motion platform, and even to communicate between them. Python software activates both systems simultaneously. Thus, the system knows the position of the user and, as a result, the angles of his body due to the ZED 2 camera. And the desired perturbation can be created while controlling the motion platform. The motion platform controlled thanks to its API (SRS) as mentioned earlier. With the help of Python software, you can send simple commands to the SRS for the type of perturbation, the size of its angle and its intensity. The SRS receives the information and sends the translated information to the controller of the motion platform that takes care of care of moving the motors according to the command sent.
Safety harness
Safety is an extremely important issue since, in this training, we apply unexpected perturbations that may cause older people to fall off the EPES system. A safety harness keeps the trainee secure during training. For the safety system there are two options to secure the trainer according to the nature of the required system: mobile or stationary.
For mobile system (see Fig. 1a)
A waist harness is used to secure this system. Four adjustable straps are attached to this harness, two on each side (right and left), when each of which is fixed to one of the four side rods of the system. This system is mobile because it does not depend on the site in which it is placed.
For stationary system (see Fig. 3)
A body harness is used to secure this system. This safety harness is suspended from the ceiling by two ropes above the trainee. This system is stationary because it is necessary to fix the straps to which the harness is attached to the ceiling of the site where the system is placed. The experimenters were secured to this system.
In both systems, the harness is slightly loose to be safe, and not restrict balance response, but in case the trainee fails to recover, the safety harness will arrest the fall.
Software
Currently, the program can be activated with Python software and the trainer can select the type of training by answering questions at the beginning of the training run.
Creating training
When running the program, at first the system asks the trainer to enter the length of the workout in minutes. This time includes the first two minutes of system calibration. Following that, the trainer defines training content by defining segments of perturbations based on his preferences. Each perturbation is defined separately and added in chronological order to the training. For each segment of perturbations, the trainer sets the duration of the segment, the type of the perturbations (the tilting direction, the size of its angle, its intensity (range between 1 to 3), and the delay time between each two consecutive perturbations (frequency). Defining segments ends when all the defined training minutes have been used. After creating the training, the system is activated, and the trainee is instructed to start walking on the elliptical. In the first two minutes of the training, the EPES is calibrated according to the trainee trunk motion, so there are no perturbations in this part. The purpose of this part is to define the range of user trunk and shoulders angles when the user walks on the system while it is at rest (no perturbations). After this time, the perturbations begin according to the training trunk and shoulders segments that the trainer created. Also, the system saves the calculated trunk and shoulders angles of the trainer and the system angles perturbations in a CSV file.
Types of perturbations
The EPES system provides 3D tilting balance perturbations that aim to challenge specifically trunk, and upper body balance reactive reactions but also lower-limb reactive responses are triggered. When tilting, the trainee's CoM aside rapidly, the trainee is forced to decelerate the CoM by responding reactively with lower-limbs, trunk and upper limb balance reactive response during in place walking. Balance perturbations are provided in two forms: 1) machine-induced unannounced external perturbations and 2) self-induced internal perturbations during hands-free in place walking. The external perturbations are controlled programmed machine-induced and are ranged from low to high magnitude (0°-80 for each direction). They can be programmed expectedly as a block perturbation training (fixed time, order, and magnitude), or be given unexpectedly as random perturbation training (in onset, direction, and time interval) which evoke fast trunk and upper-body reactive balance motion. The internal self-induced perturbations are provided by the trainee during self-pace in place walking on the EPES. These situations fall under proactive balance control training when the trainee shift his/her body weight during the elliptical in place walking.
EPES system communication and activation
To activate the EPES system first you need to make sure that the PC has access to the motion platform controller, i.e., tarn on the motion platform controller and open the SRS application. Next, for running a training program, the trainer first creates a customized perturbation training program with the user interface (activate Python software). When, the computer program runs the training program by utilizing the motion control system and the ZED 2 camera, both controlled by the computer program. It is also necessary to enable communication between the two systems (the camera and the motion platform) to produce implicit feedback for the trainee. The camera transmits information about the trainee’s shoulder trunk and arms movements, while the motion control system is responsible for receiving information and delivering commands to the motors. In case the software detects an appropriate trunk balance reactive response during a perturbation i.e., counter rotating the trunk and shoulder, the perturbation is stopped, and returns to its horizontal position. Also, with the help of the graphics library (GL), the system displays the camera image with the body frame (body sticks diagram) throughout the training session. The desired mode of activation and communication between the camera and the motion platform is shown in Fig. 5.
After the training program, the trainer may review the trainee's balance reactive movement graphs, check if an effective reactive response was triggered and balance was recovered, and the data collected in order to determine how to proceed in the next training session.
The 3D camera function during the training:
The training session is divided into two stages: 1) the calibration stage (the first 2 minutes) and 2) the balance exercise stage.
1) The calibration stage - for automatically customizing the EPES system to the trainee who is currently using it. It consists of two parts:
A) The trainee's adaptation phase – 20 seconds of warning-up slow pedaling to allow the trainee to ease into a comfortable position. In this phase, the computer program does not make any reference point calculations due to the noisy data that was gathered by the camera until the trainee gets used to walking on the elliptical.
B) Measuring and calculating each individual upper body sway (forward-backward and left-right) trunk angle; the body sway base noise range (natural angles) – 100 seconds of self-paced pedaling while the ZED 2 camera provides data to the computer program for calculating angles α1-α5. At the end of the calibration stage, the reference angles that will be used later to determine whether the trainee's reactive balance responses were effective or not will be obtained. Based on the defined natural body sway angles, the computer program detects if the trainee responded to a given perturbation or whether their current trunk and shoulder angle was a part of natural movement during pedaling (i.e., into the body-sway-base-noise range). Thus, first, the computer program records the data of the α1-α5 angles during the 100-seconds of calibrated self-paced in place walking. And secondly, throughout the calibration phase, the system updates the noise domain of angels α1-α3 according to the update of the maximum trunk and shoulder angles obtained from the calculation. The natural movements are the angles at which the trainee is naturally walking in place, and this is necessary because often older people naturally tend to lean a few degrees to either the left, right, forward, or backward side.
2) The balance exercise stage - contains block and expected or random and unexpected balance perturbations. When a new perturbation is executed, the computer program compares between the trainee's trunk and shoulder angles (which are calculated and kept continuously throughout the training) to the natural angles range. This is to see if there has been a significant movement other than a pedaling movement.
For separating a balance reactive response to a normal self-induced voluntary pedaling movement following a perturbation the system programed to check these conditions: If the EPES motion platform angle and the trainee’s body are leaning in opposite directions, and if the current body angle is larger than the natural angle of this side self-induced pedaling. This tilt requires the trainee for a larger distinct balance reactive response to recover their balance for passing above the response threshold and stopping the perturbation by turning the EPES angle back to its neutral position (horizonal to the ground). This option deals with a trainee who exhibits large trunk and shoulders angles during the exercise session versus the calibration phase.
Exploring balance reactive responses
Here, we present results of a proof of concept pilot study to explore whether the EPES software was able to identify balance reactive responses. Two people young adults were exposed to unannounced right-left and anterior-posterior balance perturbations while in-place walking on the EPES system. The tilting perturbations evoked balance reactive trunk, head, and arm movements almost always in the opposite direction of the perturbation to quickly move the upper body's CoM toward the base of support provided by the EPES pedals. Both participants performed upper body balance reactive responses during the training session.
The trunk, shoulder and arms reactive responses during the training sessions is demonstrated in Fig. 6 and Fig. 7. Upper-bodies balance reactive responses are presented by the shoulder line and trunk/back and arms angles. These angles were found to be the best parameters to distinguish the presence of an upper-bodies balance reactive response.
In Fig. 6, the participant pedaled with, and then without, holding the elliptical system handlebars (i.e., hands-free). Than the participant was exposed to 8º right announced perturbations during hands-free in-place pedaling (Fig. 6A). Figure 6B shows that the EPES software was able to accurately identify shoulder and trunk/back balance reactive responses and return the system to its initial horizontal plane (Fig. 6B).
In Fig. 7, the participant pedaled with, and then without, holding the elliptical system handlebars (i.e., hands-free). Than the participant was exposed to 8º left announced perturbations during hands-free in-place pedaling (Fig. 7A). Figure 7B shows that the EPES software was able to accurately identify shoulder and trunk/back balance reactive responses and return the system to its initial horizontal plane (Fig. 7B).
In Fig. 8, the participant pedaled with, and then without, holding the elliptical system handlebars (i.e., hands-free). Than the participant was exposed to 8º backward announced perturbations during hands-free in-place pedaling (Fig. 8A). Figure 8B shows that the EPES software was able to accurately identify trunk/back and arms balance reactive responses and return the system to its initial horizontal plane (Fig. 8B).
Observing and analyzing a trainee's balance reactive performance can be useful for making a clinical decision regarding the progress of rehabilitation, and for indicating skill acquisition and motor learning progress of the balance upper bodies reactive responses. Additional software enables the trainer to observe the kinematic data of a specific trainee in a specific training session. It presents kinematic graphs and a moveable timeline that allows the trainer to observe the upper bodies angles and the camera's body stick figure at every timestamp, compared to the horizontal angle position of the EPES at that timestamp.