In an online survey for the PIL and TL of teams preparing for the CYBATHLON 2020 Global Edition, we collected data from the unique pool of developers and users of state-of-the-art AAT, with the objective of shedding light on the influence of a competition like CYBATHLON on the development and acceptance of AAT. More specifically, we analyzed the PIL role during device development, the actual daily usage and usability ratings of the devices, as well as the effects of those user-centered design characteristics on the individual CYBATHLON race performances.
Pilot involvement in CYBATHLON teams
One of the main results of this study was to confirm that across all investigated disciplines, more than 85% of the PIL played a vital role in large parts of the device development. From the 35 PIL surveyed, 57% have been involved in their team's endeavors for more than one year. This somewhat implies that for most of the teams, the PIL role is not limited to testing and training the finalized device to compete at the event but rather to be an integral part of the team during development. Still, when looking at the individual UCD phases, we can understand that – not surprisingly – the pilots are primarily involved in empathizing and defining the problem or goal to be addressed and when evaluating the technical solution (mostly referred as “training”). Especially in the last weeks before the competition, the PIL involvement becomes more intensive, with multiple PIL reporting more than 20 hours of training per week. An increase in training intensity up to daily practice in the last days before the competition were previously reported by several teams participating in the 2016 competition [19, 23]. Other experiences, such as reported from the VUB-CYBERLEGS team in 2016, show that as little training as a total of 14 hours with the device can suffice for the PIL to successfully participate at the competition [21]. An intensive involvement of the PIL in any CYBATHLON-related activities might be expected, but surely cannot be taken for granted. From anecdotal evidence shared in the past, we understand that in a majority of the participating teams, the PIL contribute on voluntary and non-paid basis, while following regular day-to-day lives and jobs. In the meantime, most technical leads (students, researchers, staff, industrial engineers, etc.) dedicate their daily work to device development. Reasons for the minor, or lacking PIL involvement in the technical development could therefore be: (i) limited resources, i.e., time and budget, (ii) limited technical expertise of PIL (iii) limited motivation or need. A reduced involvement of the PIL in the more technical phases of device development might be a reason for partial disagreement of the PIL's personal impact and consideration in the final design, as observed in the Likert Scale answers. Only in a few teams, the PIL simultaneously are among the leading engineers, or researchers behind the technical solutions to implement their needs and wishes directly [32, 33]. The employment status and actual workload of PIL within development teams remain somewhat undocumented, restricting a more detailed analysis.
In the majority of AAT developments, target users only get involved once the engineering teams have spent substantial resources developing and testing with neurologically intact, mock-users [33]. At this point, there is a high probability that the designs shot past their target of solving user-defined problems of daily life, and to do so with satisfactory end-user usability. It appears that the CYBATHLON may actively help tackling this issue, as many PIL get involved in the projects from early on. Involvement in technical development phases such as prototyping does not necessarily imply that the PIL have to program software, or solder electronics themselves. User-centered design can be supported with various collaborative design methods such as focus groups, workshops, storyboards, or cognitive walkthroughs to establish or discuss concepts, ideas, and prototypes [34]. In any stage of development of (advanced) assistive technology, user feedback is essential to maintain a user-centered focus by following up on user ideas, needs, or emotions [9, 14]. The combined insights on the PIL and TL perceptions and experiences on their involvement in AAT development suggest that the quality and quantity of user involvement might still be improved with additional co-creation and usability methods or be enhanced with more resources to incorporate the PIL closer to the hands-on development. Still, we can conclude that UCD is common practice among the CYBATHLON teams, as it also had been reported in experience reports from teams of the 2016 competition [23, 30]. The CYBATHLON as a platform promotes the exchange and collaboration between people with disabilities and AAT developers, thereby reaching one of its seminal objectives [12, 13].
Daily life applicability and usability
Strongly linked to user involvement, another main interest of this study was to investigate the daily life usage and subjectively rated usability of the AAT developed for the CYBATHLON. From the mean QUEST scores, we understand that, surprisingly, the TL appeared to be less satisfied with their developed technology than the actual end-users, the PIL. Specifically, in the disciplines that ask for more complex technical solutions, namely, the EXO and WHL races, the TL see plenty of room for improvement. This is in contrast to the PIL perceptions, which indicate an encouraging level of satisfaction (QUEST mean = 3.98) with the AAT usability [30]. Given that the QUEST is one of the most used scales for the subjective rating of perceived usability [31, 33], the mean QUEST scores from our data might additionally serve as a benchmark for other advanced assistive technologies. Still, besides all other means of measuring user experience and usability, the actual daily use – and hence technology acceptance and adoption – is arguably the most reliable usability outcome. With only 25% of all PIL using their AAT in everyday life, prevalent usability limitations and adoption barriers were observed. Given that the devices are often tailored to the individual PIL, it is somewhat surprising that only very few of them use their AAT in daily life, besides any activities related to the CYBATHLON. A large number of PIL reported limited availability as the main inhibiting factor, meaning that they do not have access to the technology in daily life. Additional adoption barriers such as too high costs or incompatibility with the daily environment were also prevalent. Such a limited availability and accessibility of AAT showcased at the CYBATHLON can be explained by the fact that most teams work with devices that are either exclusively built for the CYBATHLON, or existing research devices that are optimized for the specific race conditions. Only a minority of devices are commercially available and have regulatory approval to be used without technical supervision. Hardware and software functions are often optimized to overcome the exact obstacles or racetracks that are known to the participants months in advance. Usage of the devices outside of the controlled environment of a CYBATHLON obstacle course requires significantly more dynamic control, feature adaptability, and additional safety precautions. Still, given that the devices are often tailored to the individual PIL, it is somewhat surprising that only very few of them use their AAT in daily life, besides any activities related to the CYBATHLON. It thus remains difficult to judge if the specific developments towards a CYBATHLON participation only positively influence the daily life usability of the AAT. In terms of usability limitations, insufficient comfort and overcomplicated use (requiring technical assistance) were identified factors inhibiting daily use. From the factors listed, PIL indicated that the effectiveness (functionality, reliability, helpfulness, etc.) of their AAT is not the leading limitation, but rather that usage efficiency and overall satisfaction need to be improved to motivate for or enable daily use. These insights align with recent reports on the current state of usability of wearable robotic technologies [18, 31]. It is to be expected, that if such usability limitations cannot be addressed, target users will likely abandon their AAT soon after an eventual acquisition [2, 3].
CYBATHLON race performance indicators
Lastly, we were interested to investigate the potential link between performance in the CYBATHLON and user-centered design variables like user involvement intensity and perceived usability. For this purpose, we fitted a mixed model of several UCD variables expected to have an effect on the individual race performances at the CYB. We analyzed to what extent user involvement, usage- and training intensity, and usability ratings could be defined as performance indicators. Daily AAT use beyond CYBATHLON-specific training was found to be most positively affecting race performance (RR). PIL who can maneuver and control their device in daily life are thus likely to perform well at the CYBATHLON, indicating the racetracks’ relevances to activities of daily living. Also, we could identify that the duration of PIL involvement (involvement quantity) appears to be a main performance-indicator. The increasing, positive effect of long-term involvement, however, appears to reach a plateau with PIL contributions beyond four years. This does not indicate that involving a PIL for more than four years is not beneficial for a good performance at the CYBATHLON, but rather that there is no significant difference in the positive effect compared to involvements of 2-4 years duration. The re-occurring success stories from teams that participated in both competitions of 2016 and 2020 with the same PIL, like HSR enhanced (WHL), Cleveland (FES), TWIICE (EXO), or IHMC (EXO) confirm this identified benefit of long-term PIL involvement [23, 26, 37, 38]. Iterestingly, the quality of PIL involvement, i.e., their role within the five different development phases, was not found to significantly affect race performance. According to our data, a training intensity higher than two hours per week, as well as prolonged training over more than six months do not appear to significantly improve the performances at the 2020 competition. From reports of participants in 2016, however, we can understand that prolonged training might positively affect performances based on observations of reduced mental, or physical load, as well as reduced task times in the progression of prolonged task-specific training [21, 23]. The satisfaction of the PIL regarding the usability of their AAT also does not appear to be a significant factor affecting their race performance, indicating that the ability and willingness of the PIL to adapt to the available functions (or limited usability) of the device might be more crucial within the competition. We can link this observation to the results from the ARM discipline in both the 2016 and 2020 competition, where more simple technologies with limited number of features, in these cases body-powered prostheses, appeared to perform better. This may be due to the fact, that the PIL have to control less parameters, inevitably making simple systems easier to use in demanding, or stressful situations [38]. If such simplified systems also yield a high usability in daily life is yet to be investigated. A solid interpretation of the negative estimate that was linked to the interaction effect is rather difficult, as the two variables appeared to be more collinear than interactive. Overall, it is important to state that the performance of the teams at the CYBATHLON is not only influenced by the few UCD variables discussed in this work. Innovative technological solutions and competition tactics to complete the CYBATHLON tasks more efficiently, or effectively can potentially bring a large advantage to the PIL [31]. Still, it is the PIL and their expertise in handling the AAT under the pressure of a competition format, which decides what they can make out of the few chances during race qualifications and finals. Our explorative analysis solely aimed to investigate if and how UCD can help achieving a better performance at the CYBATHLON, and thus imply that it is likely to improve daily life usability and technology acceptance of AAT.
Limitations
As for most usability studies that use co-creators as subjects, our survey results are likely influenced by certain cognitive biases (e.g., recency bias, social desirability bias, etc.) limiting the respondent's fair and critical judgment [39]. Also, survey results are inherently subject to response biases such as recency-, recall-, or acquiescence bias. The cross-sectional design of the study limits the analysis of actual effects of PIL involvement in individual teams and projects. A longitudinal assessment would be needed, tracking the team's development and the user sentiment before, during, and after a CYBATHLON competition. This survey therefore only represents a starting point of user-centered design data collection at and around the CYBATHLON platform. Another limitation worth mentioning was the multi-lingual approach that was taken. Even though professionals and subject matter experts were involved in the translations, the fact that the surveys were offered and collected in nine different languages might have led to minor misunderstandings, or misinterpretations. However, the benefit of increasing sample numbers likely overweighs those limitations, also because numerous respondents appreciated and complimented the comprehensible translations. The still relatively small and varying sample sizes between disciplines can be listed as a factor limiting certain conclusions drawn from this study. Moreover, the sample group of TL did not solely consist of lead engineers but also of project managers, application specialists or other team members. The interpretations from the linear mixed model should be understood as an explorative analysis, as the user-centered design variables used for the model not only differed in factor levels and variable types, but also showed certain collinearities that make a clear statistical distinction of their effects very difficult. Also, the list of variables that influence the CYBATHLON performance is clearly more extensive, with many psychological (stress, cognitive load, etc.), as well as technical factors (team tactics, device malfunctions, etc.) not being addressed within this explorative analysis.
Implications and importance of the study.
Our survey results show that the CYBATHLON appears to achieve its conceptual goals of encouraging user-centered design among the participating teams by promoting inclusion and exchange between AAT stakeholders. While CYBATHLON successfully stimulates new developments of AAT, translation to devices that can be used in daily life outside of the competition remains a challenge. The limited daily use of the CYBATHLON-specific AAT also raises the question of whether developments specifically directed towards the races, and their clearly defined tasks, may somewhat be counterproductive to the generalization and robustness of the showcased functionalities for the more dynamic, unstructured environments in daily life. It remains difficult to assess to what extend the CYBATHLON directly influences the AAT market and thus end-user products for the general public. The competition and its outreach are attractive for sponsors, granting projects funding for developments towards the competition. After the competition, that funding alone is likely insufficient to allow for further developments to push towards commercialization of promising devices, or individual features. Besides monetary resources, it is crucial that conditions prevail which favor the transformation of prototypes into market-ready products such as structured knowledge transfers, reimbursement strategies, or AT market readiness. Besides individual projects that directly benefited from their CYBATHLON participation to eventually incorporate and commercialize their technology, like TWIICE [40], Caterwil [41], or Scewo [42], we understand that the majority of AAT showcased at the competitions might not become commercially available per se. However, the competitive spirit and challenges posed at the CYBATHLON are expected to bring forward technical innovations, social interactions, and manifold learnings that positively influence the overall technology readiness and transfer of novel AAT.
Recommendation for future research and applications
Based on our interactions with the teams of the CYBATHLON 2020 Global Edition competition, we could gather and analyze valuable data to investigate user involvement strategies, usability hurdles, and performance-indicating factors and further get an in-depth understanding of how the platform promotes user-centered design on a general level. The PIL are, one way or another, an active part of the development teams. We learned that the quality and quantity of this user involvement may differ among the teams and that a majority of PIL appeared to be only minorly involved in the technical development phases. Despite potentially limited resources, we would recommend to promote AAT target user involvement in the phases of conceptualization and prototyping in order to maintain focus on their needs. Further, we could indicate that PIL are likely to perform better at the competitions if they spend as much time with the devices as possible. Optimally, PIL should be able to use their AAT in daily life and therefore have access to it outside of the laboratory environment where training typically takes place. In the near future, we hope to see the number of PIL using their CYBATHLON-specific AAT in daily life to grow. For this to happen, both the teams and the competition rules have to make sure to maintain focus on solving actual daily life problems of people with disabilities. As mentioned above, this survey now marks a starting point of further, longitudinal assessments at, and around the CYBATHLON competitions. Eventually, we might be able to track which innovations translate to industry and how the CYBATHLON impacts the AAT in the long-term perspective. The insights generated by this survey are further transferrable to CYBATHLON-unrelated AAT developments, highlighting the importance of a user-centered mindset and development strategy.