Ethool, a tool to enable ethical and legal decision-making in research involving humans using new technologies in Europe: design and usability validation

doi:10.21203/rs.3.rs-2328902/v2

Download PDF

Research Article

Ethool, a tool to enable ethical and legal decision-making in research involving humans using new technologies in Europe: design and usability validation

https://doi.org/10.21203/rs.3.rs-2328902/v2

This work is licensed under a CC BY 4.0 License

Journal Publication

published 27 Nov, 2023

Read the published version in Universal Access in the Information Society →

You are reading this latest preprint version

Human participation in technological research projects has become more frequent in recent years. However, most of the technological researchers who organize such experimentation are unaware about the various ethical and legal implications involved. Further, when it is a medical device prototype that is being evaluated, the ethical and legal implications may be even more complex. A review of the laws, standards and recommendations in Europe has been drawn up regarding human participation in new technology co-design, development and evaluation, focusing on technological research for assistive and medical devices. An easy-to-use tool, called Ethool, which acts as a guideline for European technological researchers has been designed and developed. In this manuscript, the iterative usability evaluation of Ethool is explained as well as the improvements made, following participants’ feedback. Ethool was rated as acceptable and usable by participants obtaining a SUS score of 93.0 in its last evaluation. The tool is currently available to be used by any interested parties with the aim of gathering additional feedback.

Usability testing

ethics

tool development

evaluation

committee

It is known that techniques such as User Centred Design, Human Centred Design (HCD), or User eXperience are often applied in technological research as well as development projects with the aim of creating usable and acceptable products and/or services[1]. The participation of humans or the use of personal data in research has ethical implications which must, at the very least, be supervised and approved by an ethical committee [2] and it is fair to say that the applicable legislation is often difficult to be understood by researchers, especially in the area of engineering [3]. Universities such as Harvard, UC Berkeley, or Colorado Boulder in the USA, or also the University of Tübingen in Germany, for example, have special training for engineers who involve humans in their research [4–7]. However, it is still common to find a lack of understanding regarding these issues among technical researchers, especially in Europe. In previous research, an analysis of the applicable European Directives and Regulations, and recommendations was drawn up [8]. In the present manuscript the description of the tool designed following the analysis of results is included. The tool named Ethool was developed to help European researchers to take decisions in their projects and to determine the ethical and legal issues applicable to their research as it is often not easy to discern which applicable recommendations and regulations are experimentation dependent. This paper includes, not only the Ethool design and development, but also the iterative usability evaluation of it, including the improvements made following participants’ feedback.

The present study was carried out based on the previous analysis carried out by the authors on the recommendations and the European directives for research with technological developments including human involvement [3]. According to that initial work, several guidelines can be found such as the Declaration of Helsinki [9] which is well accepted nowadays, with its 2009 version being mentioned in the European Regulation relating to clinical trials with medicines [10] or the Oviedo convention that was converted into an agreement by the Council of Europe as a reference point for human dignity and rights in medicine and biology [11].

In Europe, Directive 93/42/CEE regulated the use of medical devices with research purposes for many years including research studies using those devices [12]. However, a new European Regulation (2017/745) on medical devices has been in force since May 2021 [13, 14]. In both documents, it is mentioned that the competent authority of the country where the clinical study will be carried out must approve all research with medical devices. The role of the competent authority is to, at least, evaluate the risk-benefit balance of the device for participants and determine whether the study is sufficiently relevant from a scientific point of view. Most countries of the European Union have their own competent authorities for the supervision of research with medical devices, and if this were not the case, the competent authority of another country could take on this responsibility [15].

In previously mentioned documents, it is specified that any research with medical devices must follow the recognized ethical principles and the proposed research must be approved by the corresponding Ethical Committee. Moreover, as in any research involving humans, the relevance of the informed consent procedure of the participants is also highlighted.

The European Regulation 536/2014 concerning clinical trials [10] came into effect in May 2016 and reinforces the mandatory nature of obtaining endorsement from an Ethical Committee in order to carry out any clinical research.

In addition to the presented legislation and regarding the research with medical devices, the UNE-EN ISO 14155 standard may be considered as a reference document in any country [16]. This document includes information regarding the ethical aspects to be considered when designing a study, the need to obtain the ethical committee approval and the description of the requirements for this purpose, and the most appropriate informed consent process for the study. In the same vein, the International Conference on Harmonisation is a board for the harmonization of technical requirements to be applied with humans by the pharmacists [17]. This board has developed several good clinical practice regulations that are recognized worldwide. The regulations and recommendations mentioned above were drawn up for clinical trials focused on drugs, but almost all of them should be applied in a similar way in any research with medical devices in which humans are involved. These rules also explain some guidelines for treating participants, including the informed consent process and the ethical issues to be borne in mind when designing the study protocols. They also underline the significance of compiling documentation such as a researchers’ manual, in order to provide all the details about the study and the medicines or medical devices to be evaluated.

2.1. Conclusions Of The Legislation Review

The main outcomes of the reviewed documentation about ethical and legal aspects to be taken into account for researchers who want to evaluate their technological developments with humans are summarized as follows:

The research to be carried out is not considered to be necessarily under the supervision of an ethical committee when there is no recording or treatment of personal data, or when human beings are not directly involved in the trials, or when participating they do not use a device, or no risk (either physical, or psychological) is expected and the participants target group is not considered vulnerable [18]. For example, when participants are filling in an anonymous survey in which the collected data is not of a sensitive nature. However, in the case that the researchers have any doubts, ethical committee support is recommended.

The study is considered as observational research when humans are involved and the devices and/or the software to be used are for medical purposes, these must have CE certification, be employed according to their intended use and following the manufacturer’s instructions. In this case, there is a need for Ethical Committee approval. This Ethical Committee must have the corresponding competences to approve that type of research.

The study is considered to be clinical research which needs Ethical Committee approval, and, in addition, the approval or the notification to the competent authority for clinical trials with medical devices, depending on the features of the system to be evaluated. In any case, the competent authority must be also informed about any relevant adverse events. The Ethical Committee involved in this case, must also have the competences to approve this type of research.

Otherwise, the study is considered a research project and therefore Ethical Committee approval is required in the remaining cases were humans and/or their personal data are involved.

In any case when humans and/or their personal data are involved, and moreover, when ethical approval is needed, an informed consent process for the participants would be also required. When the process is anonymous, participants must be informed about the project objectives and procedure. It is also mandatory to register clinical research with medical devices in the European database named EudraCT [19]. Other clinical trial public databases can be found, such as the Swiss National Clinical Trials Portal [20], or the platforms managed by the U.S. National Library of Medicine [21], or the World Health Organisation [22].

It was decided to build a flowchart according to the conclusions of the documentation review mentioned in the previous section (Fig. 1), which was used by several researchers within the same centre [8]. The flowchart was designed with the aim of identifying the ethical and legal aspects that apply to any experimentation with humans when technological developments are involved including medical devices or software developments. Users were confronted with several issues and had requested an easy tool based on the proposed flowchart. Based on that feedback the Ethool tool was built according to the flowchart and thus according to the documentation review.

This tool has been conceptualized based on a questionnaire format, in which the researchers are easily able to answer some simple questions and obtain a customized answer for their particular case. It was drawn up with the idea of giving a simple solution to researchers, mainly in the engineering and software computing area, in which the reviewed legislation is summarized, rather than requesting researchers to study the entire range of legislation and recommendations applicable to their developments. This solution can also give researchers the opportunity to register the features of their projects and to obtain a report according to their input.

As it was decided to build Ethool based on a questionnaire format, Microsoft Forms® software was selected [23]. This technology is easy to use and, moreover, an agreement to comply with the European Regulation on Personal Data treatment is included with the company in which it has been implemented and evaluated [24]. This technology, due to its responsive interfaces, also provides the option of being accessible from any device with a browser and an Internet connection. Ethool was designed with simple questions which can mostly be answered with a “Yes” or “No” and depending on the answer the next question will only be asked when needed, avoiding unnecessary steps. The early prototype versions of the Ethool were only developed in Spanish, as the tool evaluation was carried out only in that language.

Ethool’s usability was considered to be really important. According to Nielsen, usability comprises learnability, efficiency, memorability, error control, satisfaction and utility [25]. An evaluation of Ethool was carried out with the objective of measuring and improving usability. The evaluation was carried out in an iterative process including two trials with the same participants as proposed in any HCD methodology[1]. Between each trial, improvements were also developed on prototypes according to participants’ feedback. At the end of each trial, participants were requested to fill in the System Usability Scale (SUS) questionnaire in the Spanish version [26, 27]. The aim was to measure the usability of the tool by using this scale and to compare the results between the two prototypes including the improvements made between each trial, with the expectation of better results in the second prototype.

4.1. Participants

The target group for the tool are technological researchers who might involve humans in the design and development of their projects. For this reason, participants were recruited within the same research centre in which the tool was developed. However, researchers with different profiles (with and without engineering profile) were also considered in order to identify the difficulties and usability issues that might be faced by researchers from different backgrounds.

The inclusion criteria for participating were to have worked in the aforementioned research centre within the Health Unit, to be over 18 years old and to voluntarily accept to participate in the research, including signing the informed consent prepared for the study. Participants had to be fluent in Spanish, and the tool usability would be assessed for language comprehension. The foreseen distribution was for 60% of participants to be from the neuro-engineering and medical robotics area (engineering profile) and 40% to be non-researchers or researchers in other areas who were not used to doing direct research with humans (out of the target group).

Nielsen and Landauer proposed a mathematical model to calculate the number of participants needed to detect a system’s usability problems [28]. According to Nielsen’s proposal 15 users are sufficient to identify all the usability problems [29]. However, if the study is iterative, it is proposed that testing may be made with only 5 users. In the present study, a total of 12 people were recruited according to the inclusion criteria, and these participated in the two organized trials. One participant dropped-out during the study reporting that she did not have enough time to complete the tasks. Therefore, no more people than needed were involved according to the minimizing ethic principle.

In the end, 36% of the participants were researchers who were not used to involving humans in their research. It should be noted that 45% of the participants were male, 55% female and 0% non-binary. Information was also collected regarding the number of years that participants had spent working in the research centre, with the aim of identifying some differences between the people with and without experience in the field.

4.2. Methodology

As previously mentioned, the usability evaluation was designed with an iterative methodology divided into two trials in which the same participants evaluated Ethool and the feedback gathered from both trials was used to improve it.

The trials were organized as a summative test, where the participants were asked to perform several guided tasks using a script and other unguided tasks. Two different scripts were prepared to guide the participants in the tasks to be performed. 50% of the participants received Script 1 for the first trial to evaluate Prototype 1, and the other 50% were given Script 2. To evaluate Prototype 2 in the second trial, the scripts were alternated between the participants, with the aim of assessing any comprehension problems caused by the scripts that might affect the usability evaluation of the tool. Participants were requested to fill in a questionnaire with some additional open questions, with the aim of collecting qualitative information to improve the Ethool usability. Open questions were as follows:

Did you find any difficulty to understand the terminology used? Please, tell us which term(s) posed a problem.
Was it especially difficult for you to answer any of the questions? Which one(s)?
Did you have any difficulty to follow these guidelines?
Could you give us any feedback to improve this tool?

As previously explained, a SUS questionnaire was also completed by the participants at the end of each trial with the objective of measuring the usability of the tool.

As Ethool was available online, the participants were able to perform the trial from any device (computer, tablet or smartphone) with an Internet connection. Each participant was requested to use the tool at least twice for each evaluation trial, with an estimated total duration of 30 minutes including filling in the questionnaire. At the end of each use of the tool, a Likert of 5 points was also included to evaluate the process of a questionnaire-based system for the proposed evaluation, using the question “What do you think about this questionnaire?”.

The study protocol and informed consent for the proposed evaluation were approved on 23 May 2019 under protocol number M10_2019_128 (document 113/2019) by the Ethical Committee of the University of the Basque Country University for Research involving Human Beings (CEISH-UPV/EHU) which has number BOPV 32, 17/2/2014 [30]. The methods used for the participants’ involvement were carried out according to the Declaration of Helsinki [9], Oviedo Convention [11] and biomedical research Spanish law [31].

Between 22 and 33 uses of the tool were planned for each trial. Each script provided to the participants included 2 scenarios to be tested with the tool, and the user was requested to add a further scenario. In Prototype 1 evaluation, 32 uses were reported, and in Prototype 2, 27 uses, with an average of 2.9 and 2.45 uses per participant respectively, which could be considered as the mean of the expected use.

In the evaluation of Prototype 1, 8 of 11 participants considered that Ethool could help them in the future. It should also be noted that one of the participants reported some issues with the provided script in the evaluation of Prototype 1. Another reported some problems with the question about the personal data collection included in Prototype 1. Nevertheless, in the evaluation of the Prototype 2, which included some improvements over Prototype 1, all the participants considered it helpful.

The average time taken by participants using Prototype 1 for each project was 541 seconds, while the average time needed with Prototype 2 was 430 seconds, with a gain of 20.5%. Two positive conclusions could be taken from this result: 1) the improvements following the participants’ evaluation in the first trial and included in Prototype 2 made Ethool more effective; 2) participants experienced a very rapid learning process between the first and the second trials.

The participants were requested to fill in a short Likert of 5 points regarding their subjective perception of the use of Ethool in each trial. In Fig. 2, the collected data using this Likert is shown. In this figure, the percentage of each answer for both prototypes are represented. In the evaluation of Prototype 1 the average rating of the tool was 4.59, while in the evaluation of Prototype 2 was 4.67 out of 5, with a gain of 1.71%.

SUS questionnaire results were processed using a specific Python script which was developed according to guidelines [32] and is available in GitHub [33]. Participant ID09 did not rate Question 9 in the evaluation of Prototype 2, but following Bangor, Kortum & Miller recommendation [34] it was rated with a value of 3 in order to calculate the average with the rest of the recorded data. The SUS questionnaire results from both trials are represented in Fig. 3. There, it can be clearly seen that the mean calculation for the Prototype 2 evaluation, which included the improvements from the participants’ feedback from the first trial, achieved a higher rating than the former. Specifically, the obtained average rate were 87.5 and 93.0 for Prototype 1 and Prototype 2 respectively. According to the work published by Bangor, Kortum & Miller [34] the usability of both prototypes can be considered as “Excellent” as the averages of the two evaluations were higher than 85.0 points.

The improvements made after the first trial and therefore included in Prototype 2 are summarized in Table 1.

From the results gathered in both trials the low ratings from participants ID05 and ID07 are noticeable. In the evaluation of Prototype 1 Participant ID05 reported that Ethool was not sufficiently useful and requested more information in the answer to be added as improvement; a request which was implemented in Prototype 2. Participant ID07 reported some problems to understand the guidelines provided for the test thereby making it more difficult for him/her to use the tool. However, both participants rated Ethool much higher in Prototype 2, although they still evaluated the tool below the average of the rest of the participants.

On the other hand, participants ID06 and ID09 rated Prototype 2 lower than Prototype 1. In the case of ID09, the final calculation was lower because this participant did not provide a rating for Question 9. The remaining answers from this participant were very positive, so it could be considered that she would have rated it with a better value than 3, which was used for the average calculation. In the case of ID06 there is no noticeable reason to explain this difference. Moreover, this participant commented that the tool had improved in Prototype 2 compared to the previous version.

Regarding the profile of the participants, the division between neuro-engineering and medical robotics and other areas, as well as the number of years spent in the company were considered relevant, as the language used in some descriptions was fairly technical. However, there was no evidence of any connection between the results and the previous experience or background of the participants.

It needs to be pointed out that some additional improvements were suggested by participants, but they could not be implemented for various reasons. Those proposals are listed below:

In the question “Do you need to save or register some of this kind of data?”, it was reported that more than one answer was possible. It was not feasible to give this choice because Microsoft Forms® tools does not give the option of programming different branches depending on the answers with different combinations. With the aim of avoiding any misunderstanding, some instructions were added to the question, as has been previously explained.
Some examples of medical devices were requested to make it easier to understand the objective of the product and to help decide whether it was a medical device or not. Looking for correct examples for every case is a complex task, so this suggestion was postponed to be potentially added in the future if requested by more users.

In this manuscript, a tool called Ethool intended to facilitate decision-making on the European ethical and legal issues in research involving humans has been presented. Ethool also carries out the function of collecting information from the experiments in the case that a report is needed.

Moreover, the usability validation of the developed tool, in which 11 people participated in two iterative trials, has been explained. These volunteers reported subjective opinions regarding the user experience and ease of use of the tool. Participant feedback was objectively and subjectively processed according to the information gathered. Between each trial various improvements in the tool were implemented which had some impact on the participants evaluation. The SUS questionnaire was used to evaluate the usability of Ethool as other researchers do in similar usability evaluations [35]. Ethool was highly rated with an improvement of 5.5 points in the SUS questionnaire between each trial, with a relative gain of 6%.

In the DIH-HERO European project, in which the authors are participating, several semi-structured interviews were carried out about awareness and the use of standards in robotic development, including any ethical and legal issues that might apply. One of the conclusions from those interviews was that a tool like Ethool could be useful for small and medium-sized enterprises (SMEs) or researchers, with the aim of summarizing the applicable best practices, standards, and legislation. The inclusion of the information regarding specific details from other European countries (ethical committees contact points, i.e.) and the translation of Ethool into English were carried out in the context of this European project. Ethool was presented in various events organized by the DIH-HERO consortium. The objective is for the tool to be used by SMEs or researchers interested in carrying out experiments with technological developments involving humans. For this reason, Ethool has been made available online through the DIH-HERO services portal [36]. Authors expect that wider use of the tool will generate additional feedback on its usefulness or usability, which would be extremely valuable with regard to improving Ethool in the mid-term. At the same time, the Spanish version of Ethool is available only for researchers working at the same centres as the authors. With this approach in which the tool could be tested with real projects the authors expect to receive more feedback from real users about needs or possible improvements. The additional improvements suggested by participants but not implemented during this evaluation, will be also considered in the future if more users agree on them. Information from additional countries will also be added in the mid-term thanks to the participation of DIH-HERO partners.

The authors expect that Ethool will be able to help SMEs and researchers, mainly those from the field of engineering, to take the appropriate decisions in the future.

Financial interests. No potential competing interest was reported by the authors.

Author contribution (CRediT): Conceptualization: Ainara Garzo; Methodology: Ainara Garzo; Formal analysis and investigation: Ainara Garzo; Writing – Original Draft: Ainara Garzo; Writing – Review & Editing: Ainara Garzo, Nestor Garay-Vitoria; Visualization: Ainara Garzo, Nestor Garay-Vitoria; Project administration: Ainara Garzo; Conceptualization: Nestor Garay-Vitoria; Supervision: Nestor Garay-Vitoria.

Acknowledgements. The authors would like to thank all the volunteers who gave up their free time to participate in the usability evaluation. Special thanks are due to Digital Innovation Hubs in HEalthcare Robotics (DIH-HERO) European project partners for their help on the documentation interpretation for each country and the volunteers who participated in the interviews within this project.

Funding. This work was supported by the Ministry of Economy and Competitiveness of the Spanish Government and by the European Regional Development Fund under Grant TIN2017-85409-P; Department of Education, Universities and Research of the Basque Government under Grant IT1437-22; European Union’s Horizon 2020 research and innovation programme under Grant 825003 (DIH-HERO project).

Ethical approval. Presented work was approved on 23 May 2019 under protocol number M10 2019 128 (document 113/2019) by the Ethical Committee of the University of the Basque Country University for Research involving Human Beings (CEISH-UPV/EHU) which has number BOPV 32, 17/2/2014

Data availability. Data set associated can be found in IEEE DataPort public data repository, DOI: 10.21227/m8yv-0133 [37]

International Organization for Standardization: ISO 9241 – 210: 2010 Ergonomics of human-system interaction – Part 210: Human-centred design for interactive systems. (2010)
Marcos Muñoz, M.J., Marijuan, M.: Investigación con personas fuera del ámbito sanitario: estudios de comportamiento, psicología, ciencias del deporte y otros. ¿Cómo la evaluamos en los CEI? In: Arribas, M.C.M. (ed.) VI Congreso ANCEI. pp. 39–41. Asociación Nacional de Comités de Ética de la Investigación (ANCEI), Tarragona (2019)
Garzo, A., Garay-Vitoria, N.: Ethical issues for user involvement in technological research projects: Directives and recommendations. In: Sethy, S.S. (ed.) Contemporary Ethical Issues in Engineering. pp. 251–269. IGI Global (2015)
Harvard University. Committee on the Use of Human Subjects: Required Ethics Training Overview, https://cuhs.harvard.edu/required-ethics-training, (2020)
UC Berkeley: CITI Training: Required Human Subjects Research Training for UC Berkeley Faculty, Staff, and Students, https://cphs.berkeley.edu/training.html, (2020)
University of Colorado Boulder: CITI Training, https://www.colorado.edu/researchinnovation/irb/getting-started/citi-training
Eberhard Karls Universität Tübingen: Certificate “Ethics in practice,” https://uni-tuebingen.de/en/facilities/central-institutions/international-center-for-ethics-in-the-sciences-and-humanities/teaching/ethics-in-practice/, (2020)
Garzo, A., Garay-Vitoria, N.: Ethical and legal implications for technological devices in clinical research in Europe: Flowchart design for ethical and legal decisions in clinical research. ACM International Conference Proceeding Series. (2021). https://doi.org/10.1145/3471391.3471403
World Medical Association: WMA Declaration of Helsinki - Ethical Principles for Scientific Requirements and Research Protocols, https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/, (2013)
European Parliament and the Council: Regulation No 536/2014 of the European Parliament and of the Council of 16 April 2014 on clinical trials on medicinal products for human use, and repealing Directive 2001/20/EC., Strasbourg (2014)
Council of Europe: Convention for the Protection of Human Rights and Dignity of the Human Being with regard to the Application of Biology and Medicine: Convention on Human Rights and Biomedicine. (1997)
European Council: Directive 93/42/EEC of 14 June 1993 concerning medical devices., Luxembourg (1993)
European Parliament and the Council: Regulation (EU) 2017/745 of the European parliament and of the council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EE., Strasbourg (2017)
European Parliament and the Council: Regulation 2020/561 of the European Parliament and of the Council of 23 April 2020 amending Regulation (EU) 2017/745 on medical devices, as regards the dates of application of certain of its provisions., Brussels (2020)
European Commission. Medical Devices - Sector: Contact Points of National Authorities, https://ec.europa.eu/health/sites/health/files/md_sector/docs/md_contact_points_of_national_authorities.pdf
International Organization for Standardization: ISO 14155:2011. Clinical investigation of medical devices for human subjects — Good clinical practice. (2011)
International Conference on Harmonisation (ICH): ICH Harmonised Tripartite Guideline for Good Clinical Practice E6(R1). (1996)
Florencia Santi, M.: Vulnerabilidad y ética de la investigación social: perspectivas actuales. Rev Latinoam Bioet. 15, 52 (2015). https://doi.org/10.18359/rlbi.535
EudraCT Public website - Home page, https://eudract.ema.europa.eu/
Federal Office of Public health (FOPH) and Federal Department of Home Affairs (FDHA).: Swiss National Clinical Trials Portal (SNCTP Portal), https://www.kofam.ch/en/snctp-portal/
U.S. National Library of Medicine: ClinicalTrials.gov, https://clinicaltrials.gov/, (2020)
World Health Organization: International Clinical Trials Registry Platform (ICRTRP), https://www.who.int/clinical-trials-registry-platform, (2020)
Microsoft 365: Microsoft Forms, https://forms.office.com/, (2020)
European Parliament and the Council: Regulation (EU) 2016/679 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data (General Data Protection Regulation – GDPR). (2016)
Nielsen, J.: Usability 101: Introduction to Usability, https://www.nngroup.com/articles/usability-101-introduction-to-usability/, (2012)
Floría Cortés, A.: Algunos cuestionarios conocidos, http://www.sidar.org/recur/desdi/traduc/es/visitable/nuevos/CuestCon.htm, (2000)
Sauro, J.: Measuring Usability with the System Usability Scale (SUS), https://measuringu.com/sus/, (2011)
Nielsen, J., Landauer, J.: A mathematical model of finding the usability problem. Proceedings of the CHI 93 proceedings of the Interact conference on human factors in computing systems. In: Proceedings of ACM INTERCHI’93 Conference. pp. 206–213., Amsterdam (1993)
Nielsen, J.: Why You Only Need to Test with 5 Users, https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/, (2000)
Euskal Herriko Unibertsitatea: CEID: Comisión de Ética, https://www.ehu.eus/es/web/ceid/ceish/comite
Ley 14/2007, de 3 de julio, de Investigación biomédica., Madrid (2007)
Brooke, J.: SUS: a retrospective. J Usability Stud. 8, 29–40 (2013)
Garzo, A.: sus_process, https://github.com/agarzo/sus_process, (2020)
Bangor, A., Kortum, P., Miller, J.: Determining what individual SUS scores mean: adding an adjective rating scale. Jurnal of Usability Studies. 4, 114–123 (2009)
Pedroli, E., Cipresso, P., Greci, L., Arlati, S., Mahroo, A., Mancuso, V., Rossi, M., Stefanelli, L., Goulene, K., Sacco, M., Stramba-badiale, M., Riva, G., Gaggioli, A.: A new application for the motor rehabilitation at home : structure and usability of Bal-App. 1–12 (2020). https://doi.org/10.1109/TETC.2020.3037962
DIH-HERO: Our Services, https://dih-hero.eu/find-partner/
Garzo, A., Garay-Vitoria, N.: Ethool usability evaluation. IEEE Dataport. (2021). https://doi.org/10.21227/m8yv-0133

Table 1 is available in the Supplementary Files section.

No competing interests reported.

Table1.docx

Download PDF

Journal Publication

published 27 Nov, 2023

Read the published version in Universal Access in the Information Society →

Editorial decision: Major revision
11 May, 2023
Reviews received at journal
20 Feb, 2023
Reviewers agreed at journal
10 Feb, 2023
Reviewers invited by journal
09 Feb, 2023
Editor assigned by journal
29 Dec, 2022
Submission checks completed at journal
23 Dec, 2022
First submitted to journal
13 Dec, 2022

You are reading this latest preprint version

Ethool, a tool to enable ethical and legal decision-making in research involving humans using new technologies in Europe: design and usability validation

Status:

Journal Publication

Version 2

Abstract

Figures

1. Introduction

2. State Of The Art

2.1. Conclusions Of The Legislation Review

3. Ethool Design And Development

4. Ethool Usability Evaluation

4.1. Participants

4.2. Methodology

5. Results

6. Discussion

7. Conclusions And Future Work

Declarations

References

Tables

Additional Declarations

Supplementary Files

Status:

Journal Publication

Version 2