Comparison Usability Evaluation Methods in a Health Information System: Heuristic Evaluation versus Cognitive Walkthrough Method

doi:10.21203/rs.3.rs-871961/v1

Download PDF

Research Article

Comparison Usability Evaluation Methods in a Health Information System: Heuristic Evaluation versus Cognitive Walkthrough Method

https://doi.org/10.21203/rs.3.rs-871961/v1

This work is licensed under a CC BY 4.0 License

You are reading this latest preprint version

Background

There are some inconsistencies regarding the selection of the most appropriate usability evaluation method. The present study aimed to compare two expert-based evaluation methods in a nursing module, as the most widely used module of a Hospital Information System (HIS).

Methods

The Heuristic Evaluation (HE) and Cognitive Walkthrough (CW) methods were used by five independent evaluators to evaluate the nursing module of Shafa HIS. In this regard, the number, severity and ratio of the recognized problems according to the usability attributes were compared using two evaluation methods.

Results

The use of the HE and CW evaluation methods resulted in the identification of 104 and 24 unique problems, respectively. The average severity of the recognized problems was 2.32 in the HE method and 2.77 in the CW evaluation method; however, there was a significant difference between the number and severity of recognized usability problems by these methods (P < 0.001). Some problems, which were associated with effectiveness, satisfaction and error, were better recognized by the HE method; however, CW evaluation method was more successful in recognizing problems of learnability, efficiency and memorability.

Conclusion

The HE method recognized more problems with a lower average severity. On the other hand, CW could recognize fewer problems with a higher average severity. Regarding the evaluation goal, HE method would be used to improve effectiveness, increase satisfaction and decrease the number of errors. Furthermore, CW evaluation method is recommended to be used to improve the learnability, efficiency and memorability of the system.

Health Policy

Information Retrieval and Management

Health Information System

Hospital Information System

Heuristic Evaluation

Cognitive Walkthrough

User-Computer Interface

In recent years, healthcare organizations in many countries have largely invested on the development, acquisition, and implementation of various information technology systems [1, 2]. Moreover, the usability of health information technology systems has always been an area of concern for such organizations [1]. The well-designed health information systems can support the flow of clinical information in several ways, thereby improving patient care [3]. The nursing module of Hospital Information System (HIS) can be regarded as one of the most crucial and widely used healthcare information systems [4], which needs to be integrated into nurses’ daily practices [5]. In this regard, the efficient and effective use of this system heavily depends on its proper design and the fulfillment of the users’ expectations as such a poor design and improper usage can trigger several problems in accepting and using systems [6–8]. The problems would lead to user dissatisfaction [9, 10], increased rate of error, and decreased safety and quality of patient care services [11]. Accordingly, various usability attributes should be concerned in the development of interactive health information systems such as the nursing module [8]. According to International Organization for Standardization (ISO), usability attributes encompass efficiency, effectiveness and satisfaction [12, 13], while Nielsen considers learnability, efficiency, memorability, error and satisfaction as usability attributes of the information systems [13, 14].

The usability evaluation of information system user interfaces can be performed by several methods [15], which are divided into two expert-based and user-based groups, with the former being more preferred since this method can identify more frequent and more unique problems of higher severity[16]. In this regard, the Heuristic Evaluation (HE) and Cognitive Walkthrough (CW) are two common expert-based evaluation methods, which can recognize the system usability problems effortlessly, quickly and economically [17–19]. In HE method, the system user interface is evaluated with regard to a set of predefined principles, known as heuristics [20]. This method, performed by three to five evaluators, can recognize 75–80% of the usability problems [21]. CW evaluation method, as a task-specific and structured evaluation method, adopts the principles of cognitive psychology to simulate cognitive processes and user actions to perform specific tasks using a computer system [22, 23].

Although there are many usability problems in the information systems, most of the problems can be recognized and modified by proper planning and selecting an appropriate evaluation method [24]. In other words, an advance in the usability of the health information systems requires a deep understanding of various usability evaluation methods and relevant knowledge of how to implement the methods [1]. However, there might be several challenges and inconsistencies in selecting the most appropriate usability evaluation method and implementing the findings obtained from each method [24]. In this case, if there are a large number of evaluation methods available to be applied in a study, the findings of the previous studies, particularly comparative studies, should be considered to select the most appropriate method as such they would be highly beneficial in describing the advantages and disadvantages of each method and predicting what is to be achieved by applying each method. Consequently, the most appropriate evaluation method can be selected based on the evaluation goal.

Numerous studies have compared the findings of the HE and CW evaluation methods or the other usability evaluation methods [25–31]. In a study by Khajouei et al.[26], the HE and CW evaluation methods were compared not only according to the number and severity of recognized problems but also according to a blend of ISO and Nielsen usability attributes. Khajouei et al. study [29], as the only research in this regard, was carried out on the clinical module of a medical office management system. Regarding the limitations of their concerned system, it was specifically designed for physicians and had a limited number of installations across the country, compared to the other health information systems. Accordingly, it seems necessary to perform similar studies to compare the abovementioned evaluation methods on large-scale health information systems with more capabilities, which are expected to have a more appropriate design due to a larger number of their installations and users across the country. This study aimed to compare the number, severity and type of the recognized problems according to ISO and Nielsen’s usability attributes using the HE and CW evaluation methods in a nursing module of one of the largest HISs installed in many hospitals across the country.

System

A comparison between the HE and CW evaluation methods was performed in the nursing module of Shafa HIS (developed by Tirajeh Rayaneh Co.). The system is installed in more than 200 hospitals across the country. It possesses various capabilities as follows: admission of patient and allocation of bed and room for the patient in the inpatient ward, registration of requests for paraclinical services and monitoring the results, registration of the request for a surgery, registration a request to transfer of the patient to another ward, and registration of the patient's information at the discharge time. This system is fully integrated into the daily routines of the nurses and secretaries of inpatient wards. The nursing module was selected for this comparative evaluation since it encompasses the largest group of users and is the most critical clinical module in any HIS.

Evaluation Methods

In this study, the usability evaluation of the nursing module of Shafa HIS was carry out in the Laboratory of Health Information Technology at Kashan University of Medical Sciences using the HE and CW evaluation methods. The results of these expert-based evaluation methods were then compared.

Heuristic Evaluation

This method was first introduced by Nielsen and Molich [32]. Based on the Nielsen approach, a summarized list of heuristic principles is provided to evaluators as a guideline and each evaluator independently examines the user interface and recognizes usability problems [33]. Nielsen developed ten heuristics principles, including visibility of system status; match between system and the real world; User control and freedom; consistency and standards; help users recognize, diagnose, and recover from errors; error prevention; recognition rather than recall; flexibility and efficiency of use; aesthetic and minimalist design; and help and documentation that must be observed in the user interface design [21]. According to this method, evaluators should not communicate with one other before the evaluation is completed since a single evaluator may miss recognizing a large number of problems, while different evaluators can recognize a wide range of unique problems. Therefore, more comprehensive results can be obtained after combining the findings of several HE evaluations [33]. If time and resources are limited, this method can rapidly and economically identify the usability problems with the participation of three to five evaluators [21, 34, 35].

Cognitive Walkthrough Evaluation

The CW is one of the most common expert-based methods that emphasizes the ease of learning the system [22, 36]. This method is especially suitable where users have to master a new application or function by learning through exploration. To carry out CW, a precise description of the system user interface design, tasks scenario, clear assumptions about the users and the scope of use, as well as the series of actions that users take to successfully accomplish a given task are determined. Following this, a number of cognitive processes which are followed by users during the performance of a series of actions are simulated by an evaluator or group of evaluators in order to accomplish specific tasks. During the CW, the evaluators try to identify actions which seem to be difficult for ordinary users by understanding the interface behavior and its effects on the user. Therefore, this evaluation method could be done at the early phase of system development to meet user needs [36].

Participants

Since a number of (three to five) evaluators are adequate to perform HE and CW evaluations [29], the maximum number of evaluators (five evaluators) were chosen in a random and purposeful fashion so as to participate in this study. Three evaluators were PhD students in health information management and two evaluators had a M.Sc. in health information technology. Furthermore, evaluators had experience in the field of HE and CW evaluations and were familiar with various healthcare information systems [13].

Data Collection

In this study, the CW evaluation was initially performed by the evaluators in order to prevent the adverse effect of system learning on the CW evaluation results. Moreover, Nielsen's usability principles were provided to the evaluators to perform the HE method and they were asked to evaluate the user interface within the same range of evaluated tasks in the CW method in accordance with the checklist. Therefore, performing the CW evaluation did not influence the HE results. On the other hand, in the study by Khajouei et al [29], which was conducted in two rounds to prevent the possible effect of system learning on the results obtained from the HE and CW evaluation methods, no significant difference was found between the number of recognized problems in the first and second rounds.

A method suggested by Polson and Lewis was used to perform the CW evaluation [22]. Five scenarios were identified according to daily routine tasks performed by nurses and secretaries of inpatient wards using the nursing module based on nurses' opinions and the approval of the head nurse. For each scenario, users’ goals and sub-goals, the series of actions for each task and the system response were defined. Table 1 shows an example of a scenario and its tasks and actions. Then, the evaluators independently accomplished the series of actions for each task and, by assuming themselves as real users, they expressed any potential problems to the researcher. The comments, questions, and ambiguities raised by the evaluators’ as well as the problem and its location were recorded by the researcher, who was also considered as the observer. At the end of the evaluation process, the evaluators reviewed their problems list either added a comment or corrected a previously given comment if necessary. Then, in a meeting with the researcher and evaluators, all lists were compared and duplicate problems were eliminated and a list of individual usability problems was prepared. At the end of the meeting, this list was provided to the evaluators, who independently determined the severity of each problem on a scale of 0 to 4 according to the frequency of the problem, its impact on users, and its persistence [13, 37]. Problem severity was graded as follows:

0 = No problem- I don’t agree that this is a usability problem at all.

1 = Cosmetics - No need for correction unless more time is available for the project.

2 = Minor- Correcting this problem is a low priority.

3 = Major- Important to correction, therefore, it is a high priority.

4 = Catastrophe- Correcting this problem is necessary before the product can be released [13, 37, 38].

The recognized usability problems were categorized according to ISO and Nielsen usability attributes [12, 14]. The usability attributes of both the ISO and Nielsen is shown in Table 2 [14, 29]. Thus, evaluators independently assigned the recognized problems to one of the usability attributes.

Table 1
An example of a scenario and the related tasks and actions.
Scenario	Task	Actions
Recording a surgery request for femoral fracture reduction for a patient in the orthopedic ward	Determining a Patient	Click on inpatient ward name (orthopedic ward).
		Icons and patients' names are displayed.
		Right-click on the particular patient’s icon.
		The drop-down menu is displayed next to the patient’s icon.
	Entering a request for a surgical procedure	Choose “send to operating room waiting list” from the drop-down menu.
		A list of available operating rooms is displayed.
		Click on “general operating room”.
		The general operating room window is displayed.
		Patient’s surgery information is added.
	Submitting a surgery request to the operating room	Click on the Save button.
		A message titled “Information was saved successfully” is displayed.
		Click on the Confirm button.
		“Patients in the surgery waiting list” is displayed.
		Click on the Return button.
		Inpatient ward window is displayed.

Table 2
The usability attributes according to the ISO and Nielsen [14, 29]
Attribute	Definition
Effectiveness	How well do the users reach the goals set with the usage of the system?
Efficiency	How much of each resource (e.g. time and mental effort) is required so that the goals can be obtained by users?
Satisfaction	How pleasant is the use of the system for users?
Learnability	How easy is it for users to do basic tasks when using the system for the first time?
Memorability	When the system has not been in use for a while, How easy can users remember how to use it?
Errors	When using the system, how many errors are made, how severe are these errors and how easily can they be retrieved?

After a period of two weeks, once the CW evaluation was completed, five evaluators were asked to independently evaluate the user interface of the nursing module using the Nielsen’s usability Principles Checklist, which was based on Xerox heuristics Checklist [39]. This checklist was according to Nielsen’s ten usability principles and included 254 items with multiple-choice questions that contained “yes”, “no”, and “not applicable” as the available answers. The validity and reliability of the checklist were previously confirmed in a study by Rezaei-Hachesu et al.[40]. The recognized problems were then listed in the problem report form, which consisted of a four-column table that contained “problem title”, “problem description”, “location of the problem”, and “violated usability principle” as the head of columns.

Subsequently, the recognized problems were examined in a meeting with the five evaluators, where the repeated problems were discarded and a comprehensive list of unique problems was developed. Moreover, disagreements about the recognized problems were resolved by discussion in this meeting. Finally, similar to the CW evaluation, the evaluators independently determined the severity of each problem on a scale of 0 to 4 and assign the recognized problems to a specific usability attribute.

Data analysis

The data were analyzed using SPSS Statistics for Windows, version 20.0 (SPSS Inc., Chicago, Ill., USA) both descriptive and inferential statistical techniques. The average severity of the usability problems was calculated [13]. Then, each usability problem was assigned into one of the five categories shown in Table 3, according to its average severity [13, 41]. Furthermore, each problem was associated with one of the usability attributes in which it had the most frequency. In case of same frequency in more than one attribute, the problem was associated with the most relevant usability attribute according to the evaluators' opinion [13]. The Chi-squared test was used to compare the number of recognized usability problems by the two methods and to cover the usability attributes in each of them. Moreover, the Chi-squared test was used to compare the ratio of usability attributes in the two evaluation methods. The average severity of the recognized problems by the two methods was compared using the Mann-Whitney U test. A significance level of 0.05 was considered in this study.

Table 3
Problem categories according to their average severity [13, 41]
Problem category	Average severity
No problem	0-0.5
Cosmetic	0.6–1.5
Minor	1.6–2.5
Major	2.6–3.5
Catastrophic	3.6-4

In the present study, the usability evaluation results obtained by the HE and CW methods in the nursing module of a HIS were compared. The HE method was able to identify 317 problems, of which 104 unique problems remained after merging the problems and discarding the repeated ones. On the other hand, 46 problems were recognized in the CW evaluation, of which 24 unique problems remained after merging the problems and discarding the repeated ones; the Chi-squared test indicated a significant difference between the number of recognized problems in the two evaluation methods (P < 0.001).

In this study, 57 problems (54.8%) were minor problems and 33 of them (31.7%) were major problems out of the unique problems recognized in the HE method. On the other hand, 12 problems (50%) were major problems and 10 problems (41.7%) were minor problems out of the unique problems recognized in the CW evaluation. The average severity of usability problems was 2.32 and 2.77 in the HE and CW methods, respectively, and the Mann-Whitney U test revealed a significant difference based on these figures (Table 4).

Table 4
Number and severity of the recognized problems by the two evaluation methods
Methods	Number of problems		Number of unique problems based on severity				Mean ± SD	Mann-Whitney test
Methods	Total	Unique	Cosmetic N(%)	Minor N(%)	Major N(%)	Catastrophe N(%)	Mean ± SD	Mann-Whitney test
HE	317	104	8(7.7)	57(54.8)	33(31.7)	6(5.8)	2.32 ± 0.65	Z=–3.037 P = 0.001
CW	46	24	-	10(41.7)	12(50)	2(8.3)	2.77 ± 0.51	Z=–3.037 P = 0.001

Based on the usability attributes, the number and average severity of the recognized problems are shown in Fig. 1. A significant difference was seen between the number of recognized problems according to the usability attributes in the HE evaluation (P < 0.001). In this method, the most number of recognized problems was associated with effectiveness (35 problems) and the most average severity was associated with effectiveness and error (2.5). No significant difference was found between the number of recognized problems according to the usability attributes in the CW evaluation (P = 0.156). In this method, the most number of recognized problems was associated with learnability (8 problems) and the most average severity was associated with learnability and error with an average severity of 3.1 and 3, respectively.

As shown in Table 5, a significant difference was observed between the HE and CW methods regarding the ratio of the recognized problems according to the usability attributes. Thus, in the investigated nursing module, a number of problems associated with effectiveness, satisfaction and error were better recognized in the HE method while problems related to learnability, efficiency and memorability were better recognized in the CW evaluation.

Table 5
Comparison of the two methods in recognizing problems according to usability attributes
Usability problems Usability attributes	HE N(%)	CW N(%)	Chi- Square tests
Effectiveness	13(12.5)	6(25)	P = 0.012
Efficiency	35(33.7)	2(8.3)
Satisfaction	22(21.2)	2(8.3)
Learnability	18(17.3)	8(33.3)
Memorability	5(4.8)	4(16.7)
Errors	11(10.6)	2(8.3)
Total	104(100)	24(100)

The usability evaluation of the nursing module of Shafa HIS was performed by the HE and CW evaluation methods. The most number of problems was recognized by the HE method as compared to the CW method, although the average severity of recognized usability problems by the HE was less than the CW method. Concerning the usability attributes, the most number of recognized problems by the HE method was associated with effectiveness, while the most number of recognized problems by the CW evaluation was associated with system learnability.

A significant difference was indicated between the total number of usability problems recognized by the HE and CW methods in the usability evaluation of the nursing module. In addition, a higher number of problems were recognized by the HE method compared to that of the CW evaluation. In the studies by Jeffries et al. [27] and Frøkjær and Lárusdóttir [26], it was also indicated that the HE method can recognize more usability problems compared to the CW evaluation, which is compatible with the results of the present study. It should be noted that the number of recognized problems in the HE method depends on the expertise and skill of the evaluators [27]. In a study by Desurvire et al. [42], it was revealed that expert evaluators were able to recognize a higher number of problems in the HE method compared to the CW evaluation, while the number of recognized problems by other evaluators was equal in both methods. Other studies have also shown that the number of recognized problems in the HE method by experienced evaluators is higher than that of inexperienced evaluators [43–45]. In another study performed by Khajouei et al.[29] it was indicated that despite identifying a higher number of usability problems in the HE method compared to the CW evaluation, no significant difference was seen between the numbers of recognized problems in these two evaluation methods, which was incompatible with the findings of the present study. This may be due to the fact that Nielsen’s usability principles checklist was used in the present study, while in the study conducted by Khajouei et al. [29], no usability checklist was used in order to carry out the HE method. As a way to confirm this notion, another study by Khajouei et al. [35] showed that using a checklist in performing the HE method can lead to the identification of a higher number of usability problems compared to performing this evaluation without a checklist.

Based on the obtained results, more than half of the recognized problems by the HE method were in the minor problem category, while 50 percent of problems recognized by the CW evaluation were placed in the major problem category. In a study by Jeffries et al. [27], four usability evaluation methods were compared and it was revealed that two-thirds of low-severity problems were recognized by the HE method. Other studies indicated that most of the recognized problems by the HE method are in the minor or cosmetic categories [15, 44, 46], which is compatible with the findings of the present study. Contrary to these findings, other studies reported that most of the recognized problems by the HE method are in the major and catastrophe categories [47, 48], the different types of recognized problems due to investigating different health information systems can be considered as a source of this inconsistency. The skill and expertise of evaluators, which is highly effective in estimating the severity of recognized problems, can be another reason for this inconsistency in findings. As the study by Atashi et al. [43] indicated that the average severity assigned by experienced evaluators to the problems in the HE method was significantly higher than that of the other evaluators.

Based on the obtained results, the average severity of usability problems in the CW evaluation was significantly higher than that of the HE method, which is consistent with the findings of the study by Khajouei et al. [29]. The study by Cuomo and Bowen [25] proposed that the CW evaluation is the superior method when identifying certain problems that either affect user performance or focus on how users interact with the user interface while performing specific tasks [49]. On the other hand, the HE method primarily focuses on identifying problems that are associated with the system user interface design and its compliance with a set of predefined usability principles [18]. Therefore, as a task-independent method, the HE method illustrates a wide range of various aspects of a system user interface and identifies specific problems that users hold in lower regard and priority for taking corrective actions compared to the CW method.

According to the findings of this study, the most number of recognized problems by the HE method was associated with effectiveness and secondly, satisfaction. Moreover, the ratio of usability attributes was compared for both methods and it was indicated that the HE method is superior in identifying a number of problems associated with effectiveness, satisfaction and error. The study by Khajouei et al. [29] also reported that the HE method is significantly superior in identifying problems associated with satisfaction, which is consistent with the results of the present study, however, in the present study, satisfaction was at the second place, following effectiveness in terms of the usability attributes having the most number of recognized problems by the HE method. As a general rule, effectiveness can be achieved when users are able to perform their activities and tasks accurately and completely via the system [50, 51]. As a result, correcting the recognized usability problems by the HE method leads to a significant increase in the effectiveness and satisfaction of users, provided that a system is well-designed with minimal complexity.

According to the findings of this study, the most number of recognized problems by the CW evaluation was associated with system learnability. Furthermore, the findings of the present study are similar to that of the study by Khajouei et al. [29] in terms of the largest number of problems associated with learnability when performing the CW evaluation. Moreover, in a study by Jaspers [18], it was stated that the CW evaluation, as a usability evaluation method, focuses on evaluating a system design in terms of learnability by exploration. The learnability refers to the speed at which new or novice users learn a system user interface in order to perform various tasks [52]. In the present study, the second largest number of problems in the CW evaluation belonged to efficiency. Moreover, the ratio of usability attributes was compared for both methods and it was indicated that the CW evaluation is superior in recognizing problems related to learnability, efficiency and memorability. This finding was not in line with the findings of Khajouei et al. study [29], that showed that efficiency problems were better recognized by the HE method. The inconsistency might be caused by the differences in the problems recognized by the systems evaluated in these two studies. Efficiency refers to the users’ time and mental effort and the costs involved in achieving goals so that the users can achieve their goals with minimum resources and costs [29, 53]. Since the amount of training time for health information systems is highly limited for the users such as nurses, who are essentially required to learn how to work with the system while doing their duties [54], performing the CW evaluation and correcting the usability problems recognized by this method can ultimately lead to a decrease in the users’ cognitive load, a reduction in the system learning time, and an increase in the system efficiency.

Strengths and Weaknesses

This study aimed to compare two expert-based usability evaluation methods in a nursing module, as the most important and the most widely used module of a HIS. Moreover, five evaluators participated in this study, an adequate number of evaluators to perform the HE and CW evaluations. Furthermore, the evaluators had previous experiences with conducting several HE and CW evaluations and either had work experience or were familiar with various HISs and based on previous studies [42–44], the participation of skilled and experienced evaluators is beneficial in identifying the largest number and the most severe usability problems. As yet another point of strength in this study, five scenarios were identified according to the most important and frequent daily tasks of nurses performed via the nursing module. These scenarios were formed according to the opinion of three experienced nurses working in the inpatient ward, which were then approved by the head nurse. On the other hand, evaluating the nursing module of a HIS can be considered as a limitation of this study. Since designing different user interfaces for various HISs can influence the number and severity of recognized problems and consequently the number of problems attributed to each usability attribute in each of the HE and CW methods, the findings of this study cannot be fully generalized to all systems.

Based on a comparison between the HE and CW methods for the usability evaluation of a nursing module, the HE method can identify a higher number of problems compared to the CW evaluation due to its broad overview of a system design based on a set of predefined principles, while the evaluation of specific tasks in detail is avoided; though users may hold lower regard and priority for taking corrective actions for these problems. On the other hand, this study indicated that performing a reasonable number of tasks in the CW evaluation leads to identifying fewer problems with a higher average severity compared to the HE method. Finally, according to the obtained results, it was shown that the HE method is superior in identifying usability problems associated with effectiveness and satisfaction, while the CW evaluation method is superior in identifying problems related to system learnability and efficiency.

HIS

Hospital Information System

Heuristic Evaluation

Cognitive Walkthrough

ISO

International Organization for Standardization

Ethics approval and consent to participate

The present study was approved by the Research Ethics Committee of Kashan University of Medical Sciences Research Council (Number: 1397.18) and conducted following the guidelines of the Declaration of Helsinki. In accordance with the opinion of the above-mentioned Ethics Committee and given the fact that no information about participants is provided in this paper, participants who participated in usability studies gave verbal consentto participate in this research.

Consent for publication

Not applicable

Availability of data and material

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Competing interests

The authors declare that they have no competing interests.

Funding

The present research was supported by the Research Council of Kashan University of Medical Sciences [grant number: 97031] and did not receive any grants from non-profit organizations and funding agencies in the public and commercial sectors.

Authors' contributions

All authors contributed to the creation of the manuscript. MF: design, conception, acquisition of funding, interpretation of data, drafting of the manuscript and critical revision. EN: design, conception, interpretation of data, drafting of the manuscript and critical revision. HT: design, conception, data collection, statistical analysis, interpretation of data and drafting of the manuscript. MSJ: design, conception, data collection, statistical analysis, interpretation of data, drafting of the manuscript and critical revision. . All authors read and approved this manuscript.

Acknowledgments

The authors would like to express their gratitude to the esteemed Vice-Chancellor of Kashan University of Medical Sciences and the management of the Health Information Technology Laboratory at Kashan University of Medical Sciences and the evaluators who participated in usability evaluation.

Carayon P, Hoonakker P: Human factors and usability for health information technology: old and new challenges. Yearb Med Inform 2019, 28(01):071–077.
Farzandipour M, Ahmadi M, Sadoughi F: Adopting confidentiality principles for electronic health records in Iran: a Delphi study. J Med Syst 2011, 35(3):333–343.
Ammenwerth E, Ehlers F, Hirsch B, Gratl G: HIS-Monitor: an approach to assess the quality of information processing in hospitals. Int J Med Inform 2007, 76(2–3):216–225.
Farzandipour M, Meidani Z, Riazi H, Jabali MS: Nursing information systems requirements: a milestone for patient outcome and patient safety improvement. Comput Inform Nurs 2016, 34(12):601–612.
Tubaishat A: Perceived usefulness and perceived ease of use of electronic health records among nurses: Application of Technology Acceptance Model. nform Health Soc Care 2018, 43(4):379–389.
Farzandipour M, Riazi H, Jabali MS: Proposing electronic health record usability requirements based on enriched ISO 9241 metric usability model. Acta Inform Med 2018, 26(1):29.
Gholami M, Damanabi S, Hachesu PR, Ghyassi FS: Evaluation of Nursing Information Systems Using the HIS-Monitor Instrument: Nurses Perspectives. Iranian Journal of Medical Informatics 2019, 8(1):9.
Moghaddasi H, Rabiei R, Asadi F, Ostvan N: Evaluation of Nursing Information Systems: Application of Usability Aspects in the Development of Systems. Healthc Inform Res 2017, 23(2):101–108.
Chow SK, Chin WY, Lee HY, Leung HC, Tang FH: Nurses’ perceptions and attitudes towards computerisation in a private hospital. J Clin Nurs 2012, 21(11-12):1685–1696.
Farzandipour M, Meidani Z, Riazi H, Sadeqi Jabali M: Task-specific usability requirements of electronic medical records systems: Lessons learned from a national survey of end-users. Inform Health Soc Care 2018, 43(3):280–299.
Bowman S: Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag 2013, 10(Fall).
ISO 9241-11: Ergonomic reqirements for Office work with visual display terminals (VDTs)- Guidance on usability specifications and measures, 1998.
Farzandipour M, Nabovati E, Tadayon H, Sadeqi Jabali M: Usability evaluation of a nursing information system by applying cognitive walkthrough method. Int J Med Inform 2021, 152:104459.
Nielsen J: Usability 101: Introduction to Usability. Nielsen Norman Group; 2012.
Joshi A, Perin DMP, Amadi C, Trout K: Evaluating the usability of an interactive, bi-lingual, touchscreen-enabled breastfeeding educational programme: application of Nielson’s heuristics. J Innov Health Inform 2015, 22(2):265–274.
Karlsson F: Assessing usability evaluation methods for smartwatch applications. KTH Royal Institute of Technology,Stockholm. Sweden; 2016.
Bligård L-O, Osvalder A-L: Enhanced cognitive walkthrough: Development of the cognitive walkthrough method to better predict, identify, and present usability problems. Adv. Hum.-Comput. Interact 2013, (2013).
Jaspers MW: A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009, 78(5):340–353.
Qiu S, Du L, Han T, Hu J: Flavor explore: rapid prototyping and evaluation of user interfaces. In: International Conference on Distributed, Ambient, and Pervasive Interactions. Las Vegas, NV, USA: Springer; 2018: 114–123.
Nielsen J: Usability engineering. San Diego: Academic Press; 1993.
Nielsen J: Heuristic Evaluation in Usability Inspection Methods (RL Mack). New York: John Wiley & Sons, Inc; 1994.
Polson PG, Lewis C, Rieman J, Wharton C: Cognitive walkthroughs: a method for theory-based evaluation of user interfaces. Int. J. man-mach. Stud. 1992, 36(5):741–773.
Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J biomed inform. 2004,37(1):56–76.
Mendes D: The Reliability of Usability Evaluations. Bentley University; 2011.
Cuomo DL, Bowen CD: Understanding usability issues addressed by three user-system interface evaluation techniques. Interact Comput 1994, 6(1):86–108.
Frøkjær E, Lárusdóttir MK: Prediction of usability: comparing method combinations. In: 10'th international Conference of the Information Ressources Management Association. Hershey, Pa, USA; 1999: 284 – 257.
Jeffries R, Miller JR, Wharton C, Uyeda K: User interface evaluation in the real world: a comparison of four techniques. In: Proceedings of the SIGCHI conference on Human factors in computing systems:Reaching Through Technology. New York, NY, ACM Press; 1991: 119–124.
Khajouei R, Hasman A, Jaspers MW: Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system. Int J Med Inform 2011, 80(5):341–350.
Khajouei R, Zahiri Esfahani M, Jahani Y: Comparison of heuristic and cognitive walkthrough usability evaluation methods for evaluating health information systems. J Am Med Inform Assoc 2017, 24(e1):e55-e60.
Maguire M, Isherwood P: A comparison of user testing and heuristic evaluation methods for identifying website usability problems. In: Marcus A. and Wang W. (eds). Design, User Experience, and Usability: Theoty and Practice:7th international conference (DUXU 2018), Las Vegas, NV, USA, 15–20 July 2018, Part I:429–438.
Yen P-Y, Bakken S: A comparison of usability evaluation methods: heuristic evaluation versus end-user think-aloud protocol–an example from a web-based communication tool for nurse scheduling. In: AMIA annual symposium proceedings, American Medical Informatics Association; 2009: 714–718.
Nielsen J, Molich R: Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI conference on Human factors in computing systems. Empowering People, Seattle, Washington, DC, USA; 1990: 249–256.
Ahmed SZ: A comparison of usability techniques for evaluating information retrieval system interfaces. Performance Measurement and Metrics 2008, 9(1):48–58.
Brewster S, Dunlop M: Mobile Human-Computer Interaction-Mobile HCI 2004: 6th International Symposium, Glasgow, UK; 2004, 3160.
Khajouei R, Gohari SH, Mirzaee M: Comparison of two heuristic evaluation methods for evaluating the usability of health information systems. J biomed inform 2018, 80:37–42.
Zaini NA, Noor SF, Wook TS. Evaluation of api interface design by applying cognitive walkthrough. International Journal of Advanced Computer Science and Applications 2019,10(2):306–315.
Nielsen J: Severity ratings for usability problems. Papers and Essays 1995, 54:1–2.
Okhovati M, Karami F, Khajouei R. Exploring the usability of the central library websites of medical sciences universities. Journal of Librarianship and Information Science 2017,49(3):246–55.
Pierotti D: Heuristic evaluation-a system checklist. Xerox Corporation, Society for Technical Communication 1995, 12.
Rezaei-Hachesu P, Pesianian E, Mohammadian M: Evaluating Usability of Radiology Information Systems in Hospitals of Tabriz University of Medical Sciences. Acta Inform Med 2016, 24(1):42.
Ahmadian L, Salehi F, Abedinzadeh A, Khatibi F: Usability evaluation of a Radiology Information System. Journal of Health Administration (JHA) 2017, 20(69):67–78.
Desurvire H, Kondziela J, Atwood ME: What is gained and lost when using methods other than empirical testing. In: Posters and short talks of the 1992 SIGCHI conference on Human factors in computing systems. New York, NY, United States, ACM; 1992: 125–126.
Atashi A, Khajouei R, Azizi A, Dadashi A: User Interface problems of a nationwide inpatient information system: a heuristic evaluation. Appl Clin Inform 2016, 7(01):89–100.
Lilholt PH, Jensen MH, Hejlesen OK: Heuristic evaluation of a telehealth system from the Danish TeleCare North Trial. Int J Med Inform 2015, 84(5):319–326.
Nielsen J: Finding usability problems through heuristic evaluation. In: Proceedings of the SIGCHI conference on Human factors in computing systems. onterey, California, United States, ACM; 1992: 373–380.
Ebnehoseini Z, Tara M, Meraji M, Deldar K, Khoshronezhad F, Khoshronezhad S: Usability Evaluation of an Admission, Discharge, and Transfer Information System: A Heuristic Evaluation. Open Access Maced J Med Sci 2018, 6(11):1941.
Nabovati E, Vakili-Arki H, Eslami S, Khajouei R: Usability evaluation of Laboratory and Radiology Information Systems integrated into a hospital information system. J Med Syst 2014, 38(4):35.
Sadeqi Jabali M, Nabovati E, Farzandipour M, Farrahi SR, Abbasi R: Evaluation of the Usability of Admission and Medical Record Information System: A Heuristic Evaluation. Journal of Health and Biomedical Informatics 2019, 6(2):80–89.
Lewis C, Wharton C: Cognitive walkthroughs. In: Handbook of human-computer interaction. Second Completely Revised Edition, Amsterdam: Elsevier Science BV; 1997: 717–732.
Abu Raddaha AH: Nurses’ perceptions about and confidence in using an electronic medical record system. Proceedings of Singapore Healthcare 2018, 27(2):110–117.
Farrahi R, Jeddi FR, Nabovati E, Sadeqi Jabali M, Khajouei R: The relationship between user interface problems of an admission, discharge and transfer module and usability features: a usability testing method. BMC Med Inform Decis Mak 2019, 19(1):172.
Johnson C, Johnston D, Crowle P: EHR usability toolkit: A background report on usability and electronic health records. Rockville, MD: Agency for Healthcare Research and Quality 2011.
Meidani Z, Farzandipour M, Farrokhian A, Haghighat M: A review on laboratory tests’ utilization: A trigger for cutting costs and quality improvement in health care settings. Med J Islam Repub Iran 2016, 30:365.
Berge M, Fossum M, Fruhling A: A Cognitive Walkthrough and Focus Group Study of Nursing Personnel to Improve EHRs Used in Nursing Homes. In: Scandinavian Conference on Health Informatics. Linköping, Sverige, Linköping University Electronic Press; 2012: 13–18.

No competing interests reported.

Download PDF

Editorial decision: Major revision
02 Dec, 2021
Reviews received at journal
18 Oct, 2021
Reviewers agreed at journal
15 Oct, 2021
Reviewers invited by journal
14 Oct, 2021
Editor assigned by journal
14 Oct, 2021
Editor invited by journal
05 Oct, 2021
Submission checks completed at journal
05 Oct, 2021
First submitted to journal
03 Sep, 2021

You are reading this latest preprint version

Comparison Usability Evaluation Methods in a Health Information System: Heuristic Evaluation versus Cognitive Walkthrough Method

Status:

Version 1

Abstract

Background

Methods

Results

Conclusion

Figures

Background

Methods

System

Evaluation Methods

Heuristic Evaluation

Cognitive Walkthrough Evaluation

Participants

Data Collection

Data analysis

Results

Discussion

Strengths and Weaknesses

Conclusion

Abbreviations

Declarations

References

Additional Declarations

Status:

Version 1