The Development of Heuristics for Evaluation of Dashboard Visualizations

Keywords

requirements analysis and design - clinical decision support - home health - dashboard - methodologies

Background and Significance

Heuristic evaluation is a common method used in human–computer interaction studies to assess the usability of information systems. The approach is an inspection method of evaluation; where experts or evaluators use defined criteria (heuristics) to evaluate the interface and provide feedback on potential problems with system usability.[1] It is frequently employed as a method of usability evaluation, due to its low cost and simplicity in approach.[2] Normally based on the principles identified by Nielsen[3] [4] with guidelines expanded by Pierotti[5] evaluators are asked to use a set of heuristic principles to identify potential usability problems, assigning a severity rating from no problem through to catastrophic.

While it is generally agreed that the original 10 heuristics identified by Nielsen are appropriate for evaluating general system usability and user interface issues, their appropriateness for evaluating the usability of systems that use visualizations have been questioned.[6] [7] [8] [9] The goal of information visualization systems are to provide “tools and techniques for gaining insight and understanding in a data set, or more generally to amplify cognition.”[10] Usability related to information visualization tools focuses on the ability of the tool to display data in a way that is understandable, and to enable the user to explore and interact with that data.[11] Elements such as visual representation of information, perceptual/cognitive issues related to that representation, and data interaction mechanisms are not specifically addressed in existing sets of more general heuristics.[9]

However, studies which have evaluated the use of various information visualization-specific heuristics have found that they miss more general usability issues,[7] suggesting that both information visualization-specific and more general usability heuristics may be most effective for conducting heuristic evaluations of systems that use visualized data displays. To conduct a heuristic evaluation of a point of care dashboard for home care nurses, which uses data visualization to summarize information, we developed and refined a tailored set of heuristics for visualized data displays.

Objective

In this article, we outline an approach taken to combine principles from Nielsen's original 10 usability heuristics[3] [5] with heuristics specifically derived to evaluate the usability of information visualization systems,[9] to aid with heuristic evaluation of a common form of technology used in health care settings; clinical dashboards. Our approach has produced a set of usability heuristics that can be used to evaluate similar systems.

Methods

Development of the Checklist

To identify a candidate set of visualization evaluation heuristics that could be used specifically for health information visualizations, we implemented a nominal group technique, using online survey methods. This consensus building method is a structured way to obtain input on developing a process or solving a problem from several people with expertise in that area. It was originally developed by sociologists to assist with community program planning.[12] [13] The process involves the generation of ideas (often initiated from a candidate set), participation and contribution of all members, and voting or ranking ideas according to importance to the problem at hand.

For this study, a list of general heuristic principles and specific usability factors derived from Nielsen's 10 usability principles,[3] and items from the checklist outlined by Pierotti[5] was developed. The seven general principles used to evaluate the dashboard (visibility of system status, match between system and the real world, user control and freedom, consistency and standards, recognition rather than recall, flexibility and efficiently of use, aesthetic and minimalist design) with examples of specific usability factors (36 in total) are provided in [Table 1]. These general principles were chosen to reflect the specific functionality of the clinical dashboard being evaluated. The three principles identified by Nielsen that were not included in the checklist (error prevention, help users recognize, diagnose and recover from errors, help and documentation) were judged irrelevant for the functionality of the dashboard we were developing.

Table 1 General usability principles and associated usability factors

Usability principle	Usability factors
1. Visibility of system status The system should always keep user informed about what is going on through appropriate feedback within reasonable time	Does every screen have a title or header that describes its contents? Is there a consistent icon design scheme and stylistic treatment across the system? Is there visual feedback in menus or dialog boxes about which choices are selectable? Is there a clear indication of the current location? Is the menu-naming terminology consistent with the users' task domain? Does the system provide visibility: that is, by looking, can the user tell the state of the system and the alternatives for action?
Match between system and the real world The system should speak the user's language, with words, phrases, and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order	Are icons concrete and familiar? Are the section headings and subsections in each screen ordered in the most logical way? Is there a natural sequence to the menu choices for a data item? Do the selected colors correspond to common expectations about color codes? Are the words/concepts and phrases used in each screen familiar to users?
User control and freedom Users should be free to select and sequence tasks (when appropriate), rather than having the system do this for them. Users will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Users should make their own decisions regarding the costs of exiting current work	Is there a clear exit on each document screen? Are all screens accessible across the system? Is there an “undo” function? Do users have the option of either clicking on menu items with a mouse or using a keyboard shortcut? Can users easily move forward and backward between screens?
Consistency and standards Users should not have to wonder whether different words, situations or actions mean the same thing	Have formatting standards been followed consistently in all screens within the system? Does each window have a title? Are there salient visual cues to identify the active screen? Are there no more than four to seven colors and are they far apart along the visible spectrum? Are names consistent, both within each tab and across the system, in grammatical style and terminology? Are menu choice lists presented vertically?
Recognition rather than recall Make objects, actions and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for the use of the system should be visible or easily retrievable whenever appropriate	Are prompts, cues, and messages placed where the eye is likely to be looking on the screen? Is white space used to create symmetry and lead the eye in the appropriate direction? Have items been grouped into logical zones, and have headings been used to distinguish between zones? Is color highlighting used to get the user's attention? Is color coding consistent throughout the system?
Flexibility and efficiency of use The system should offer users several options when it comes to finding content. Users should be able to achieve their goals in an efficient manner	Is navigation between screens simple and visible? If menu lists are short (seven items or fewer), can users select an item by moving the cursor? If the system uses a pointing device, do users have the option of either clicking on fields or using a keyboard shortcut? On menus, do users have the option of either clicking directly on a menu item or using a keyboard shortcut?
Aesthetic and minimalist design Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.	Is only (and all) information essential to decision making displayed on the screen? Have large objects, bold fonts, and simple areas been used to distinguish sections? Are field labels brief, familiar, and descriptive? Is the visual layout well designed? Are there any unnecessary data elements in each screen?

A second list of information visualization-specific heuristics were developed, based on a study conducted by Forsell and Johansson.[9] Their study describes 10 information visualization heuristics, which were derived from six previous heuristic sets.[3] [10] [11] [14] [15] [16] The source heuristic sets collected by Forsell and Johansson were all visualization specific. However, four sets were primarily focused on the perceptual and cognitive processing theories underlying the perception of visual data[10] [11] [14] [16] and two sets were more general in their focus, but included some elements of visualization design.[15] [17] Forsell and Johansson provided the participants in their study with well-known examples of usability problems in information visualization systems, and asked them to evaluate how well each heuristic explained the problem on a scale from 0 (doesn't explain the usability problem at all) through to 5 (provides a complete explanation of the problem). Through this process, they narrowed down the 6 heuristic sets into 10 heuristics that explained the highest proportion of a set of known usability problems for information visualization systems.[9] Our study utilized the 7 most salient of these 10 heuristics (spatial organization, information coding, orientation and help, data set reduction, flexibility, consistency, remove the extraneous [ink]). The definition and specific usability factors related to each of these information visualization heuristics were extracted from the original papers ([Table 2]).[11] [15] [18] The heuristics that we did not use were: recognition and recall; minimal actions; and prompting. These three heuristics and their specific usability factors were already captured by the general heuristics we identified.

Table 2 Information visualization usability principles and associated usability factors

Usability principle	Usability factors
Spatial organization [11] Relates to the overall layout of a visual representation and refers to how easy it is to locate an information element in the display and the distribution of elements in representations	Are all information elements clear and visible? Does the information follow a ‘logical’ organization? Does the information provide detail on the context and detail of the data element?
Information coding [11] Refers to the use of symbols or representations to aid perception	Are symbols appropriate for the data represented? Are realistic characteristics used to represent data or information elements?
Orientation and help [11] Provision of support for the user and help to orientate them in the visualization	Can the user control the level of detail they see in a representation? Can the user redo/undo their actions? Can the user see the path they have followed to navigate through the representation?
Data set reduction [11] The ability of the visualization to manipulate the data to reduce the amount seen at any time	Can the user filter information to adjust rapidly to the focus of interest? Can the user cluster information into a subset of data elements? Can the user prune information, cutting off information that is irrelevant to their understanding of the visual representation?
Flexibility [15] The means available to the users to customize the interface, to take account of their working strategies, and/or habits. It reflects the capacity for the interface to adapt to users' needs	Do the users have the ability to control display configurations? Can the users enter default or baseline ranges? Can the user remove or hide unnecessary displays?
Consistency [15] The way the interface design choices (codes, naming, formats, procedures) are maintained in similar contexts and different when applied to different contexts	Are window titles always located in the same place? Are screen formats similar across windows? Are similar procedures used to access options?
Remove the extraneous (ink) [18] Present the largest amount of data with the least amount of ink	Is the data presented in a simple format? Is there white space between color representations?

The two lists of usability principles and associated factors were distributed via email using established survey software (surveygizmo.com). The survey was distributed to a group of 12 individuals selected for expertise in both informatics and visualization. These individuals were recruited from an information visualization studio that meets monthly at our institution. This group consists of approximately 20 researchers and postdoctoral students. Participants were asked to rate each factor on its importance as an evaluation heuristic for visualization systems on a scale from 1 (definitely don't include) to 10 (definitely include). The distribution of scores for each item, including mean, median, and range, were calculated. A median score of ≥ 8 was determined to represent consensus for an item to be included in a final set.

Validation and Use of the Checklist

The final set of items were further refined based on text comments from the experts, removing duplicated items and grouping similar usability factors together. The items then were formatted with description of the usability principle (heuristic) followed by a checkbox list with each usability factor related to that principle. This format allowed an evaluator to assess whether or not the heuristic had been violated and give it a severity rating based on Neilsen's system, that is, from 1= cosmetic problem only through to 4 = usability catastrophe (imperative to be fixed before a product can be released).

The heuristic evaluation checklist was utilized in a heuristic evaluation of a prototype dashboard. The evaluation was conducted with three experts in the field of nursing informatics. Criteria for selecting experts were: a nurse (as nurses were the end users of the dashboard); published in the field of informatics; and experience with data visualization. Three potential participants, all of whom worked at our academic medical center, were recruited via email. All three targeted participants accepted. They were provided with an extended task list designed to enable them to explore the full functionality of the dashboard and the heuristic evaluation checklist. At a mutually agreed date and time, they used the task list to examine and navigate within the dashboard, complete the evaluation checklist, and provide written feedback on the dashboard design.

Results

Checklist Development

A total of 10 out of 12 invited experts responded to the online survey. The responses resulted in 5 usability factors with median inclusion scores less than 8, which then were eliminated from the final checklist ([Table 3]). In response to the respondent's written comments, further modifications were made to the usability checklist. The first was to harmonize usability factors associated with consistency across the two sets of heuristics into one category that incorporated both general and information visualization-specific usability factors. Second, factors associated with flexibility from the information visualization-specific list were combined with those from the general heuristics, adding the one remaining usability factor associated with data set reduction. Finally, the usability factors from “remove the extraneous (ink)” were combined with the more general heuristic category of “aesthetic and minimalist design.” Minor edits to language included changing “keyboard” to “touchscreen/stylus” (as this represents how our particular users interacted with the system) and renaming the “Orientation and Help” category to “Orientation.”

Table 3 Usability factors removed from final checklist

Usability principle	Usability factor (median score)
General usability heuristics
4. Consistency and standards	Are menu choices presented vertically? (6.5)
6. Flexibility and efficiency of use	If menu lists are short (seven items or fewer), can users select an item by moving the cursor? (7.5)
Visualization-specific heuristics
3. Orientation and help	Can the user see the path they have followed to navigate through the representation? (7)
4. Data set reduction	Can the user cluster information into a subset of data elements? (7.5)
	Can the user prune information, cutting off information that is irrelevant to their understanding of the visual representation? (7.5)

The final checklist consists of 10 usability principles (7 generated from the general principles and 3 that were information visualization specific) substantiated with 49 usability factors. The final checklist is available in the [Supplementary Material], (available in the online version). It is formatted for use as a heuristic evaluation checklist with each usability principle associated with its corresponding usability factors, as used in our study.

Validation and Use of the Checklist

Three nursing informatics experts applied the final usability evaluation checklist to review our prototype dashboard, which summarized vital sign data for home health nurses ([Figs. 1] and [2] are screenshots indicating various elements of the dashboard design). The dashboard had few usability issues in the categories of information coding, spatial organization, visibility of system status, orientation, and aesthetic and minimalist design. The main issues were related to the flexibility and efficiency of use and user control and freedom ([Table 4] and [Fig. 3]). Overall, the usability issues identified by the experts were classified as being either a minor usability issue (n = 5) or a cosmetic problem only (n = 12).

Table 4 Heuristic evaluation ratings of prototype dashboard

	Maximum score	Mean score	Result (%)
Visibility of system status	6	5.7	95
Match between system and the real world	5	4	80
User control and freedom	5	3	60
Consistency and standards	6	5.3	88
Recognition rather than recall	4	3	75
Flexibility and efficiency of use	7	4	57
Aesthetic and minimalist design/remove the extraneous (ink)	7	6	86
Spatial organization	3	2.67	89
Information coding	2	2	100
Orientation	4	3.33	83
Total	49	39	79.6

Fig. 1 Screenshot of prototype dashboard: blood pressure display.

Fig. 2 Screenshot of prototype dashboard: display of weight measurement in vital signs documentation.

Fig. 3 Heuristic evaluation of dashboard prototype.

All three experts used the usability evaluation checklist without problem. In their feedback on the utility of the checklist, they all reported that it covered every major usability problems identified during completion of the evaluation tasks.

Discussion

In this study, we used expert opinion to integrate available heuristic criteria for both general and visual information systems and formatted the integrated criteria as a rating checklist. We implemented the checklist in a heuristic evaluation of a clinical dashboard that displayed a visual summary of vital sign data for health nurses to use at the point of care. Using this approach, we have taken heuristic criteria developed from a theoretical perspective and demonstrated their use in an evaluation tool that can be used to evaluate health information systems that display visualized data.

Our study is limited by use of a small number of experts from one institution that were recruited to prioritize items for inclusion in the checklist, only 10 of the 12 experts responded to the original invitation to participate in the study and validation with just 3 heuristic evaluators. In addition, our checklist was only used with the dashboard that was the focus of our study; further validation is necessary with other visualization systems and the reliability of the checklist will also need to be evaluated. Nevertheless, the checklist we have described represents, to our knowledge, the first tool that integrates general and specific heuristic evaluation criteria for evaluating information visualization systems in health. We expect the heuristic criteria developed here will undergo revision if and when it is used with different experts to evaluate different visualization systems.

Visual representations of data enable individuals to process information more efficiently than text, and transform even complex information to efficiently guide behavior.[19] Data visualization is recognized as an effective way to convey information in health care, where advanced computational power, the ubiquity of electronic health records, and accompanying advances in data-gathering methods means that clinical providers are being called upon to derive actionable information from larger and larger sets of increasingly complex data. The increasing capacity to generate and electronically process health data suggests that data visualization is an important tool for supporting human perception, enabling efficient processing of information for decision-making in health care. With specific reference to the dashboard developed in this study, in our previous work nurses had expressed a need to process information to detect patterns about vital signs (including weight and blood pressure) at the beginning of the home visit.[20] They stressed the importance of tracking trends and of having indicators for abnormal readings. The dashboard enabled these cognitive processes through visualization of vital sign data over time, and use of color to highlight abnormal readings.

Visualization is crucial for helping nurses and other members of health care teams make sense of the structure and underlying patterns in their patients' data. The insights gained from these underlying patterns have potential to answer vital questions at both at the point of care and for the field of nursing.[21] However, perceptual errors stemming from suboptimally designed visual elements may pose a risk to the efficacy and safety of such systems.[22] [23] [24] Therefore, it is critical that systems that capitalize on the power of visualization techniques are evaluated on the perceptual strengths and weakness of such displays.[25] [26]

Conclusion

The growing capacity to generate and electronically process health data suggests that data visualization will be increasingly important.[27] Adequate heuristic criteria for evaluating visualization systems will increase the value of such systems by contributing to decreasing the chance of errors of interpretation or meaning generated by poorly developed visualizations. A checklist of usability heuristics for evaluating information visualization systems can be valuable for assuring high quality in electronic data systems developed for health care.

Clinical Relevance Statement

The checklist of usability heuristics for evaluating information visualization systems that was created in this study can be valuable for assuring high quality in electronic data systems developed for health care.

Multiple Choice Question

What is the main purpose of using an information visualization system to display data?

• Summarize results

• Amplify cognition

• Improve coordination

• Enhance safety

Correct Answer: The correct answer is option b, amplify cognition. Information visualization systems provide “tools and techniques for gaining insight and understanding in a data set, or more generally to amplify cognition” (the “Background and Significance” section). Visual representations of data enable individuals to process information more efficiently than text, and transform even complex information to efficiently guide behavior. Visualization is an effective way to convey information to health care teams that are being called upon to derive actionable information from larger and larger sets of increasingly complex data. Visualization systems are important tools for supporting human perception, enabling efficient processing of information for decision-making in health care.

• References

• 1 Andrews K. Evaluation comes in many guises. In: Proceedings of the 2008 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization (BELIV, 2008); Florence, Italy. New York: ACM; 2008
• 2 Hermawati S, Lawson G. Establishing usability heuristics for heuristics evaluation in a specific domain: is there a consensus?. Appl Ergon 2016; 56: 34-51
• 3 10 usability heuristics for interface design. Available at: https://www.nngroup.com/articles/ten-usability-heuristics/ . Accessed June 10, 2018
• 4 Nielsen J, Molich R. Heuristic evaluation of user interfaces. In: CHI '90 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Seattle, Washington. New York: ACM; 1990 :249–256
• 5 Pierotti D. Heuristic Evaluation - A System Checklist. Available at: ftp://ftp.cs.uregina.ca/pub/class/305/lab2/example-he.html . Accessed June 10, 2018
• 6 Tarrell A, Forsell C, Fruhling AL, Grinstein G, Borgo R, Scholtz J. Toward Visualization-Specific Heuristic Evaluation. In: BELIV '14 Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization Paris, France. New York: ACM; 2014 :110–117
• 7 Väätäjä H, Varsaluoma J, Heimonen T. , et al. Information visualization heuristics in practical expert evaluation. In: BELIV '16 Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization Baltimore, MD. New York: ACM; 2016 :36–43
• 8 Gonzalez-Holland E, Whitmer D, Moralez L, Mouloua M. Examination of the use of Nielsen's 10 usability heuristics & outlooks for the future. Proc Hum Factors Ergon Soc Annu Meet 2017; 61 (01) 1472-1475
• 9 Forsell C, Johansson J. An heuristic set for evaluation in information visualization. In: AVI '10 Proceedings of the International Conference on Advanced Visual Interfaces Roma, Italy. New York: ACM; 2010 :199–206
• 10 Zuk T, Schlesier L, Neumann P, Hancock MS, Carpendale S. Heuristics for information visualization evaluation. In: BELIV '06 Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization Venice, Italy. New York: ACM; 2006 :1–6
• 11 Freitas C, Luzzardi P, Cava R, Winckler M, Pimenta MS, Nedel L. Evaluating usability of information visualization techniques. In: Proceedings of the 5th Symposium on Human Factors in Computer Systems, IHC 2002; Brazil. Brazil: Brazilian Computer Society; 2002 :40–51
• 12 Delbecq AL, Van de Ven AH, Gustafson DH. Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Glenview, IL: Scott Foresman; 1975
• 13 Delbecq AL, Van de Ven AH. A group process model for problem identification and program planning. J Appl Behav Sci 1971; 7 (04) 466-492
• 14 Amar R, Stasko J. A knowledge task-based framework for design and evaluation of information visualizations. In: Proceedings of IEEE Symposium on Information Visualization, 2004 INFOVIS 2004; Austin, TX, USA. Washington, DC: IEEE Computer Society; 2004
• 15 Scapin DL, Bastien JMC. Ergonomic criteria for evaluating the ergonomic quality of interactive system. Behav Inf Technol 1997; 16 (4–5): 220-231
• 16 Shneiderman B. The eyes have it: a task by data type taxonomy for information visualizations. In: Proceedings of the IEEE Symposium on Visual Languages; Boulder, CO, USA. Washington, DC: IEEE Computer Society Press; 1996 :336–343
• 17 Nielsen J. Enhancing the explanatory power of usability heuristics. In: CHI '94 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: April 24–28, 1994; Boston, MA. New York, NY: ACM; 1994 :152–158
• 18 Zuk T, Carpendale S. Theoretical analysis of uncertainty visualizations. Proc. SPIE 6060, Visualization and Data Analysis 2006, 606007. Doi: 10.1117/12.643631
• 19 Stone JV. Principles of Neural Information Theory: A Tutorial Introduction. Sheffield, UK: Sebtel Press; 2016
• 20 Dowding DW, Russell D, Onorato N, Merrill JA. Technology solutions to support care continuity in home care: a focus group study. J Healthc Qual 2017
• 21 Docherty SL, Vorderstrasse A, Brandon D, Johnson C. Visualization of multidimensional data in nursing science. West J Nurs Res 2016; 39 (01) 112-126
• 22 Schumacher R, Lowry S. NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records. Gaithersburg, MD: National Institute of Standards and Technology; 2010
• 23 Rind A, Wang T, Aigner W. , et al. Interactive information visualization to explore and query electronic health records. Foundations Trends Human Comp Interact 2013; 5 (03) 207-298
• 24 Wang TD, Wongsuphasawat K, Plaisant C, Shneiderman B. Extracting insights from electronic health records: case studies, a visual analytics process model, and design recommendations. J Med Syst 2011; 35 (05) 1135-1152
• 25 West VL, Borland D, Hammond WE. Innovative information visualization of electronic health record data: a systematic review. J Am Med Inform Assoc 2015; 22 (02) 330-339
• 26 Shneiderman B, Plaisant C, Hesse B. Improving healthcare with interactive visualization. Computer 2013; 46 (05) 58-66
• 27 National Library of Medicine (U.S.). Board of Regents: a platform for biomedical discovery and data-powered health: National Library of Medicine strategic plan 2017–202. In: NIH Publication. Bethesda, MD: National Institutes of Health, National Library of Medicine; 2017