A Concise and Practical Framework for the Development and Usability Evaluation of Patient Information Websites

L.W. Peute; S.L. Knijnenburg; L.C. Kremer; M.W.M. Jaspers

doi:10.4338/ACI-2014-11-RA-0109

Applied Clinical Informatics, Table of Contents

Appl Clin Inform 2015; 06(02): 383-399
DOI: 10.4338/ACI-2014-11-RA-0109

Research Article

Schattauer GmbH

A Concise and Practical Framework for the Development and Usability Evaluation of Patient Information Websites

L.W. Peute^*

¹Center for Human Factors Engineering of Health Information Technology (HIT-Lab), Academic Medical Center, University of Amsterdam, The Netherlands

²Dept. of Medical Informatics, Academic Medical Center, Amsterdam, the Netherlands

,

S.L. Knijnenburg^*

¹Center for Human Factors Engineering of Health Information Technology (HIT-Lab), Academic Medical Center, University of Amsterdam, The Netherlands

²Dept. of Medical Informatics, Academic Medical Center, Amsterdam, the Netherlands

³Dept. of Pediatric Oncology, Emma’s Children Hospital/Academic Medical Center, Amsterdam, The Netherlands

,

L.C. Kremer

³Dept. of Pediatric Oncology, Emma’s Children Hospital/Academic Medical Center, Amsterdam, The Netherlands

,

M.W.M. Jaspers

¹Center for Human Factors Engineering of Health Information Technology (HIT-Lab), Academic Medical Center, University of Amsterdam, The Netherlands

²Dept. of Medical Informatics, Academic Medical Center, Amsterdam, the Netherlands

› Author Affiliations

Abstract

Summary

Background: The Website Developmental Model for the Healthcare Consumer (WDMHC) is an extensive and successfully evaluated framework that incorporates user-centered design principles. However, due to its extensiveness its application is limited. In the current study we apply a subset of the WDMHC framework in a case study concerning the development and evaluation of a website aimed at childhood cancer survivors (CCS).

Objective: To assess whether the implementation of a limited subset of the WDMHC-framework is sufficient to deliver a high-quality website with few usability problems, aimed at a specific patient population.

Methods: The website was developed using a six-step approach divided into three phases derived from the WDMHC: 1) information needs analysis, mock-up creation and focus group discussion; 2) website prototype development; and 3) heuristic evaluation (HE) and think aloud analysis (TA). The HE was performed by three double experts (knowledgeable both in usability engineering and childhood cancer survivorship), who assessed the site using the Nielsen heuristics. Eight end-users were invited to complete three scenarios covering all functionality of the website by TA.

Results: The HE and TA were performed concurrently on the website prototype. The HE resulted in 29 unique usability issues; the end-users performing the TA encountered eleven unique problems. Four issues specifically revealed by HE concerned cosmetic design flaws, whereas two problems revealed by TA were related to website content.

Conclusion: Based on the subset of the WDMHC framework we were able to deliver a website that closely matched the expectancy of the end-users and resulted in relatively few usability problems during end-user testing. With the successful application of this subset of the WDMHC, we provide developers with a clear and easily applicable framework for the development of healthcare web-sites with high usability aimed at specific medical populations.

Citation: Peute LW; Knijnenburg SL; Kremer LC; Jaspers MWM. A concise and practical framework for the development and usability evaluation of patient information websites. Appl Clin Inf 2015; 6: 383–399

http://dx.doi.org/10.4338/ACI-11-RA-0109

Keywords

Patient education - childhood cancer - survivorship - website - human factors

Full Text

References

References
1 Heymans HS, Caron HN. Childhood cancers in the Netherlands (1989–1997). Ned Tijdschr Geneeskd 2001; 145: 1442-1444. PM:11503310
2 Geenen MM, Cardous-Ubbink MC, Kremer LC, van den Bos C, van der Pal HJ, Heinen RC, Jaspers MW, Koning CC, Oldenburger F, Langeveld NE, Hart AA, Bakker PJ, Caron HN, van Leeuwen FE. Medical assessment of adverse health outcomes in long-term survivors of childhood cancer. JAMA 2007; 297: 2705-2715. PM:17595271
3 Jaspers MW, Van den Bos C, Heinen RC, Bakker PJ, Geenen MM, Kremer LC, Van Leeuwen F, Caron HN. Development of a national protocol to screen Dutch cancer survivors on late cancer treatment effects. Int J Med Inform 2007; 76: 297-305. PM:16533618
4 Jaspers MW, Caron H, Behrendt H, van den Bos C, Bakker P, Van Leeuwen F. The development of a new information model for a pediatric cancer registry on late treatment sequelae in The Netherlands. Stud Health Technol Inform 2000; 77: 895-899. PM:11187683
5 Kremer LCM, Jaspers MWM, van Leeuwen FE. et al. Landelijke richtlijnen voor follow-up van overlevenden van kinderkanker. Tijdschrift voor Kindergeneeskunde 2006; 74: 214-218.
6 Byrne J, Lewis S, Halamek L, Connelly RR, Mulvihill JJ. Childhood cancer survivors’ knowledge of their diagnosis and treatment. Ann Intern Med 1989; 110: 400-403. PM:2916808
7 Bashore L. Childhood and adolescent cancer survivors’ knowledge of their disease and effects of treatment. J Pediatr Oncol Nurs 2004; 21: 98-102. PM:15125553
8 Kadan-Lottick NS, Robison LL, Gurney JG, Neglia JP, Yasui Y, Hayashi R, Hudson M, Greenberg M, Mertens AC. Childhood cancer survivors’ knowledge about their past diagnosis and treatment: Childhood Cancer Survivor Study. JAMA 2002; 287: 1832-1839. PM:11939869
9 Lewis D. Computer-based approaches to patient education: a review of the literature. J Am Med Inform Assoc 1999; 6: 272-282. PM:10428001
10 Tuil WS, ten Hoopen AJ, Braat DD, de Vries Robbé PF, Kremer JA. Patient-centred care: using online personal medical records in IVF practice. Hum Reprod 2006; 21: 2955-2959. PM:16982658
11 Grant RW, Wald JS, Poon EG, Schnipper JL, Gandhi TK, Volk LA, Middleton B. Design and implementation of a web-based patient portal linked to an ambulatory care electronic health record: patient gateway for diabetes collaborative care. Diabetes Technol Ther 2006; 8: 576-586. PM:17037972
12 Johnson CM, Turley JP. A new approach to building web-based interfaces for healthcare consumers. Elec J Health Inform 2007; 2: e2. http://www.ejhi.net
13 Taylor HA, Sullivan D, Mullen C, Johnson CM. Implementation of a user-centered framework in the development of a web-based health information database and call center. J Biomed Inform 2011; 44: 897-908. PM:21396486
14 Dix A, Finlay J, Abowd GD, Beale R. Cognitive models. In. Human-Computer Interaction. Harlow, United Kingdom: Pearson Education Limited; 2004. p 436-441. ISBN: 0130-461091.
15 Ruland CM, Jeneson A, Andersen T, Andersen R, Slaughter L, Bente-Schjødt-Osmo Moore SM. Designing tailored Internet support to assist cancer patients in illness management. AMIA Annu Symp Proc 2007; 635-639. PM:18693913
16 Grama LM, Beckwith M, Bittinger W, Blais D, Lollar C, Middleswarth A, Noone M, Price D, Quint-Kasner S, Shields V, Wright LW. The role of user input in shaping online information from the National Cancer Institute. J Med Internet Res 2005; 7: e25. PM:15998616
17 Sox CM, Gribbons WM, Loring BA, Mandl KD, Batista R, Porter SC. Patient-centered design of an information management module for a personally controlled health record. J Med Internet Res 2010; 12: e36. PM:20805091
18 Kushniruk AW, Patel VL, Cimino JJ. Evaluation of Web-based patient information resources: application in the assessment of a patient clinical information system. Proc AMIA Symp 2000; 443-447. PM:11079922
19 Jaspers MW. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int J Med Inform. 2008 PM:19046928
20 Nielsen J. Designing Web Usability. Indianapolis: New Riders Publishing; 1999. ISBN: <[26] ISSN/ISBN>.
21 Joshi A, Arora M, Dai L, Price K, Vizer L, Sears A. Usability of a patient education and motivation tool using heuristic evaluation. J Med Internet Res 2009; 11: e47. PM:19897458
22 Nielsen J, Molich R. Heuristic evaluation of user interfaces. In. ACM INTERCHI’93 Conference on Human Factors in Computing Systems. New York: ACM Press; 1990. p 249-256.
23 Nielsen J. Heuristic Evaluation. In: John Wiley & Sons, editor. Usability Inspection Methods. New York: 1994. p 25-62
24 Nielsen J. Estimating the Number of Subjects Needed for A Thinking Aloud Test. International Journal of Human-Computer Studies 1994; 41: 385-397. ISI:A1994PR10100006
25 Brooke J. SUS: a ,,quick and dirty“ usability scale. In: Jordan P, Thomas B, Weerdmeester B, McClelland A.. editors. Usability Evaluation in Industry.. London: Taylor and Francis 1996
26 Law ELC, Hvannberg ET. Complementarity and Convergence of Heuristic Evaluation and Usability Test: A Case Study of UNIVERSAL Brokerage Platform. NordiCHI 2002: 71-80.
27 Fu LM, Salvendy G, Turley L. Effectiveness of user testing and heuristic evaluation as a function of performance classification. Behaviour & Information Technology 2002; 21: 137-143. ISI:000177523400005
28 Beuscart-Zephir MC, Leroy N, Alao O, Darmoni S. Usability assessment study of a web site displaying medical resources on line: the CISMeF. Stud Health Technol Inform 2002; 90: 133-137. PM:15460675
29 Ostergren MJ, Karras BT. ActiveOptions: leveraging existing knowledge and usability testing to develop a physical activity program website for older adults. AMIA Annu Symp Proc 2007; 578-582. PM:18693902
30 Nahm ES, Preece J, Resnick B, Mills ME. Usability of health Web sites for older adults: a preliminary study. Comput Inform Nurs 2004; 22: 326-334. PM:15602301
31 Lai TY. Iterative refinement of a tailored system for self-care management of depressive symptoms in people living with HIV/AIDS through heuristic evaluation and end user testing. Int J Med Inform 2007; 76 (Suppl. 02) S317-S324. PM:17616431
32 Knijnenburg SL, Kremer LC, Van den Bos C, Braam KI, Jaspers MW. Health information needs of childhood cancer survivors and their family. Pediatr Blood Cancer 2010; 54: 123-127. PM:19743299
33 Kinzie MB, Cohn WF, Julian MF, Knaus WA. A user-centered model for web site design: needs assessment, user interface design, and rapid prototyping. J Am Med Inform Assoc 2002; 9: 320-330. PM:12087113
34 Hess SL, Johannsdottir IM, Hamre H, Kiserud CE, Loge JH, Fosså SD. Adult survivors of childhood malignant lymphoma are not aware of their risk of late effects. Acta Oncol 2011; 50: 653-659. PM:21261507
35 Ossebaard HC, Seydel ER, van Gemert-Pijnen L. Online usability and patients with long-term conditions: A mixed-methods approach. Int J Med Inform 2012; 81: 374-387. PM:22261086
36 Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. In. CHI ’93. New York, NY, USA: ACM; 1993. p 206-213.
37 Nielsen J. Usability Evaluation and Inspection Methods. In: Ashlund S, Mullet K, Henderson A. , et al., editors. Reading, MA: Addison-Wesley; 1993
38 Lathan CE, Sebrechts MM, Newman DJ, Doarn CR. Heuristic evaluation of a web-based interface for internet telemedicine. Telemed J 1999; 5: 177-185. PM:10908430
39 Knijnenburg SL, Kremer LC, Versluys AB, Braam KI, Mud MS, van der Pal HJ, Caron HN, Jaspers MW. Evaluation of a patient information website for childhood cancer survivors. Support Care Cancer. 2012 PM:23007883