Download PDF
Appl Clin Inform 2015; 06(02): 383-399
DOI: 10.4338/ACI-2014-11-RA-0109
Research Article
Schattauer GmbH

A Concise and Practical Framework for the Development and Usability Evaluation of Patient Information Websites

L.W. Peute   *
1   Center for Human Factors Engineering of Health Information Technology (HIT-Lab), Academic Medical Center, University of Amsterdam, The Netherlands
2   Dept. of Medical Informatics, Academic Medical Center, Amsterdam, the Netherlands
,
S.L. Knijnenburg   *
1   Center for Human Factors Engineering of Health Information Technology (HIT-Lab), Academic Medical Center, University of Amsterdam, The Netherlands
2   Dept. of Medical Informatics, Academic Medical Center, Amsterdam, the Netherlands
3   Dept. of Pediatric Oncology, Emma’s Children Hospital/Academic Medical Center, Amsterdam, The Netherlands
,
L.C. Kremer
3   Dept. of Pediatric Oncology, Emma’s Children Hospital/Academic Medical Center, Amsterdam, The Netherlands
,
M.W.M. Jaspers
1   Center for Human Factors Engineering of Health Information Technology (HIT-Lab), Academic Medical Center, University of Amsterdam, The Netherlands
2   Dept. of Medical Informatics, Academic Medical Center, Amsterdam, the Netherlands
› Author Affiliations
Further Information

Publication History

received: 25 November 2014

accepted: 09 April 2015

Publication Date:
19 December 2017 (online)

   Permissions and Reprints

Summary

Background: The Website Developmental Model for the Healthcare Consumer (WDMHC) is an extensive and successfully evaluated framework that incorporates user-centered design principles. However, due to its extensiveness its application is limited. In the current study we apply a subset of the WDMHC framework in a case study concerning the development and evaluation of a website aimed at childhood cancer survivors (CCS).

Objective: To assess whether the implementation of a limited subset of the WDMHC-framework is sufficient to deliver a high-quality website with few usability problems, aimed at a specific patient population.

Methods: The website was developed using a six-step approach divided into three phases derived from the WDMHC: 1) information needs analysis, mock-up creation and focus group discussion; 2) website prototype development; and 3) heuristic evaluation (HE) and think aloud analysis (TA). The HE was performed by three double experts (knowledgeable both in usability engineering and childhood cancer survivorship), who assessed the site using the Nielsen heuristics. Eight end-users were invited to complete three scenarios covering all functionality of the website by TA.

Results: The HE and TA were performed concurrently on the website prototype. The HE resulted in 29 unique usability issues; the end-users performing the TA encountered eleven unique problems. Four issues specifically revealed by HE concerned cosmetic design flaws, whereas two problems revealed by TA were related to website content.

Conclusion: Based on the subset of the WDMHC framework we were able to deliver a website that closely matched the expectancy of the end-users and resulted in relatively few usability problems during end-user testing. With the successful application of this subset of the WDMHC, we provide developers with a clear and easily applicable framework for the development of healthcare web-sites with high usability aimed at specific medical populations.

Citation: Peute LW; Knijnenburg SL; Kremer LC; Jaspers MWM. A concise and practical framework for the development and usability evaluation of patient information websites. Appl Clin Inf 2015; 6: 383–399

http://dx.doi.org/10.4338/ACI-11-RA-0109

Keywords

Patient education - childhood cancer - survivorship - website - human factors

* These authors contributed equally to the paper


 

  • References

  • 1 Heymans HS, Caron HN. Childhood cancers in the Netherlands (1989–1997). Ned Tijdschr Geneeskd 2001; 145: 1442-1444. PM:11503310
    PubMedGoogle Scholar
  • 2 Geenen MM, Cardous-Ubbink MC, Kremer LC, van den Bos C, van der Pal HJ, Heinen RC, Jaspers MW, Koning CC, Oldenburger F, Langeveld NE, Hart AA, Bakker PJ, Caron HN, van Leeuwen FE. Medical assessment of adverse health outcomes in long-term survivors of childhood cancer. JAMA 2007; 297: 2705-2715. PM:17595271
    CrossrefPubMedGoogle Scholar
  • 3 Jaspers MW, Van den Bos C, Heinen RC, Bakker PJ, Geenen MM, Kremer LC, Van Leeuwen F, Caron HN. Development of a national protocol to screen Dutch cancer survivors on late cancer treatment effects. Int J Med Inform 2007; 76: 297-305. PM:16533618
    CrossrefPubMedGoogle Scholar
  • 4 Jaspers MW, Caron H, Behrendt H, van den Bos C, Bakker P, Van Leeuwen F. The development of a new information model for a pediatric cancer registry on late treatment sequelae in The Netherlands. Stud Health Technol Inform 2000; 77: 895-899. PM:11187683
    PubMedGoogle Scholar
  • 5 Kremer LCM, Jaspers MWM, van Leeuwen FE. et al. Landelijke richtlijnen voor follow-up van overlevenden van kinderkanker. Tijdschrift voor Kindergeneeskunde 2006; 74: 214-218.
    CrossrefPubMedGoogle Scholar
  • 6 Byrne J, Lewis S, Halamek L, Connelly RR, Mulvihill JJ. Childhood cancer survivors’ knowledge of their diagnosis and treatment. Ann Intern Med 1989; 110: 400-403. PM:2916808
    CrossrefPubMedGoogle Scholar
  • 7 Bashore L. Childhood and adolescent cancer survivors’ knowledge of their disease and effects of treatment. J Pediatr Oncol Nurs 2004; 21: 98-102. PM:15125553
    CrossrefPubMedGoogle Scholar
  • 8 Kadan-Lottick NS, Robison LL, Gurney JG, Neglia JP, Yasui Y, Hayashi R, Hudson M, Greenberg M, Mertens AC. Childhood cancer survivors’ knowledge about their past diagnosis and treatment: Childhood Cancer Survivor Study. JAMA 2002; 287: 1832-1839. PM:11939869
    CrossrefPubMedGoogle Scholar
  • 9 Lewis D. Computer-based approaches to patient education: a review of the literature. J Am Med Inform Assoc 1999; 6: 272-282. PM:10428001
    CrossrefPubMedGoogle Scholar
  • 10 Tuil WS, ten Hoopen AJ, Braat DD, de Vries Robbé PF, Kremer JA. Patient-centred care: using online personal medical records in IVF practice. Hum Reprod 2006; 21: 2955-2959. PM:16982658
    CrossrefPubMedGoogle Scholar
  • 11 Grant RW, Wald JS, Poon EG, Schnipper JL, Gandhi TK, Volk LA, Middleton B. Design and implementation of a web-based patient portal linked to an ambulatory care electronic health record: patient gateway for diabetes collaborative care. Diabetes Technol Ther 2006; 8: 576-586. PM:17037972
    CrossrefPubMedGoogle Scholar
  • 12 Johnson CM, Turley JP. A new approach to building web-based interfaces for healthcare consumers. Elec J Health Inform 2007; 2: e2. http://www.ejhi.net
    PubMedGoogle Scholar
  • 13 Taylor HA, Sullivan D, Mullen C, Johnson CM. Implementation of a user-centered framework in the development of a web-based health information database and call center. J Biomed Inform 2011; 44: 897-908. PM:21396486
    CrossrefPubMedGoogle Scholar
  • 14 Dix A, Finlay J, Abowd GD, Beale R. Cognitive models. In. Human-Computer Interaction. Harlow, United Kingdom: Pearson Education Limited; 2004. p 436-441. ISBN: 0130-461091.
    Google Scholar
  • 15 Ruland CM, Jeneson A, Andersen T, Andersen R, Slaughter L, Bente-Schjødt-Osmo Moore SM. Designing tailored Internet support to assist cancer patients in illness management. AMIA Annu Symp Proc 2007; 635-639. PM:18693913
    PubMedGoogle Scholar
  • 16 Grama LM, Beckwith M, Bittinger W, Blais D, Lollar C, Middleswarth A, Noone M, Price D, Quint-Kasner S, Shields V, Wright LW. The role of user input in shaping online information from the National Cancer Institute. J Med Internet Res 2005; 7: e25. PM:15998616
    CrossrefPubMedGoogle Scholar
  • 17 Sox CM, Gribbons WM, Loring BA, Mandl KD, Batista R, Porter SC. Patient-centered design of an information management module for a personally controlled health record. J Med Internet Res 2010; 12: e36. PM:20805091
    CrossrefPubMedGoogle Scholar
  • 18 Kushniruk AW, Patel VL, Cimino JJ. Evaluation of Web-based patient information resources: application in the assessment of a patient clinical information system. Proc AMIA Symp 2000; 443-447. PM:11079922
    PubMedGoogle Scholar
  • 19 Jaspers MW. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int J Med Inform. 2008 PM:19046928
    PubMedGoogle Scholar
  • 20 Nielsen J. Designing Web Usability. Indianapolis: New Riders Publishing; 1999. ISBN: <[26] ISSN/ISBN>.
    Google Scholar
  • 21 Joshi A, Arora M, Dai L, Price K, Vizer L, Sears A. Usability of a patient education and motivation tool using heuristic evaluation. J Med Internet Res 2009; 11: e47. PM:19897458
    CrossrefPubMedGoogle Scholar
  • 22 Nielsen J, Molich R. Heuristic evaluation of user interfaces. In. ACM INTERCHI’93 Conference on Human Factors in Computing Systems. New York: ACM Press; 1990. p 249-256.
    Google Scholar
  • 23 Nielsen J. Heuristic Evaluation. In: John Wiley & Sons, editor. Usability Inspection Methods. New York: 1994. p 25-62
    Google Scholar
  • 24 Nielsen J. Estimating the Number of Subjects Needed for A Thinking Aloud Test. International Journal of Human-Computer Studies 1994; 41: 385-397. ISI:A1994PR10100006
    PubMedGoogle Scholar
  • 25 Brooke J. SUS: a ,,quick and dirty“ usability scale. In: Jordan P, Thomas B, Weerdmeester B, McClelland A.. editors. Usability Evaluation in Industry.. London: Taylor and Francis 1996
    PubMedGoogle Scholar
  • 26 Law ELC, Hvannberg ET. Complementarity and Convergence of Heuristic Evaluation and Usability Test: A Case Study of UNIVERSAL Brokerage Platform. NordiCHI 2002: 71-80.
    PubMedGoogle Scholar
  • 27 Fu LM, Salvendy G, Turley L. Effectiveness of user testing and heuristic evaluation as a function of performance classification. Behaviour & Information Technology 2002; 21: 137-143. ISI:000177523400005
    CrossrefPubMedGoogle Scholar
  • 28 Beuscart-Zephir MC, Leroy N, Alao O, Darmoni S. Usability assessment study of a web site displaying medical resources on line: the CISMeF. Stud Health Technol Inform 2002; 90: 133-137. PM:15460675
    PubMedGoogle Scholar
  • 29 Ostergren MJ, Karras BT. ActiveOptions: leveraging existing knowledge and usability testing to develop a physical activity program website for older adults. AMIA Annu Symp Proc 2007; 578-582. PM:18693902
    PubMedGoogle Scholar
  • 30 Nahm ES, Preece J, Resnick B, Mills ME. Usability of health Web sites for older adults: a preliminary study. Comput Inform Nurs 2004; 22: 326-334. PM:15602301
    CrossrefPubMedGoogle Scholar
  • 31 Lai TY. Iterative refinement of a tailored system for self-care management of depressive symptoms in people living with HIV/AIDS through heuristic evaluation and end user testing. Int J Med Inform 2007; 76 (Suppl. 02) S317-S324. PM:17616431
    CrossrefPubMedGoogle Scholar
  • 32 Knijnenburg SL, Kremer LC, Van den Bos C, Braam KI, Jaspers MW. Health information needs of childhood cancer survivors and their family. Pediatr Blood Cancer 2010; 54: 123-127. PM:19743299
    CrossrefPubMedGoogle Scholar
  • 33 Kinzie MB, Cohn WF, Julian MF, Knaus WA. A user-centered model for web site design: needs assessment, user interface design, and rapid prototyping. J Am Med Inform Assoc 2002; 9: 320-330. PM:12087113
    CrossrefPubMedGoogle Scholar
  • 34 Hess SL, Johannsdottir IM, Hamre H, Kiserud CE, Loge JH, Fosså SD. Adult survivors of childhood malignant lymphoma are not aware of their risk of late effects. Acta Oncol 2011; 50: 653-659. PM:21261507
    CrossrefPubMedGoogle Scholar
  • 35 Ossebaard HC, Seydel ER, van Gemert-Pijnen L. Online usability and patients with long-term conditions: A mixed-methods approach. Int J Med Inform 2012; 81: 374-387. PM:22261086
    CrossrefPubMedGoogle Scholar
  • 36 Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. In. CHI ’93. New York, NY, USA: ACM; 1993. p 206-213.
    Google Scholar
  • 37 Nielsen J. Usability Evaluation and Inspection Methods. In: Ashlund S, Mullet K, Henderson A. , et al., editors. Reading, MA: Addison-Wesley; 1993
    PubMedGoogle Scholar
  • 38 Lathan CE, Sebrechts MM, Newman DJ, Doarn CR. Heuristic evaluation of a web-based interface for internet telemedicine. Telemed J 1999; 5: 177-185. PM:10908430
    CrossrefPubMedGoogle Scholar
  • 39 Knijnenburg SL, Kremer LC, Versluys AB, Braam KI, Mud MS, van der Pal HJ, Caron HN, Jaspers MW. Evaluation of a patient information website for childhood cancer survivors. Support Care Cancer. 2012 PM:23007883
    PubMedGoogle Scholar