Download PDF
Appl Clin Inform 2014; 05(02): 571-588
DOI: 10.4338/ACI-2014-01-RA-0005
Research Article
Schattauer GmbH

Evaluation of a Korean version of a tool for assessing the incorporation of human factors into a medication-related decision support system: the I-MeDeSA

I. Cho
1   Department of Nursing, School of Medicine, Inha University, Incheon, Korea
2   Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, MA, USA
3   Harvard Medical School, Boston, MA, USA
,
J. Lee
2   Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, MA, USA
3   Harvard Medical School, Boston, MA, USA
4   Department of Emergency Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea
5   Department of Biomedical Informatics, Asan Medical Center, Seoul, Korea
,
H. Han
6   Department of Pharmacy, Asan Medical Center, Seoul, Korea
,
S. Phansalkar
2   Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, MA, USA
3   Harvard Medical School, Boston, MA, USA
7   Partners Healthcare Systems, Wellesley, MA, USA
8   Wolters Kluwer Health, Indianapolis, IN, USA
,
D.W. Bates
2   Division of General Internal Medicine, Brigham and Women’s Hospital, Boston, MA, USA
3   Harvard Medical School, Boston, MA, USA
7   Partners Healthcare Systems, Wellesley, MA, USA
› Author Affiliations
Further Information

Publication History

Received: 13 January 2014

Accepted: 07 May 2014

Publication Date:
21 December 2017 (online)

   Permissions and Reprints

Summary

Objective: The Instrument for Evaluating Human-Factor Principles in Medication-Related Decision Support Alerts (I-MeDeSA) was developed recently in the US with a view towards improving considerations of human-factor principles when designing alerts for clinical decision support (CDS) systems. This study evaluated the generalizability of this tool, in cooperation with its authors, across cultures by applying it to a Korean system. We also examined opportunities to promote user acceptance of the system.

Methods: We developed a Korean version of the I-MeDeSA (K-I-MeDeSA) and used it to evaluate drug-drug interaction alerts in a large academic tertiary hospital in Seoul. We involved four reviewers (A, B, C, and D). Two (A and B) conducted the initial independent scoring, while the other two (C and D) performed a final review and assessed feedback from the initial reviewers. The obtained scores were compared with those from 13 previously reported CDS systems. The feedback was summarized qualitatively.

Results: The translation of the I-MeDeSA had excellent interrater agreement in terms of face validity (scale-level content validity index = 0.95). The system’s K-I-MeDeSA score was 10 out of 26, with a good agreement between reviewers (κ = 0.77), which showed a lack of human-factor considerations. The reviewers readily identified two of the nine principles that needed primary improvement: prioritization and text-based information. The reviewers also expressed difficulty judging the following four principles: alarm philosophy, visibility, color, and learnability and confusability. Conclusion: The K-I-MeDeSA was semantically and operationally equivalent to the original tool. Only minor cultural problems were identified, leading the reviewers to suggest the need for clarification of certain words plus a more detailed description of the tool’s rationale and exemplars. Further evaluation is needed to empirically assess whether the implementation of changes in an electronic health record system could improve the adoption of CDS alerts.

Citation: Cho I, Lee J, Han H, Phansalkar S, Bates DW. Evaluation of a Korean version of a tool for assessing the incorporation of human factors into a medicationrelated decision support system: the I-MeDeSA. Appl Clin Inf 2014; 5: 571–588 http://dx.doi.org/10.4338/ACI-01-RA-0005

Keywords

Clinical decision support - interfaces and usability - human-computer interaction - computerized order entry - alert fatigue
 

  • References

  • 1 Linder JA, Ma J, Bates DW, Middleton B, Stafford RS. Electronic health record use and the quality of ambulatory care in the United States. Arch Intern Med 2007; 167 (13) 1400-1405.
    CrossrefPubMedGoogle Scholar
  • 2 Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med 2003; 348 (25) 2526-24534.
    CrossrefPubMedGoogle Scholar
  • 3 Fischer MA, Solomon DH, Teich JM, Avorn J. Conversion from intravenous to oral medications: Assessment of a computerized intervention for hospitalized patients. Arch Intern Med 2003; 163 (21) 2585-259.
    CrossrefPubMedGoogle Scholar
  • 4 Wang SJ, Middleton B, Prosser LA, Bardon CG, Spurr CD, Carchidi PJ, Kittler AF, Goldszer RC, Fairchild DG, Sussman AJ, Kuperman GJ, Bates DW. A cost-benefit analysis of electronic medical records in primary care. Am J Med 2003; 114 (Suppl. 05) 397-403.
    CrossrefPubMedGoogle Scholar
  • 5 Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD. Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med 2003; 18 (12) 967-976.
    CrossrefPubMedGoogle Scholar
  • 6 Khajouei R, Jaspers MWM. The impact of CPOE medication systems’ design aspects on usability, work-flow and medication orders: a systematic review. Methods Inf Med 2010; 49 (Suppl. 01) 3-19.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 7 Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752.
    CrossrefPubMedGoogle Scholar
  • 8 Johnston D, Pan E, Walker J. The value of CPOE in ambulatory settings. J Healthc Inf Manag 2004; 18 (Suppl. 01) 5-8.
    PubMedGoogle Scholar
  • 9 Van Der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006; 13 (Suppl. 02) 138-147.
    CrossrefPubMedGoogle Scholar
  • 10 Nightingale PG, Adu D, Richards NT, Peters M. Implementation of rules based computerised bedside prescribing and administration: intervention study. BMJ 2000; 320 7237 750-753.
    CrossrefPubMedGoogle Scholar
  • 11 Abookire SA, Teich JM, Sandige H, Paterno MD, Martin MT, Kuperman GJ, Bates DW. Improving allergy alerting in a computerized physician order entry system. Proc AMIA Symp 2000: 2-6.
    PubMedGoogle Scholar
  • 12 Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians’ decisions to override computerized drug alerts in primary care. Arch Intern Med 2003; 163 (21) 2625-2631.
    CrossrefPubMedGoogle Scholar
  • 13 Ash JS, Sittig DF, Dykstra RH, Guappone K, Carpenter JD, Seshadri V. Categorizing the unintended socio-technical consequences of computerized provider order entry. Int J Med Inform 2007; 76 (Suppl. 01) S21-S27.
    CrossrefPubMedGoogle Scholar
  • 14 Jung M, Hoerbst A, Hackl WO, Kirrane F, Borbolla D, Jaspers MW, Oertle M, Koutkias V, Ferret L, Mass-ari P, Lawton K, Riedmann D, Darmoni S, Maglaveras N, Lovis C, Ammenwerth E. Attitude of physicians towards automatic alerting in computerized physician order entry systems. A comparative international survey. Methods Inf Med 2013; 52 (Suppl. 02) 99-108.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 15 Payne TH, Nichol WP, Hoey P. Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp 2002: 602-606.
    PubMedGoogle Scholar
  • 16 Cho I, Kim JA, Kho YT, Song SH, Park RW. Clinical decision support system. Seoul: Elsevier Korea; 2010
    Google Scholar
  • 17 Paterno MD, Maviglia SM, Gorman PN, Seger DL, Yoshida E, Seger AC, Bates DW, Gandhi TK. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009; 16 (Suppl. 01) 40-46.
    CrossrefPubMedGoogle Scholar
  • 18 President’s Council of Advisors on Science and Technology.. Report to the president realizing the full potential of helath information technology to improve healthcare for Americans: The path forward. Washington, DC. 2010 pp. 1-108
    PubMedGoogle Scholar
  • 19 Zhang J, Walji MF. TURF: Toward a unified framework of EHR usability. J Biomed Inform 2011; 44 (Suppl. 06) 1056-1067.
    CrossrefPubMedGoogle Scholar
  • 20 Bertman J, Skolnik N, Frieden J. Poor usability keeps EHR adoption rates low. Family Practice News 2010; 40 (Suppl. 08) 54.
    PubMedGoogle Scholar
  • 21 Saitwal H, Feng X, Walji M, Patel V, Zhang J. Assessing performance of an electronic health record (EHR) using cognitive task analysis. Int J Med Inform 2010; 79 (Suppl. 07) 501-506.
    CrossrefPubMedGoogle Scholar
  • 22 Phansalkar S, Edworthy J, Hellier E, Seger DL, Schedlbauer A, Avery AJ, Bates DW. A review of human factors principles for the design and implementation of medication safety alerts in clinical information systems. J Am Med Inform Assoc 2010; 17 (Suppl. 05) 493-501.
    CrossrefPubMedGoogle Scholar
  • 23 Zachariah M, Phansalkar S, Seidling HM, Neri PM, Cresswell KM, Duke J, Bloomrosen M, Volk LA, Bates DW. Development and preliminary evidence for the validity of an instrument assessing implementation of human-factors principles in medication-related decision-support systems - I-MeDeSA. J Am Med Inform Assoc 2011; 18 (Suppl. 01) i62-i72.
    CrossrefPubMedGoogle Scholar
  • 24 Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care 2010; 19 (Suppl. 03) i68-i74.
    CrossrefPubMedGoogle Scholar
  • 25 Herdman M, Fox-Rushby J, Badia X. A model of equivalence in the cultural adaptation of HRQoL instruments: the universalist approach. Quality of life Research 1998; 7 (Suppl. 04) 323-335.
    CrossrefPubMedGoogle Scholar
  • 26 Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health 2007; 30 (Suppl. 04) 459-467.
    CrossrefPubMedGoogle Scholar
  • 27 Zachariah M, Phansalkar S, Seidling HM, Volk LA, Bloomrosen M, Bates DW. Evaluation of medication alerts for compliance with human factors principles: a multi-center study. Proc AMIA Symp 2011: 2021
    PubMedGoogle Scholar
  • 28 Lee Y, Lee J, Lee S. Analysis of drug interaction information. Kor J Clin Pharm 2009; 19 (Suppl. 01) 1-17.
    PubMedGoogle Scholar
  • 29 Park J-Y, Park K-W. The contraindication of comedication drugs and drug utilization review. J Korean Med Assoc 2012; 55 (Suppl. 05) 484-490.
    CrossrefPubMedGoogle Scholar
  • 30 Shah NR, Seger AC, Seger DL, Fiskio JM, Kuperman GJ, Blumenfeld B, Recklet EG, Bates DW, Gandhi YK. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc 2006; 13 (Suppl. 01) 5-11.
    CrossrefPubMedGoogle Scholar
  • 31 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (Suppl. 02) 104-112.
    PubMedGoogle Scholar
  • 32 Choi N-K, Park B-J. Strategy for establishing an effective Korean drug utilization review system. J Korean Med Assoc 2010; 53 (12) 1130-1138.
    CrossrefPubMedGoogle Scholar
  • 33 Phansalkar S, Desai AA, Bell D, Yoshida E, Doole J, Czochanski M, Middleton B, Bates DW. High-priority drug-drug interactions for use in electronic health records. J Am Med Inform Assoc 2012; 19 (Suppl. 05) 735-743.
    CrossrefPubMedGoogle Scholar