Download PDF
Appl Clin Inform 2016; 07(04): 1007-1024
DOI: 10.4338/ACI-2016-03-RA-0036
Research Article
Schattauer GmbH

Usability and Workflow Evaluation of “RhEumAtic Disease activitY” (READY)

A Mobile Application for Rheumatology Patients and Providers
Po-Yin Yen
1   Department of Biomedical Informatics, The Ohio State University, Columbus, OH
,
Barbara Lara
1   Department of Biomedical Informatics, The Ohio State University, Columbus, OH
,
Marcelo Lopetegui
3   Departamento de Informática Biomédica, Facultad de Medicina, Clínica Alemana Universidad del Desarrollo, Santiago, Chile
,
Aseem Bharat
4   Division of Clinical Immunology and Rheumatology, University of Alabama at Birmingham, Birmingham, AL
,
Stacy Ardoin
2   Department of Internal Medicine, The Ohio State University, Columbus, OH
,
Bernadette Johnson
4   Division of Clinical Immunology and Rheumatology, University of Alabama at Birmingham, Birmingham, AL
,
Puneet Mathur
1   Department of Biomedical Informatics, The Ohio State University, Columbus, OH
,
Peter J. Embi
1   Department of Biomedical Informatics, The Ohio State University, Columbus, OH
2   Department of Internal Medicine, The Ohio State University, Columbus, OH
,
Jeffrey R. Curtis
4   Division of Clinical Immunology and Rheumatology, University of Alabama at Birmingham, Birmingham, AL
› Author Affiliations Funding This project was supported by the National Institutes of Health (P60 AR064172) and Genentech.
Further Information

Publication History

received: 24 March 2016

accepted: 19 September 2016

Publication Date:
18 December 2017 (online)

   Permissions and Reprints

Summary

Background RhEumAtic Disease activitY (READY) is a mobile health (mHealth) application that aims to create a shared platform integrating data from both patients and physicians, with a particular emphasis on arthritis disease activity.

Methods We made READY available on an iPad and pilot implemented it at a rheumatology outpatient clinic. We conducted 1) a usability evaluation study to explore patients’ and physicians’ interactions with READY, and 2) a time motion study (TMS) to observe the clinical workflow before and after the implementation.

Results A total of 33 patients and 15 physicians participated in the usability evaluation. We found usability problems in navigation, data entry, pain assessment, documentation, and instructions along with error messages. Despite these issues, 25 (75,76%) patients reported they liked READY. Physicians provided mixed feedback because they were concerned about the impact of READY on clinical workflow. Six physicians participated in the TMS. We observed 47 patient visits (44.72 hours) in the pre-implementation phase, and 42 patient visits (37.82 hours) in the post-implementation phase. We found that patients spent more time on READY than paper (4.39mins vs. 2.26mins), but overall, READY did not delay the workflow (pre = 52.08 mins vs. post = 45.46 mins). This time difference may be compensated with READY eliminating a workflow step for the staff.

Conclusion Patients preferred READY to paper documents. Many found it easier to input information because of the larger font size and the ease of ‘tapping’ rather than writing-out or circling answers. Even though patients spent more time on READY than using paper documents, the longer usage of READY was mainly due to when troubleshooting was needed. Most patients did not have problems after receiving initial support from the staff. This study not only enabled improvements to the software but also serves as good reference for other researchers or institutional decision makers who are interested in implementing such a technology.

Citation: Yen P, Lara B, Lopetegui M, Bharat A, Ardoin S, Johnson B, Mathur P, Embi P, Curtis J. Usability and workflow evaluation of “RhEumAtic Disease activitY” (READY).

Keywords

Rheumatoid arthritis - mobile health - disease activity - time motion study - mobile application - mHealth - workflows and human interactions - mobile computing and communication - usability
 

  • References

  • 1 Hersh WR, Cimino J, Payne PR, Embi P, Logan J, Weiner M, Bernstam EV, Lehmann H, Hripcsak G, Hartzog T, Saltz J. Recommendations for the use of operational electronic health record data in comparative effectiveness research. EGEMS (Wash DC) 2013; 01 (01) 1018.
    PubMedGoogle Scholar
  • 2 Lopetegui MA, Bai S, Yen PY, Lai A, Embi P, Payne PR. Inter-observer reliability assessments in time motion studies: the foundation for meaningful clinical workflow analysis. AMIA Annu Symp Proc 2013; 2013: 889-96.
    PubMedGoogle Scholar
  • 3 Singh JA, Saag KG, Bridges Jr SL, Akl EA, Bannuru RR, Sullivan MC, Vaysbrot E, McNaughton C, Osani M, Shmerling RH, Curtis JR, Furst DE, Parks D, Kavanaugh A, O’Dell J, King C, Leong A, Matteson EL, Schousboe JT, Drevlow B, Ginsberg S, Grober J, St Clair EW, Tindall E, Miller AS, McAlindon T. 2015 American College of Rheumatology Guideline for the Treatment of Rheumatoid Arthritis. Arthritis Rheumatol 2016; 68 (01) 1-26.
    CrossrefPubMedGoogle Scholar
  • 4 Pincus T, Swearingen CJ, Bergman M, Yazici Y. RAPID3 (Routine Assessment of Patient Index Data 3), a rheumatoid arthritis index without formal joint counts for routine care: proposed severity categories compared to disease activity score and clinical disease activity index categories. J Rheumatol 2008; 35 (11) 2136-47.
    CrossrefPubMedGoogle Scholar
  • 5 Yazici Y, Bergman M, Pincus T. Time to score quantitative rheumatoid arthritis measures: 28-Joint Count, Disease Activity Score, Health Assessment Questionnaire (HAQ), Multidimensional HAQ (MDHAQ), and Routine Assessment of Patient Index Data (RAPID) scores. J Rheumatol 2008; 35 (04) 603-9.
    PubMedGoogle Scholar
  • 6 Aletaha D, Nell VP, Stamm T, Uffmann M, Pflugbeil S, Machold K, Smolen JS. Acute phase reactants add little to composite disease activity indices for rheumatoid arthritis: validation of a clinical activity score. Arthritis Res Ther 2005; 07 (04) R796-806.
    CrossrefPubMedGoogle Scholar
  • 7 Blake H. Innovation in practice: mobile phone technology in patient care. Br J Community Nurs 2008; 13 (04) 160 2-5..
    CrossrefPubMedGoogle Scholar
  • 8 Patrick K, Griswold WG, Raab F, Intille SS. Health and the mobile phone. Am J Prev Med 2008; 35 (02) 177-81.
    CrossrefPubMedGoogle Scholar
  • 9 Plaza I, Demarzo MM, Herrera-Mercadal P, Garcia-Campayo J. Mindfulness-based mobile applications: literature review and analysis of current features. JMIR Mhealth Uhealth 2013; 01 (02) e24.
    CrossrefPubMedGoogle Scholar
  • 10 Ozdalga E, Ozdalga A, Ahuja N. The smartphone in medicine: a review of current and potential use among physicians and students. J Med Internet Res 2012; 14 (05) e128.
    CrossrefPubMedGoogle Scholar
  • 11 Mathur P, Southern L, Wang S, Heckler C, Lele O, Embi P, Curtis JR. High Level Architecture and Evaluation of Patient Linkages for READY - An Electronic Measurement Tool for Rheumatoid Arthritis. Proceedings of the American Medical Informatics Association Annual Symposium. 2015
    PubMedGoogle Scholar
  • 12 Curtis JR. RhEumAtic Disease activitY. 2014 Available from: https://itunes.apple.com/us/app/rheumaticdisease-activity/id657411562?mt=8
    PubMedGoogle Scholar
  • 13 Gilbert A, Sebag-Montefiore D, Davidson S, Velikova G. Use of patient-reported outcomes to measure symptoms and health related quality of life in the clinic. Gynecol Oncol 2015; 136 (03) 429-39.
    CrossrefPubMedGoogle Scholar
  • 14 Jensen RE, Rothrock NE, DeWitt EM, Spiegel B, Tucker CA, Crane HM, Forrest CB, Patrick DL, Fredericksen R, Shulman LM, Cella D, Crane PK. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care. Med Care 2015; 53 (02) 153-9.
    CrossrefPubMedGoogle Scholar
  • 15 Schick-Makaroff K, Molzahn A. Strategies to use tablet computers for collection of electronic patient-reported outcomes. Health Qual Life Outcomes 2015; 13: 2.
    CrossrefPubMedGoogle Scholar
  • 16 Zheng K, Haftel HM, Hirschl RB, O’Reilly M, Hanauer DA. Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective. J Am Med Inform Assoc 2010; 17 (04) 454-61.
    CrossrefPubMedGoogle Scholar
  • 17 Bruland P, Forster C, Breil B, Stander S, Dugas M, Fritz F. Does single-source create an added value? Evaluating the impact of introducing x4T into the clinical routine on workflow modifications, data quality and cost-benefit. Int J Med Inform 2014; 83 (12) 915-28.
    CrossrefPubMedGoogle Scholar
  • 18 Koldby S, Schou IJensen. Clinical simulation and workflow by use of two clinical information systems, the electronic health record and digital dictation. Stud Health Technol Inform 2013; 192: 402-6.
    PubMedGoogle Scholar
  • 19 Mensah N, Sukums F, Awine T, Meid A, Williams J, Akweongo P, Kaltschmidt J, Haefeli WE, Blank A. Impact of an electronic clinical decision support system on workflow in antenatal care: the QUALMAT eCDSS in rural health care facilities in Ghana and Tanzania. Glob Health Action 2015; 08: 25756.
    PubMedGoogle Scholar
  • 20 Militello LG, Arbuckle NB, Saleem JJ, Patterson E, Flanagan M, Haggstrom D, Doebbeling BN. Sources of variation in primary care clinical workflow: implications for the design of cognitive support. Health Informatics J 2014; 20 (01) 35-49.
    CrossrefPubMedGoogle Scholar
  • 21 Lewis C. Using the “think aloud” method in cognitive interface design. New York: IBM; 1982
    Google Scholar
  • 22 Holzinger A. Usability engineering methods for software developers. Commun ACM 2005; 48 (01) 71-4.
    CrossrefPubMedGoogle Scholar
  • 23 Cohen T, Kaufman D, White T, Segal G, Staub AB, Patel V, Finnerty M. Cognitive evaluation of an innovative psychiatric clinical knowledge enhancement system. Studies in Health Technology & Informatics 2004; 107 (Pt 2): 1295-9.
    PubMedGoogle Scholar
  • 24 Peute LWP, Jaspers MWM. The significance of a usability evaluation of an emerging laboratory order entry system. International Journal of Medical Informatics 2007; 76 2-3 157-68.
    CrossrefPubMedGoogle Scholar
  • 25 Elhadad N, McKeown K, Kaufman D, Jordan D. Facilitating physicians’ access to information via tailored text summarization. AMIA Annu Symp Proc. 2005. Annual Symposium Proceedings/AMIA Symposium; 226-30.
    Google Scholar
  • 26 Yu H, Lee M, Kaufman D, Ely J, Osheroff JA, Hripcsak G, Cimino J. Development, implementation, and a cognitive evaluation of a definitional question answering system for physicians. Journal of Biomedical Informatics 2007; 40 (03) 236-51.
    CrossrefPubMedGoogle Scholar
  • 27 Horsky J, Kaufman DR, Patel VL. The cognitive complexity of a provider order entry interface. AMIA Annu Symp Proc. 2003. Annual Symposium Proceedings/AMIA Symposium; 294-8.
    Google Scholar
  • 28 Lopetegui M, Yen PY, Lai A, Jeffries J, Embi P, Payne P. Time motion studies in healthcare: what are we talking about?. J Biomed Inform 2014; 49: 292-9.
    CrossrefPubMedGoogle Scholar
  • 29 Medicine NLo. MeSH: Time and Motion Studies. 1991 Available from: http://www.ncbi.nlm.nih.gov/mesh?term=Time+and+Motion+Studies
    PubMedGoogle Scholar
  • 30 Lopetegui M, Yen PY, Lai AM, Embi PJ, Payne PR. Time Capture Tool (TimeCaT): development of a comprehensive application to support data capture for Time Motion Studies. AMIA Annu Symp Proc 2012; 2012: 596-605.
    PubMedGoogle Scholar
  • 31 Chen JW. Developing a process for reducing functional discrepancies. Doctoral dissertation: School of Health Information Sciences, University of Texas Health Sciene Center at Houston. 2008
    PubMedGoogle Scholar
  • 32 van den Brink RH, Troquete NA, Beintema H, Mulder T, van Os TW, Schoevers RA, Wiersma D. Risk assessment by client and case manager for shared decision making in outpatient forensic psychiatry. BMC Psychiatry 2015; 15: 120.
    CrossrefPubMedGoogle Scholar
  • 33 Klingaman EA, Medoff DR, Park SG, Brown CH, Fang L, Dixon LB, Hack SM, Tapscott SL, Walsh MB, Kreyenbuhl JA. Consumer satisfaction with psychiatric services: The role of shared decision making and the therapeutic relationship. Psychiatr Rehabil J 2015; 38 (03) 242-8.
    CrossrefPubMedGoogle Scholar
  • 34 Flynn D, Knoedler MA, Hess EP, Murad MH, Erwin PJ, Montori VM, Thomson RG. Engaging patients in health care decisions in the emergency department through shared decision-making: a systematic review. Acad Emerg Med 2012; 19 (08) 959-67.
    CrossrefPubMedGoogle Scholar
  • 35 Asan OP DS, Montague E. More screen time, less face time - implications for EHR design. J Eval Clin Pract 2014; 20 (06) 896-901.
    CrossrefPubMedGoogle Scholar
  • 36 Patel V, Hale TM, Palakodeti S, Kvedar JC, Jethwani K. Prescription Tablets in the Digital Age: A Cross-Sectional Study Exploring Patient and Physician Attitudes Toward the Use of Tablets for Clinic-Based Personalized Health Care Information Exchange. JMIR Res Protoc 2015; 04 (04) e116.
    CrossrefPubMedGoogle Scholar
  • 37 Asan O, Carayon P, Beasley JW, Montague E. Work system factors influencing physicians’ screen sharing behaviors in primary care encounters. Int J Med Inform 2015; 84 (10) 791-8.
    CrossrefPubMedGoogle Scholar
  • 38 Singman EL, Haberman CV, Appelbaum J, Tian J, Shafer K, Toerper M, Katz S, Kelsay M, Boland MV, Greenbaum M, Adelman R, Thomas RC, Vakili S. Electronic Tracking of Patients in an Outpatient Ophthalmology Clinic to Improve Efficient Flow: A Feasibility Analysis and Benchmarking Study. Qual Manag Health Care 2015; 24 (04) 190-9.
    CrossrefPubMedGoogle Scholar
  • 39 eHealth Observatory and School of Health Information Science University of Victoria. Canada Health Infoway Benefits Evaluation Indicators, technical report. 2012 Available from: https://www.infoway-inforoute.ca/index.php/programs-services/benefits-evaluation
    PubMed
  • 40 Agency for Healthcare Research and Quality. Health IT Evaluation Measures. 2014 Available from: http://healthit.ahrq.gov/health-it-tools-and-resources/health-it-evaluation-toolkit-and-evaluation-measures-quick-reference
    PubMedGoogle Scholar
  • 41 Spetz J, Burgess J, Phibbs C. The effect of health information technology implementation in Veterans Health Administration hospitals on patient outcomes. Healthcare 2014; 02: 40-7.
    CrossrefPubMedGoogle Scholar
  • 42 PROMIS: Dynamic Tools to Measure Health Outcomes from the Patient Perspective. Available from: http://www.nihpromis.org/
    PubMed