Abstract
Objective
This study aims to evaluate the impact of using a large language model (LLM) for generating
draft responses to patient messages in the electronic health record (EHR) system on
clinicians and support staff workload and efficiency.
Methods
We partnered with Epic Systems to implement OpenAI's ChatGPT 4.0 for responding to
patient messages. A pilot study was conducted from August 2023 to July 2024 across
13 ambulatory specialties involving 323 participants, including clinicians and support
staff. Data on draft utilization rates and message response times were collected and
analyzed using statistical methods.
Results
The overall mean generated draft utilization rate was 38%, with significant differences
by role and specialty. Clinicians had a higher utilization rate (43%) than scheduling
staff (33%). Draft message usage significantly reduced all users' message response
time (13 seconds on average). Support staff experienced a more substantial and statistically
significant time saving (23 seconds) compared to negligible time savings seen by clinicians
(3 seconds). Variability in utilization rates and time savings was observed across
different specialties.
Conclusion
Implementing LLMs for drafting patient message replies can reduce response times and
alleviate message burden. However, the effectiveness of artificial intelligence (AI)-generated
draft responses varies by clinical role and specialty, indicating the need for tailored
implementations. Further investigation into this variability, and development and
personalization of AI tools are recommended to maximize their utility and ensure safe
and effective use in diverse clinical contexts.
Keywords
artificial intelligence - clinical information systems - electronic health records
and systems - clinical information systems - burnout - messaging - clinical informatics
