Leveraging a Large Language Model for Streamlined Medical Record Generation: Implications for Health Care Informatics

Abstract

Objectives

This study aimed to leverage a large language model (LLM) to improve the efficiency and thoroughness of medical record documentation. This study focused on aiding clinical staff in creating structured summaries with the help of an LLM and assessing the quality of these artificial intelligence (AI)-proposed records in comparison to those produced by doctors.

Methods

This strategy involved assembling a team of specialists, including data engineers, physicians, and medical information experts, to develop guidelines for medical summaries produced by an LLM (Llama 3.1), all under the direction of policymakers at the study hospital. The LLM proposes admission, weekly summaries, and discharge notes for physicians to review and edit. A validated Physician Documentation Quality Instrument (PDQI-9) was used to compare the quality of physician-authored and LLM-generated medical records.

Results

The results showed no significant difference was observed in the total PDQI-9 scores between the physician-drafted and AI-created weekly summaries and discharge notes (p = 0.129 and 0.873, respectively). However, there was a significant difference in the total PDQI-9 scores between the physician and AI admission notes (p = 0.004). Furthermore, there were significant differences in item levels between physicians' and AI notes. After deploying the note-assisted function in our hospital, it gradually gained popularity.

Conclusion

LLM shows considerable promise for enhancing the efficiency and quality of medical record summaries. For the successful integration of LLM-assisted documentation, regular quality assessments, continuous support, and training are essential. Implementing LLM can allow clinical staff to concentrate on more valuable tasks, potentially enhancing overall health care delivery.

Keywords

large language model (LLM) - medical records - Physician Documentation Quality Instrument - electronic medical records (EMRs) - admission notes - discharge notes - weekly summaries

Protection of Human and Animal Subjects

This study was performed in compliance with the World Medical Association Declaration of Helsinki on ethical principles for medical research involving human subjects and was reviewed by the Institutional Review Board of Taichung Veterans General Hospital (approval number: CE24503B). Informed consent was obtained from all the participants.

Note

The authors, confirm that all figures presented in this work have been fully anonymized. They do not contain any information that could be used to identify individual patients, including but not limited to names, initials, dates, medical record numbers, or institutional identifiers. Furthermore, no third-party copyrighted material has been included in the figures.

Publication History

Received: 16 April 2025

Accepted: 22 September 2025

Accepted Manuscript online:
25 September 2025

Article published online:
29 October 2025

• References

• 1 de Hoop T, Neumuth T. Evaluating electronic health record limitations and time expenditure in a german medical center. Appl Clin Inform 2021; 12 (05) 1082-1090
• 2 Tai-Seale M, Baxter S, Millen M. et al. Association of physician burnout with perceived EHR work stress and potentially actionable factors. J Am Med Inform Assoc 2023; 30 (10) 1665-1672
• 3 Li C, Parpia C, Sriharan A, Keefe DT. Electronic medical record-related burnout in healthcare providers: a scoping review of outcomes and interventions. BMJ Open 2022; 12 (08) e060865
• 4 Lee S, Kim HS. Prospect of artificial intelligence based on electronic medical record. J Lipid Atheroscler 2021; 10 (03) 282-290
• 5 Konishi S, Manabe S, Shimai Y. et al. Implementation of a multilingual electronic medical questionnaire: for use inside or outside the hospital. Stud Health Technol Inform 2024; 310: 1360-1361
• 6 Khanna P, Dhillon G, Buddhavarapu V, Verma R, Kashyap R, Grewal H. Artificial intelligence in multilingual interpretation and radiology assessment for clinical language evaluation (AI-MIRACLE). J Pers Med 2024; 14 (09) 923
• 7 Walker KJ, Wang A, Dunlop W, Rodda H, Ben-Meir M, Staples M. The 9-item physician documentation quality instrument (PDQI-9) score is not useful in evaluating EMR (scribe) note quality in emergency medicine. Appl Clin Inform 2017; 8 (03) 981-993
• 8 Kernberg A, Gold JA, Mohan V. Using ChatGPT-4 to create structured medical notes from audio recordings of physician-patient encounters: comparative study. J Med Internet Res 2024; 26: e54419
• 9 Chi EA, Chi G, Tsui CT. et al. Development and validation of an artificial intelligence system to optimize clinician review of patient records. JAMA Netw Open 2021; 4 (07) e2117391
• 10 Kim J, Lee S, Jeon H. et al. PhenoFlow: a human-LLM driven visual analytics system for exploring large and complex stroke datasets. IEEE Trans Vis Comput Graph 2025; 31 (01) 470-480
• 11 Wiest IC, Wolf F, Lessmann ME. et al. LLM-AIx: an open source pipeline for Information Extraction from unstructured medical text based on privacy preserving large language models. medRxiv 2024
• 12 Kaur A, Budko A, Liu K, Eaton E, Steitz BD, Johnson KB. Automating responses to patient portal messages using generative AI. Appl Clin Inform 2025; 16 (03) 718-731
• 13 Proctor S, Lawton G, Sinha S. An AI-powered strategy for managing patient messaging load and reducing burnout. Appl Clin Inform 2025; 16 (04) 747-752
• 14 Karabacak M, Margetis K. Embracing large language models for medical applications: opportunities and challenges. Cureus 2023; 15 (05) e39305
• 15 Tripathi S, Sukumaran R, Cook TS. Efficient healthcare with large language models: optimizing clinical workflow and enhancing patient care. J Am Med Inform Assoc 2024; 31 (06) 1436-1440
• 16 Baker HP, Dwyer E, Kalidoss S, Hynes K, Wolf J, Strelzow JA. ChatGPT's ability to assist with clinical documentation: a randomized controlled trial. J Am Acad Orthop Surg 2024; 32 (03) 123-129
• 17 Klang E, Gill J, Sharma A. et al. Summarize-then-prompt: a novel prompt engineering strategy for generating high-quality discharge summaries. Appl Clin Inform 2025;