Assessing Electronic Note Quality Using the Physician Documentation Quality Instrument (PDQI-9)

Summary

Objective: To refine the Physician Documentation Quality Instrument (PDQI) and test the validity and reliability of the 9-item version (PDQI-9).

Methods: Three sets each of admission notes, progress notes and discharge summaries were evaluated by two groups of physicians using the PDQI-9 and an overall general assessment: one gold standard group consisting of program or assistant program directors (n = 7), and the other of attending physicians or chief residents (n = 24). The main measures were criterion-related validity (correlation coefficients between Total PDQI-9 scores and 1-item General Impression scores for each note), discriminant validity (comparison of PDQI-9 scores on notes rated as best and worst using 1-item General Impression score), internal consistency reliability (Cronbach’s alpha), and inter-rater reliability (intraclass correlation coefficient (ICC)).

Results: The results were criterion-related validity (r = –0.678 to 0.856), discriminant validity (best versus worst note, t = 9.3, p = 0.003), internal consistency reliability (Cronbach’s alphas = 0.87–0.94), and inter-rater reliability (ICC = 0.83, CI = 0.72–0.91).

Conclusion: The results support the criterion-related and discriminant validity, internal consistency reliability, and inter-rater reliability of the PDQI-9 for rating the quality of electronic physician notes. Tools for assessing note redundancy are required to complement use of PDQI-9. Trials of the PDQI-9 at other institutions, of different size, using different EHRs, and incorporating additional physician specialties and notes of other healthcare providers are needed to confirm its generaliz-ability.

Keywords

Electronic health record - documentation - note - quality - instrument

• References

• 1 DesRoches CM, Campbell EG, Rao SR, Donelan K, Ferris TG, Jha A. et al. Electronic health records in ambulatory care –a national survey of physicians. N Engl J Med 2008; 359 (01) 50-60.
• 2 Medicare and Medicaid programs; electronic health record incentive program.. Final rule. Fed Regist 2010; 75 (144) 44313-44588.
• 3 Amarasingham R, Plantinga L, Diener-West M, Gaskin DJ, Powe NR. Clinical information technologies and inpatient outcomes: a multiple hospital study. Arch Intern Med 2009; 169 (02) 108-114.
• 4 Poon EG, Wright A, Simon SR, Jenter CA, Kaushal R, Volk LA. et al. Relationship between use of electronic health record features and health care quality: results of a statewide survey. Med Care 2010; 48 (03) 203-209.
• 5 Johnson SB, Bakken S, Dine D, Hyun S, Mendonca E, Morrison F. et al. An electronic health record based on structured narrative. J Am Med Inform Assoc 2008; 15 (01) 54-64.
• 6 Kirkland LR, Bryan CS. Osler’s service: a view of the charts. J Med Biogr 2007; 15 (Suppl. 01) 50-54.
• 7 Weed LL. The problem oriented record as a basic tool in medical education, patient care and clinical research. Ann Clin Res 1971; 3 (03) 131-134.
• 8 Fries JF. Alternatives in medical record formats. Med Care 1974; 12 (10) 871-881.
• 9 Reiser SJ. The clinical record in medicine. Part 2: Reforming content and purpose. Ann Intern Med 1991; 114 (11) 980-985.
• 10 Burnum JF. The misinformation era: the fall of the medical record. Ann Intern Med 1989; 110 (06) 482-484.
• 11 Embi PJ, Yackel TR, Logan JR, Bowen JL, Cooney TG, Gorman PN. Impacts of computerized physician documentation in a teaching hospital: perceptions of faculty and resident physicians. J Am Med Inform Assoc 2004; 11 (04) 300-309.
• 12 Hartzband P, Groopman J. Off the record –avoiding the pitfalls of going electronic. N Engl J Med 2008; 358 (16) 1656-1658.
• 13 Hirschtick RE. A piece of my mind. Copy-and-paste. Jama 2006; 295 (20) 2335-2336.
• 14 Siegler EL, Adelman R. Copy and paste: a remediable hazard of electronic health records. Am J Med 2009; 122 (06) 495-496.
• 15 Thielke S, Hammond K, Helbig S. Copying and pasting of examinations within the electronic medical record. Int J Med Inform 2007; 76 (Suppl. 01) S122-S128.
• 16 Weir CR, Hurdle JF, Felgar MA, Hoffman JM, Roth B, Nebeker JR. Direct text entry in electronic progress notes. An evaluation of input errors. Methods Inf Med 2003; 42 (01) 61-67.
• 17 O’Donnell HC, Kaushal R, Barron Y, Callahan MA, Adelman RD, Siegler EL. Physicians’ attitudes towards copy and pasting in electronic note writing. Journal of general internal medicine: official journal of the Society for Research and Education in Primary Care Internal Medicine 2009; 24 (01) 63-68.
• 18 Gelzer R, Hall T, Liette E, Reeves MG, Sundby J, Tegen A. et al. Auditing copy and paste. J AHIMA 2009; Jan 80 (01) 26-9 quiz 31-2.
• 19 Payne T, Patel R, Beahan S, Zehner J. The Physical Attractiveness of Electronic Physician Notes. AMIA Annu Symp Proc 2010: 622-626.
• 20 Institute of Medicine.. Committee on Improving the Patient R Dick RS, Steen EB, Detmer DE. The computer-based patient record : an essential technology for health care. Washington, D. C.: National Academy Press; 1997
• 21 Romm FJ, Putnam SM. The validity of the medical record. Med Care 1981; 19 (03) 310-315.
• 22 Tufo HM, Speidel JJ. Problems with Medical Records. Medical care 1971; 9 (06) 509-517.
• 23 Rosenbloom ST, Crow AN, Blackford JU, Johnson KB. Cognitive factors influencing perceptions of clinical documentation tools. J Biomed Inform 2007; 40 (02) 106-113.
• 24 Stetson PD, Morrison FP, Bakken S, Johnson SB. Preliminary development of the physician documentation quality instrument. J Am Med Inform Assoc 2008; 15 (04) 534-541.
• 25 Wrenn JO, Stein DM, Bakken S, Stetson PD. Quantifying clinical narrative redundancy in an electronic health record. J Am Med Inform Assoc 2010; 17 (01) 49-53.
• 26 Aronsky D, Haug PJ. Assessing the quality of clinical data in a computer-based record for calculating the pneumonia severity index. J Am Med Inform Assoc 2000; 7 (01) 55-65.
• 27 Coakley FV, Heinze SB, Shadbolt CL, Schwartz LH, Ginsberg MS, Lefkowitz RA. et al. Routine editing of trainee-generated radiology reports: effect on style quality. Acad Radiol 2003; 10 (03) 289-294.
• 28 Efthimiadis EN, Hammond KW, Laundry R, Thielke SM. Developing an EMR simulator to assess users’ perception of document quality. Proc 43rd Hawaii Int Conf on System Sciences –2010 2010: 1-9.
• 29 Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc 1997; 4 (05) 342-355.
• 30 Logan JR, Gorman PN, Middleton B. Measuring the quality of medical records: a method for comparing completeness and correctness of clinical encounter data. Proc AMIA Symp 2001: 408-412.
• 31 Myers KA, Keely EJ, Dojeiji S, Norman GR. Development of a rating scale to evaluate written communication skills of residents. Acad Med 1999; 74 (Suppl. 10) S111-S113.
• 32 Hammond KW, Efthimiadis EN, Weir CR, Embi PJ, Thielke SM, Laundry RM. et al. Initial steps toward validating and measuring the quality of computerized provider documentation. AMIA Annu Symp Proc 2010 2010: 271-275.
• 33 Bates DW. Getting in step: electronic health records and their role in care coordination. J Gen Intern Med 2010; 25 (03) 174-176.
• 34 Bates DW, Bitton A. The future of health information technology in the patient-centered medical home. Health Aff (Millwood) 2010; 29 (04) 614-621.
• 35 O’Malley AS, Grossman JM, Cohen GR, Kemper NM, Pham HH. Are electronic medical records helpful for care coordination? Experiences of physician practices. J Gen Intern Med 2010; 25 (03) 177-185.
• 36 Naik AD, Singh H. Electronic health records to coordinate decision making for complex patients: what can we learn from wiki?. Med Decis Making 2010; 30 (06) 722-731.
• 37 Meaningful Use Workgroup Request for Comments Regarding Meaningful Use Stage 2.. Health Information Technology Policy Committee; 2011; Available from: http://healthit.hhs.gov/media/faca/ MU_RFC%20_2011–01–12_final.pdf.
• 38 Hammond KW, Helbig ST, Benson CC, Brathwaite-Sketoe BM. Are electronic medical records trustworthy? Observations on copying, pasting and duplication. AMIA Annu Symp Proc 2003: 269-273.