Keywords
electronic health records - audit logs - ophthalmology residents - call burden - clinical
activity - EHR tasks - education policy
The rate of electronic health record (EHR) adoption now exceeds 80% among ophthalmologists
nationwide.[1] Mandated in 2014 as part of the Health Insurance Portability and Accountability
Act (HIPAA), EHR audit logs provide researchers with abundant, large-scale data on
how users interact with their EHR system. These audit logs typically include time-stamped
data of users' activities, including date, time, and location of log-ins, which patients'
records were accessed, and what activities were performed. While originally intended
for audit reasons to prevent inappropriate access of patient records, in recent years
these logs have served as a rich source of data for understanding clinical workflows.[2]
[3]
[4] Previous studies have analyzed these data to characterize the activities of attending
physicians, residents, and medical students alike.[5]
[6]
[7]
[8]
Audit log data describe physicians' EHR use in granular detail and can be used to
approximate work burden. Studies have shown that increased EHR use, particularly after-hours,
is a key contributor to burnout for practicing physicians.[9]
[10]
[11] A 2015 nationwide study across all specialties documented that approximately 50%
of physicians experience some form of burnout.[12] Burnout rates may be even higher among residents, and prior studies have shown ophthalmology
residents to have especially high rates of burnout.[13]
[14]
[15]
[16]
[17] Although the Accreditation Council for Graduate Medical Education (ACGME) work hour
restrictions implemented in 2003 and 2011 have been shown to decrease emotional fatigue
and overall burnout rates, a significant number of residents across the country continue
to report feelings of exhaustion and burnout.[18] With this in mind, previous work has characterized resident workflow and time allocation,
particular after-hours while on call.[19]
[20]
[21]
[22]
[23] Although these studies span multiple specialties including internal medicine, otolaryngology,
and vascular surgery, to our knowledge there have not been any published studies that
characterize the activities of ophthalmology residents while on call outside of regular
clinic hours. Furthermore, the majority of these prior studies relied on trained observers
to record residents' actions, an approach that can be influenced by the Hawthorne
effect (i.e., changing one's behavior due to the knowledge of being actively observed)
and by variations in data collection and quality between different observers.
In this study, we aimed to objectively characterize the after-hours activities of
ophthalmology residents while taking primary call using EHR audit log data. Examining
patterns of EHR use among residents while on call may help inform education policies
regarding call and influence strategies aimed at addressing physician burnout in the
future.
Methods
Study Design and Population
This retrospective study was conducted at the University of California San Diego (UCSD).
The study was approved by the UCSD Institutional Review Board and adhered to the tenets
of the Declaration of Helsinki. Raw EHR audit logs were collected over a 12-month
period from October 1, 2018, through September 30, 2019. October 2018 was designated
as the start of the study period because the ophthalmology department implemented
the enterprise-wide EHR (Epic Systems, Verona, WI) in late September 2018, after which
all residents were asked to document all on-call interactions in the EHR. Previously,
some on-call interactions (primarily those conducted with clinic patients) were documented
on paper charts. However, the hospitals and emergency departments of the health system
had implemented the EHR in 2010, and therefore even before the ophthalmology department
implemented the EHR for ambulatory clinic encounters, ophthalmology residents had
been accustomed to documenting on-call encounters from inpatient and emergency consults
in the same system.
Per existing policies, first- and second-year ophthalmology residents were assigned
to rotating primary call shifts, with third-year ophthalmology residents taking “back-up”
call to help the primary call residents with any questions, typically answering questions
over the phone and having limited interactions with patients and the EHR. Subsequently,
for this study, we included data from only the first- and second-year residents taking
primary call shifts, as they were the ones primarily interacting with patients and
documenting in the EHR while on call. However, because the study period spanned portions
of two academic cycles, a total of three classes of ophthalmology residents were included
in the analysis. With four residents per class, this equated to 12 residents. PGY
(postgraduate year) level was denoted by training year at the end of the study, as
the study period spanned two academic year cycles.
Residents at this program also rotated and took call at the San Diego Veterans Affairs
(VA) Healthcare System. However, consultation with the clinical informatics team revealed
that granular EHR audit logs were not available at the time of this study. Therefore,
EHR data from call activities at the VA could not be included.
EHR Audit Log Data Extraction and Analysis
Raw EHR audit logs were obtained for the eligible residents from the designated study
period from the UCSD Clinical Data Warehouse. Logs were compared with resident schedules
to isolate data gathered during resident call shifts. Data were analyzed based on
the type of call shifts. Shifts were categorized as follows: weekday evenings (4 pm
to midnight), weekday overnights (midnight to 8 a.m.), weekends (Saturdays and Sundays
8 a.m. to 8 a.m. the next day), and holidays (24-hour periods on select days throughout
the year based on existing definitions by the university). Although in practice, on
weekdays, a single resident would take primary call from 4 p.m. until the following
morning at 8 a.m., for the purpose of the analysis, this was split into two portions
(“evenings” and “overnights”) to characterize “middle-of-the-night” call activities
(i.e., those after midnight) specifically, as these would be more disruptive to residents'
quality of life and impose more burden through disruption of sleep cycles, which has
been shown to be a key contributor to physician burnout.[24]
[25]
[26] Furthermore, a prior analysis in the internal medicine literature demonstrated substantial
variations in clinical volumes and workload during nondaytime shifts, prompting the
creation of a “swing shift” to accommodate the busiest hours of call between 4 p.m.
and 11 p.m.[27] This provided further motivation to examine segments of weekday call shifts separately.
Data collected from EHR logs included login time, logout time, number of unique patient
charts accessed, primary encounter diagnosis codes, and EHR tasks performed. Total
time spent logged in per shift and the number of unique patient charts accessed (clinical
volume) were calculated for each shift type. Due to variations in call shift length,
time spent logged in the EHR and clinical volume were also standardized to per-hour
metrics by dividing the EHR use time or clinical volume during the shift by the number
of hours in the shift (8 hours for weekday evenings and weekday overnights, 24 hours
for weekends and holidays). The most common diagnoses seen on call were tabulated
using patients' primary encounter diagnosis codes listed in the EHR. Diagnoses were
categorized into groups based on ophthalmic or general medicine/nonophthalmic diagnoses.
All ophthalmic diagnoses were further grouped into categories such as general ophthalmology,
plastics, pre-/postoperative, glaucoma, and so on. Of note, primary encounter diagnosis
codes were generated by the primary provider for the overall encounter (i.e., admission
diagnosis determined by an emergency physician or hospitalist) not the ophthalmology
consultation specifically. Some diagnoses such as “null” and “other” were unable to
be categorized and were excluded from the study.
Finally, to characterize residents' specific activities in the EHR, we extracted task
descriptions provided by the EHR vendor that were recorded in the audit logs. Each
specific task performed by a resident (such as viewing a note, opening a visit navigator,
or looking up a specific part of a patient's chart) was recorded by the EHR system.
We categorized these tasks into broader domains such as chart review, login/patient
searching, documentation, and so on. Some metrics such as “masked data displayed,”
“potential duplicates checked,” and “other” were unable to be categorized and were
excluded from the study.
Statistical Analyses
For demographics and all outcomes of interest (EHR use time, clinical volume, diagnoses,
EHR activities), descriptive statistics were generated using mean and standard deviation,
median and interquartile range (IQR), or counts/proportions where appropriate. Linear
models of EHR use time and clinical volume were used to evaluate differences between
shift types. To evaluate trends in EHR use time and clinical volume longitudinally
across the study period, data were aggregated and compared by monthly averages across
all call shifts. Chi-square tests of goodness-of-fit were used to assess whether significant
differences existed across months. To assess the relationship between clinical volume
and EHR use time, we constructed a scatterplot between the two variables and fit a
linear regression model to evaluate for a statistically significant correlation and
to determine the R
2 correlation coefficient. For all statistical analyses, significance was defined as
p < 0.05. All analyses were performed using R version 3.5.1 (R Foundation for Statistical
Computing, Vienna, Austria).[28]
Results
Participants and Shift Characteristics
EHR audit log data were collected for 12 ophthalmology residents taking primary call
during the study period (Table 1). Based on self-report, the majority were female
(8/12 [67%]). Eight (67%) were Asian and four (33%) were white. No residents self-reported
identifying as Hispanic or Latino. Residents were evenly distributed by training year
(four residents in each of three PGY training levels). Over the course of the year-long
study, EHR data from 638 call shifts were obtained. Most call shifts corresponded
to weekday/weekend designations as previously described. There were 15 (2%) holiday
shifts based on existing definitions of university holidays.
Table 1
Demographics of ophthalmology residents (n = 12) whose electronic health record audit logs were analyzed from October 1, 2018,
to September 30, 2019
|
Characteristics
|
N (% out of total n = 12)
|
|
Gender
|
|
Female
|
8 (67)
|
|
Male
|
4 (33)
|
|
Race
|
|
Asian
|
8 (67)
|
|
Caucasian
|
4 (33)
|
|
PGY level[a]
|
|
PGY-2
|
4 (33)
|
|
PGY-3
|
4 (33)
|
|
PGY-4
|
4 (33)
|
Abbreviation: PGY, postgraduate year.
a PGY level denoted by training year at the end of the study (September 30, 2019),
as the study period spanned two academic year cycles.
Electronic Health Record Use Time
Across all call shifts, the median (IQR) time spent logged into the EHR per shift
was 88 (131) minutes. When standardized to per-hour measures, ophthalmology residents
spent the most time using the EHR on weekday evening shifts (4 p.m. to midnight),
with a median (IQR) of 12.1 (11.3) minutes per hour, followed closely by holidays
(10.5 [5.4] minutes per hour; [Table 2]). Weekday evenings entailed significantly more EHR use time than weekends and weekday
overnights (p < 0.001). Although weekday overnights between midnight and 8 a.m. entailed the least
amount of EHR use for ophthalmology residents on a per-hour basis (4.75 [7.2]) compared
with other types of call shifts, cumulatively there was still a broad range of EHR
use during these middle-of-the-night hours, with median (IQR) of 38 (57.8) minutes
across these shifts as a whole.
Table 2
Time and clinical volume associated with after-hours EHR use for ophthalmology residents
for various call shifts over a 1-year period (October 1, 2018, to September 30, 2019)
|
Type of call shift
|
Median (IQR) number of minutes logged into the EHR
|
Median (IQR) number of unique patient charts accessed
|
|
Across entire call shift
|
Standardized to per-hour basis[a]
|
p-Value[b]
|
Across entire call shift
|
Standardized to per-hour basis[a]
|
p-Value[b]
|
|
Weekday evenings (4 p.m. to midnight)
|
97 (90.0)
|
12.1 (11.3)
|
–
|
8.0 (6.0)
|
1.0 (0.8)
|
–
|
|
Weekday overnights (midnight to 8 a.m.)
|
38 (57.8)
|
4.75 (7.2)
|
<0.001
|
4.0 (3.0)
|
0.5 (0.4)
|
<0.001
|
|
Weekends (Saturdays and Sundays 8 a.m. to 8 a.m. the next day)
|
201 (213.5)
|
8.4 (8.9)
|
<0.001
|
14.5 (10.3)
|
0.6 (0.4)
|
<0.001
|
|
Holidays (24-h periods on select days)
|
253 (129.5)
|
10.5 (5.4)
|
0.23
|
17.0 (10.5)
|
0.7 (0.4)
|
0.014
|
Abbreviations: EHR, electronic health record; IR, interquartile range.
a Values standardized to per-hour basis to allow cross-comparison across call shifts
of different durations.
b
p-Values determined by linear models of the outcome with the type of call shift as
the covariate. p-Values are based on differences between each shift type and weekday evenings (the
reference shift type).
The total time spent logged in was greatest during the summer and fall months (August
to December) and decreased during the spring months (downward trend between December
and June; [Fig. 1A]). This seasonal variation did not reach statistical significance (p = 0.21).
Fig. 1 Longitudinal view of (A) time spent logged into the EHR per call shift and (B) number of unique patient charts accessed per call shift (October 2018 to September
2019). EHR, electronic health record.
Clinical Volume
The total number of patient charts accessed by on-call residents during the study
period was 6,303. The median (IQR) unique patient charts accessed per shift was 7
(9) patients, regardless of type of call shift. Similar to the patterns of time spent
logged into the EHR, on average ophthalmology residents encountered the highest clinical
volumes during weekday evening call shifts, with median (IQR) of 1.0 (0.8) patients
per hour followed by holidays (0.7 [0.4]; [Table 2]). The number of patient charts accessed during these shifts was significantly more
than weekday overnights and weekends (p < 0.001). The number of unique patient charts accessed on call was also higher during
the summer months compared with later parts of the academic year ([Fig. 1B]), although again this did not quite reach statistical significance (p = 0.16). Increased clinical volume was positively correlated with EHR use time ([Fig. 2]). Residents who accessed a greater number of patient charts during a call shift
had a significantly longer duration of time logged into the EHR (p < 0.001; R
2 = 0.51).
Fig. 2 Correlation between clinical volume (X-axis) and EHR use time (Y-axis). EHR, electronic
health record.
Diagnoses and Electronic Health Record Activities
A total of 2,272 diagnoses were associated with the patients seen by ophthalmology
residents on call. Of these, 1,611 (70.9%) were eye-related, 332 (14.6%) were postoperative,
and 329 -(14.5%) were not eye-related. The top three diagnoses were “screening for
eye condition” (285/2,272 [12.5%]), “postoperative state” (271/2,272 [11.9%), and
dermatochalasis (110/2,272 [4.8%]). It is likely that patients carrying a diagnosis
of “dermatochalasis” were actually status post blepharoplasty and were simply not
characterized as “postoperative state” in the system, given that physicians often
vary in assignment of diagnosis codes. Diagnoses were grouped based on field and further
subdivided into categories, as seen in [Table 3]. The ophthalmic categories with the greatest number of diagnoses were plastics (427/2,272
[18.8%]), general ophthalmology/anterior segment (351/2,272 [15.4%]), and pre-/postoperative
(332/2,272 [14.6%]).
Table 3
Categorization of primary encounter diagnosis codes for cases seen on call by ophthalmology
residents
|
Diagnosis category
|
Example diagnoses
|
Number (total N = 2,272)
|
%
|
|
Ophthalmic
|
Plastics
|
Dermatochalasis thyroid eye disease, eyelid lesion
|
427
|
18.8
|
|
General/nonspecific
|
Screening for eye condition, visual disturbance, eye pain
|
351
|
15.4
|
|
Glaucoma
|
Glaucoma suspect, primary open-angle glaucoma, congenital glaucoma
|
276
|
12.1
|
|
Cataracts
|
Lens replaced, cataract, nuclear sclerotic cataract
|
162
|
7.1
|
|
Retina/uveitis
|
Retinal detachment, uveitis, diabetic retinopathy
|
123
|
5.4
|
|
Trauma
|
Assault, fall, trauma, ruptured globe
|
87
|
3.8
|
|
Infectious/inflammatory
|
Orbital cellulitis, chalazion, preseptal cellulitis, corneal ulcer
|
81
|
3.6
|
|
Cornea/refractive
|
Corneal abrasion, keratoconus, myopia
|
60
|
2.6
|
|
Ocular surface issues
|
Dry eye syndrome, allergic conjunctivitis, conjunctivitis
|
44
|
1.9
|
|
Postoperative
|
Pre-/postoperative
|
Postoperative state, Preoperative evaluation, H/O eye surgery
|
332
|
14.6
|
|
Nonophthalmic
|
Cardiac/vascular
|
Aortic valve replacement, essential hypertension, hyperlipidemia
|
132
|
5.8
|
|
Infectious
|
Sepsis, HIV disease, cellulitis
|
60
|
2.6
|
|
Oncology
|
Skin neoplasm, chronic lymphocytic leukemia, metastatic melanoma
|
61
|
2.7
|
|
Neuro/psychiatric
|
Headache, altered mental status, seizure
|
42
|
1.8
|
|
General
|
Routine medical examination, diarrhea
|
34
|
1.5
|
Abbreviation: HIV, human immunodeficiency virus.
EHR activities were tabulated in a similar fashion. A total of 104,682 individual
tasks were recorded in EHR audit logs for ophthalmology residents on call during the
study period. The top three task descriptions (as listed by the EHR vendor) were “notes
viewed” (14,079/104,682 [13.4%]), “report with patient data viewed” (13,549/104,682
[12.9%]), and “visit navigator template” (5,634/104,682 [5.4%]). We grouped tasks
based on activity domains, as seen in [Table 4]. The most common categories of EHR activities were chart review (66,372/104,682
[63.4%]), login/patient searching (17,230/104,682, 16.5%), and documentation (11,719/104,682
[11.2%]).
Table 4
Categorization of ophthalmology resident tasks performed within the EHR system while
on-call
|
Task category
|
Top examples
|
Number
(total N = 104,682)
|
%
|
|
Chart review
|
Notes viewed, patient data report viewed, encounter viewed in chart review
|
66,372
|
63.4
|
|
Login, patient searching
|
User authenticated, patient selected from patient lookup, patient lookup search
|
17,230
|
16.5
|
|
Documentation
|
Visit navigator template, sign clinical note, eye exam saved, diagnosis updated
|
11,719
|
11.2
|
|
Inbox/communication
|
In basket message created, in basket message viewed, communication management accessed
|
4,962
|
4.7
|
|
Orders
|
Orders viewed, order entry accessed, open inpatient order set
|
3,876
|
3.7
|
|
Administrative
|
Patient data report printed, patient emergency contacts accessed, order printed from
chart review
|
523
|
0.5
|
Abbreviation: EHR, electronic health record.
Discussion
Based on a broad-based literature search of several databases (PubMed, Embase, and
Scopus), to our knowledge this study represents the first analysis of work patterns
of ophthalmology residents while on call using EHR audit log data. Our key findings
were as follows: (1) EHR use time and clinical volume varied based on the type of
call shift and season, (2) chart review comprised a majority of ophthalmology residents'
on-call EHR activities, and (3) EHR audit logs demonstrated substantial call burden
for ophthalmology residents outside of regular clinic hours. Overall, EHR audit logs
represent an emerging data source for improving understanding educational experience
and work hours, and informing future policies and interventions to mitigate resident
burnout. As detailed in this study, residency programs can potentially utilize these
data to better understand their residents' training experience.
First, we found variations in EHR use time and clinical volume between different call
shift types. When standardized to per-hour metrics, weekday evenings were the busiest
call shifts, with a higher number of patients and longer EHR use time compared with
other types of call shifts. Although weekends and holidays were both 24-hour shifts,
holidays were busier based on EHR use time and on the number of patient charts. Program
directors could potentially integrate these types of data from their individual institutions
when formulating call schedules. Currently, call shifts are generally assigned to
achieve an equal number of days to each resident within various categories (e.g.,
weeknights, weekends, holidays). However, there may be additional sources of variation,
for example, certain holidays may be consistently busier than others. Historical data
regarding EHR use time and patient volume could therefore refine future scheduling
efforts. Additionally, similar to previous studies, the number of patients seen was
highest during the summer months and lowest in the spring.[6] Although the differences did not quite reach statistical significance, there were
clearly trends of seasonal variation. This likely reflects academic cycles, as new
interns and residents start in the summer months and are likely to call more ophthalmology
consults. Time spent logged in also followed a similar pattern. This may have resulted
from an increased workload as described previously. However, for new PGY-2 residents,
greater time spent logged in at the beginning of the academic year may have also reflected
learning a new EHR system and learning how to evaluate ophthalmology patients for
the first time. The log data alone were not sufficient to understand the differential
effects of these two learning processes.
In terms of activities performed in the EHR, our study estimates that more than 60%
of ophthalmology residents' EHR tasks on call were geared toward chart review. This
supports prior studies across multiple specialties, which have reported a similar
disproportionate amount of EHR activity geared toward medical chart review during
regular work hours.[6]
[7]
[29]
[30] Interestingly, this contrasts studies of daytime activities of ophthalmology attending
physicians, where audit log analyses have shown that the number and percentage of
notes reviewed were very low.[31] The authors of that study found that ancillary staff accessed significantly more
notes than ophthalmologists. While they contended that ophthalmology attending physicians
were not using the vast majority of content in the EHR during daytime outpatient encounters,
here we found that ophthalmology residents are still dedicating the majority of their
on-call EHR activity to chart review, likely by necessity as they are often encountering
new patients with acute problems and are typically not supported by ancillary staff.
This information is valuable to inform future EHR training sessions for ophthalmology
residents, where more efficient chart search functions and navigation features could
be emphasized. Thus, using EHR audit log data to characterize usage patterns can facilitate
more targeted training for residents, who may have different needs than attending
physicians. Additionally, these results could be used to improve patient data visualization
and chart summation tools by identify common resident EHR activities. Prior studies
have found that training quality has a significant impact on perceived efficiency
and EHR satisfaction,[32] and therefore these strategies could improve residents' experience.
The data reflect the breadth of educational exposure of these residents, as well as
substantial work activities after-hours while on call. Ophthalmology residents accessed
patients with a wide range of diagnoses. Combining the median values for weekday evening
(4 p.m. to midnight) and overnight (midnight to 8 a.m.), ophthalmology residents accessed
a median of 12 patient charts on weekday calls between two regularly scheduled workdays.
These patients are in addition to patients accessed during busy daytime rotations.
To provide context, studies have estimated that internal medicine residents view an
average of 14 unique charts per day.[29] Although a few ophthalmology residency programs have “eye emergency departments”
with dedicated shifts (meaning the resident is “in house” during the call shift and
then does not work the next day), the vast majority of ophthalmology residency programs
have “home call” policies, where residents travel between clinical sites and their
homes to see ophthalmology consultations and are often expected to work full days
both before and after the call shift. Although weekday overnight shifts (midnight
to 8 a.m.) were the least busy in terms of EHR use time and the number of patient
charts accessed, ophthalmology residents still accessed a median of four charts during
those middle-of-the-night hours, during which work tends to be the most disruptive
to sleep–wake cycles. Weekends and holidays also imposed substantial work burden.
Our data have some limitations. They underestimated the actual time spent interacting
with the EHR and the time the residents spent working on call for several reasons.
First, one hospital in which residents take primary call in this program was excluded
from the study. This hospital used a different EHR system from which comparable data
could not be obtained. Furthermore, this study did not capture all residents' on-call
activities, such as performing surgeries or procedures, patient encounters when the
resident was interacting with patients but not actively logged into the EHR, or time
spent on telephone calls speaking with patients. Lastly, we recognize there are several
additional factors contributing to resident call experience, including length of call
shift and time spent driving between clinical sites (not an insignificant burden given
that the distance between the university eye clinic and the level 1 trauma center
exceeded 13 miles). Therefore, the true work burden on call exceeds what is represented
here, although the log data at least provide some representation of the minimum. In
addition, this study was limited to analysis of EHR use time while residents were
taking primary call and did not include after-hours EHR use during noncall periods.
Given a prior study showing significant after-hours EHR use by ophthalmology attending
physicians after regularly scheduled clinics,[33] it is possible that residents may have incurred similar after-hours EHR use on noncall
days that was not represented here. However, our observations during prior time-motion
studies of ophthalmology residents[34] demonstrated that in faculty clinics, residents generally completed documentation
during the clinic prior to the faculty attending ophthalmologist initiating their
interaction with the patient. However, in resident-run clinics where residents serve
as primary providers rather than in supporting roles (such as at the VA), there may
be after-hours burden of EHR documentation even when not on call. Finally, this was
also a single-center analysis examining residents within a single program. In future
studies, it will be interesting to compare variation in call burden and clinical exposure
between different hospitals and regions of the country. Larger sample sizes of residents
may also enable adequate power to evaluate differences in time expenditures and EHR
tasks across different demographic groups based on gender, race, and/or ethnicity.
In summary, this study leverages EHR audit log data to describe ophthalmology residents'
activities while on call. This is a more objective approach that can supplement resident
self-report, which is the current standard. Using EHR audit log data for understanding
residents' experiences may have broader implications in the fields of physician burnout
and education policy. As evidenced previously, these methods can be used to roughly
estimate work hours and educational exposure of resident physicians. There have been
several studies focused on addressing these goals in the field of internal medicine,[35]
[36]
[37] but ongoing work is needed to apply these methods to surgical specialties such as
ophthalmology, where programs are smaller and residents often take home call to cover
multiple clinical sites. Other foreseeable applications of EHR audit log data could
be to identify programs violating work-hour restrictions or even to better inform
evolving work-hour policies in the future. This is particularly relevant because residents
may underreport their work hours, particularly if after-hours EHR use is considered.[38]
[39] Given the growing epidemic of physician burnout, understanding how to leverage these
data to inform strategies for improving physicians' experience at a critical (and
vulnerable) stage of their training is increasingly important.
