Keywords audit and feedback - electronic health records and systems - care transition - provider-provider
communication - pediatrics
Background and Significance
Background and Significance
Clinicians develop expertise by learning from past performance.[1 ] Clinicians who receive information about their performance can become better calibrated
and thus make better diagnosis and treatment decisions.[2 ] Calibration is the alignment between clinicians' confidence in their accuracy and
their actual accuracy[3 ] and can be achieved by receiving consistent feedback about one's patient outcomes.[3 ]
[4 ]
[5 ]
[6 ]
[7 ]
A total of 40% critically ill children with acute illness and injury are initially
diagnosed and treated at community hospitals.[8 ]
[9 ] Front-line clinicians have variable training in pediatrics and may infrequently
encounter very ill children which can result in suboptimal care, most commonly undertreatment.[10 ] Timely interventions based on accurate assessments by community physicians have
been shown to improve outcomes in critically ill children[11 ]
[12 ]; thus, it is essential for referring clinicians to learn about their transferred
patients' outcomes to improve future performance. However, in our highly compartmentalized
model of health care, referring clinicians do not receive consistent feedback on individual
patient outcomes to continuously improve their practice. In a statewide survey, we
conducted in Iowa, referring clinicians received patient outcome feedback on only
40% of patients they transferred to pediatric intensive care units (PICUs) over a
year. Among clinicians who obtained feedback,13% received information they did not
expect; 40% of these clinicians indicated that this experience changed their practice.[13 ]
Objectives
Given this significant gap in feedback and its potential to improve the emergency
care of critically ill children, our objective was to develop, implement, and evaluate
a semiautomated electronic health record (EHR)-supported system at a single institution
to deliver timely and relevant PICU patient outcome feedback to referring emergency
department (ED) physicians. In this case study, we report the feasibility, usability,
and relevance of the feedback system to both ED and PICU physicians.
Methods
Using a human factors engineering approach informed by the Health Information Technology
(IT) Safety Framework,[14 ] we iteratively designed, implemented, and evaluated a semiautomated electronic feedback
system leveraging the Epic EHR at a single institution. This project was reviewed
and determined to be exempt from human subjects research oversight by the University
of Iowa Institutional Review Board. In the next sections, we describe the clinical
settings for the project and how we determined the need for feedback, designed the
feedback system/report, and evaluated its performance.
Clinical Settings
The EHR-supported feedback system was developed for implementation at the University
of Iowa Hospitals and Clinics (UIHC) ED and the UI Stead Family Children's Hospital
PICU. Both serve mostly rural populations within a large catchment area. The UIHC
ED is a tertiary referral academic ED with 46 beds accommodating approximately 15,000
pediatric visits per year and is staffed by 30 emergency medicine (EM) attending physicians
(six pediatric EM trained). The UI PICU is the only academic tertiary referral PICU
in Iowa, admitting approximately 1,200 patients per year and is staffed by 11 attending
intensivists.
Determining the Need for Patient Outcome Feedback
We conducted two focus groups of ED physicians (six UI ED physicians and six non-UI-affiliated
referring physicians) to substantiate our earlier survey indicating the gap in feedback[13 ] and explore physicians' perceptions of the relevance of receiving consistent patient
outcome information. Referring physicians noted the various unsystematic ways in which
they find out what happened to their patients. This included their own efforts to
reaccess patients' medical records or call the PICU to obtain an update on patients.
Sometimes, they received patient outcome information through third parties, such as
colleagues, patients' family members, or via social media. Referring physicians underscored
that knowing their patients' outcomes can help them evaluate and improve their practice.
They also indicated that open communication between medical professionals who share
patients across practice settings can improve transitions of care for critically ill
children.
Development and Evaluation of the Feedback System Using a Human Factors Engineering
Approach
Our team used the UIHC's existing Epic EHR platform to develop an electronic feedback
system. Commonly used EHR platforms, such as Epic, have existing functionality, such
as the ability to insert patient data into templated letters[15 ]
[16 ] that can automate steps in the feedback process and can be readily adapted for implementation.[17 ]
We used a human factors engineering approach as applied to biomedical informatics
projects to develop and test the EHR-supported feedback process. This approach uses
a functional model of a human factors engineering lifecycle adapted for health care
work situations and incorporates usability engineering methods to evaluate health
care products and workflows.[18 ] Throughout development, we were guided by the Health Information Technology Safety
Framework[14 ]
[17 ] which provides a multidimensional sociotechnical approach for evaluating and ensuring
safety of health IT in complex adaptive health care settings. This approach accounts
for important interactions among clinicians, tasks/workflows, and technology that
may present as facilitators or barriers to implementation. [Fig. 1 ] shows each phase of the project with its corresponding goals and methods.
Fig. 1 Overview of development and evaluation of an EHR-supported patient outcome feedback
system using a human factors engineering approach. ED, emergency department; EHR,
electronic health record; IT, information technology, PICU, pediatric intensive care
unit; UI, University of Iowa.
Phase I: Analysis of the Sociotechnical System
To develop a deep understanding of the health care work system where the feedback
process will be integrated, we performed semistructured interviews and focus groups
with various stakeholders. During the same focus groups, we conducted with referring
physicians above, in addition to determining the need for feedback, we also gathered
information on physicians' clinical workflows and environments and elicited their
perspectives on how an electronic feedback system may be best implemented. We also
interviewed support staff (e.g., UI telephone operators), health IT/EHR platform specialists,
and administrators (e.g., information officers and legal counsel), who we anticipated
will play a role in the feedback process or have knowledge of resources relevant to
delivering feedback. Finally, we reviewed organizational and state/federal data transfer
regulations to be considered.
Phase II: Cooperative Design
We collaboratively designed the EHR-supported feedback system by translating stakeholder,
organizational, and usability objectives (from Phase I) into feedback process and
feedback report requirements. We designed a workflow for how feedback reports could
be generated, delivered, and received. For each step, we identified tasks that need
to be performed, who/what will perform the tasks (staff/clinicians versus EHR), and
when/where tasks will be performed ([Fig. 2 ]). Stakeholders also designed the feedback report by considering clinical data relevant
to ED clinicians and the EHR's capabilities to pull/push and summarize clinical data
([Fig. 3 ]).
Fig. 2 Feedback process systems design showing (A ) overall feedback process requirements and (B ) feedback workflow with functions of each step and information technology/human requirements.
Epic applications and functions are shown since this was the specific EHR platform
used for the project. ATC, admission and transfer center; ED, emergency department;
EHR, electronic health record; IT, information technology; PICU, pediatric intensive
care unit; UI, University of Iowa; UIHC, UI Hospitals and Clinics.
Fig. 3 Feedback report design showing (A ) report requirements and (B ) a prototype report generated in the EHR and sources of the report content. Epic
was the specific EHR platform used for the project. EHR, electronic health record;
PICU, pediatric intensive care unit.
Phase III: Implementation and Iterative Evaluation
Over 6 months, we completed three cycles of implementation, evaluation, and redesign.
We generated feedback reports for all patients who were transferred to the PICU within
24 hours of a UIHC ED visit. As illustrated in [Fig. 2 ], patients eligible for feedback and their referring physicians were identified on
the third PICU day and the initial feedback reports were manually generated by our
team. Reports were then routed to PICU physicians who reviewed, edited, and sent them
to ED physicians.
We also developed a set of evaluation metrics that would best reflect the feedback
system's performance with regard to stakeholder goals while also considering the feasibility
of data collection. These metrics and data sources are summarized in [Table 1 ]. We evaluated the feedback system's feasibility and timeliness of feedback delivery
and receipt by reviewing EHR access logs to determine feedback report throughput.
We analyzed feedback reports sent to determine the sources of feedback report content.
We conducted usability testing[19 ] of the EHR interface with six PICU physicians, asking them to review, edit, and
send actual feedback reports while thinking aloud.[19 ]
[20 ] We audio-recorded physicians' comments and two research team members directly observed
how they interacted with the electronic interface, paying particular attention to
the functionality of relevant EHR applications (Epic In Basket and Epic Quick Letter)
and problems in human–computer interaction. We also asked each physician to assess
the process using the System Usability Scale (SUS), a validated 10-item survey to
evaluate an individual's assessment of a system's usability. SUS scoring yields a
composite measure between 0 to 100 which represents overall usability. Products with
SUS scores >85 are generally considered highly usable.[21 ] Finally, we conducted semistructured interviews with PICU physicians and ED-referring
physicians to determine how well the feedback process is integrated into physician
workflows and to gather physicians' perceptions of the relevance of feedback reports
to their clinical practice. Descriptive statistics were used to analyze and report
quantitative data. Qualitative data from interviews were transcribed, coded, and synthesized
into common themes.
Table 1
Evaluation metrics and data sources
Evaluation metrics
Data sources
Feasibility, timeliness, and impact on workload
Feedback report throughput[a ]
EHR reports and access logs
Sources of feedback report content
Feedback report
Impact on workload
Direct observation, physician interviews
Usability of electronic interface
Time for PICU physician to complete editing and sending report
Direct observation
System Usability Scale score[b ]
Rating by PICU physician
Navigation problems in electronic interface
Direct observation, physician interviews
Actions taken to confirm information automatically populated by the EHR or to seek
additional information
Direct observation, physician interviews
Changes made to information automatically populated by the EHR
Direct observation, physician interviews
Relevance to clinical practice
Physician interviews
Abbreviations: EHR, electronic health record; PICU, pediatric intensive care unit.
a Feedback report throughput includes the number of reports generated, routed to PICU
physicians, sent to emergency department (ED) physicians, and reviewed by ED physicians
and the length of time between these events.
b The System Usability Scale (SUS) is a validated 10-item survey used to evaluate an
individual's assessment of a system's usability. Scoring of the SUS yields a composite
measure between 0 and 100 that represents overall usability. Products with SUS scores
>85 are generally considered highly usable.[21 ]
Results
Feasibility and Timeliness of Feedback Delivery
Over 6 months, 119 feedback reports were generated and routed to PICU physicians.
PICU physicians edited and sent 98 (82%) feedback reports. Referring ED clinicians
received and reviewed 86 (72%) reports. Overall, feedback reports were sent by 10
unique PICU physicians and reviewed by 25 unique ED clinicians.
During the first two cycles of implementation, 21 (18%) feedback reports were inappropriately
deleted before PICU physicians were able to review them. Our IT team identified that
this was due to a preexisting custom Epic communication management workflow, wherein
pended feedback reports were deleted when other users created and sent new unrelated
reports using the same application (Epic Quick Letter). This was addressed by revising
the programmed Epic workflow, preventing further loss of reports.
Otherwise, we found that the EHR-supported feedback process performed technically
well. Majority of the feedback report content (86%) was automatically pulled in from
the EHR, with only 12% of the content typed in by PICU physicians (most commonly to
clarify/add diagnoses and include information on clinical course and status). Feedback
reports were generated and routed to PICU physicians a mean of 3.4 days from PICU
admission. PICU physicians completed and sent the reports a mean of 0.7 days after
they were routed to them. Reports were reviewed by ED clinicians a mean of 1.8 days
after being sent and a mean of 6.8 days from their encounter with the patient in the
ED ([Table 2 ]).
Table 2
Technical characteristics of EHR-enabled patient outcome feedback system and update
report
Technical characteristics
Mean or proportion
Update report content, total word count, mean (SD)
277 (59)
Information automatically pulled from EHR, word count, mean (SD, % of total)
239 (47, 86%)
Information selected by clinician from dropdown choices, word count, mean (SD, %
of total)
7 (5, 3%)
Information manually added by clinician, word count, mean (SD, % of total)
31 (26, 12%)
Feedback process delivery metrics[a ]
Time to routing of initiated update report to PICU physician from PICU admission,
days, mean (SD)
3.4 (0.7)
Time to completion of report once received by PICU physician, days, mean (SD), days,
mean (SD)
0.7 (1.9)
Time to report viewed by ED clinician once sent by PICU clinician, days, mean (SD)
1.8 (3.8)
Time to report viewed by ED clinician from ED visit, days, mean (SD)
6.8 (4.3)
Update report received by the correct ED clinician, n (%)
90 (92)
Abbreviations: ED, emergency department; EHR, electronic health record; PICU, pediatric
intensive care unit; SD, standard deviation.
a Of 98 letters sent to ED clinicians by PICU physicians.
Usability
Direct observation of six PICU physicians (two physicians per iterative cycle), four
of which were novice users (less than 10 feedback reports previously sent), showed
an overall mean of 4.7 minutes to review, edit, and send feedback reports. However,
nonnovice users performed faster (mean of 2.5 minutes). Usability testing with the
same six physicians yielded a mean SUS score of 88.3, suggesting an “excellent” usability.[22 ] Key observations included difficulties in navigation and physicians confirming information
by reviewing other sections of the EHR (both unique to novice users). Half of physicians
did not edit diagnoses automatically pulled from EHR problem lists, while all physicians
typed in additional information regarding the patient's clinical course and intermediate
outcomes ([Table 3 ]).
Table 3
Usability characteristics of EHR-enabled patient outcome feedback system and update
report
Usability characteristics[a ]
Time for PICU physician to complete reviewing, editing, and sending update report
(min), minutes, mean (SD)
4.7 (2.9)
System Usability Scale score,[b ] mean (SD)
88.3 (7.9)
Key observations from usability testing with PICU physicians, n
Frequency (n =6)
Had difficulties with navigating from EHR messaging application to letter editing/sending
application (only observed among novice users)
3
Needed to confirm information by reviewing other sections of EHR
3
Did not edit diagnoses automatically pulled from EHR problem list (i.e., did not
include a narrative summary of diagnoses)
3
Added more information describing the patient's clinical course and outcomes
6
Abbreviations: EHR, electronic health record; PICU, pediatric intensive care unit;
SD, standard deviation.
a Usability testing was performed with six PICU physicians with varying levels of experience
in sending feedback reports prior to testing.
b The System Usability Scale (SUS) is a validated 10-item survey to evaluate an individual's
assessment of a system's usability. SUS scoring yields a composite measure between
0 and 100, which represents overall usability. Products with SUS scores >85 are generally
considered highly usable.[21 ]
Impact on Workload and Relevance of Feedback Reports to Clinical Practice
Interviews with six PICU physicians and four ED physicians revealed that the feedback
process was well-integrated into their respective clinical workflows. Because reports
were routed or sent to physicians' EHR in-baskets, sending and receiving reports closely
aligned with the physicians' existing identical workflows for signing clinical notes.
ED physicians also appreciated that receiving feedback reports within the EHR provided
them easy access to additional information in the patient's chart if needed. Both
PICU and ED physicians indicated that they sent/reviewed feedback reports at the same
time that they signed their notes and that the overall time burden for completing/reviewing
reports was acceptable. ED physicians noted that they received reports within an acceptable
timeframe as well.
PICU physicians indicated that sending patient outcome feedback to their ED colleagues
is a worthwhile endeavor and will overall help improve the emergency care of children
prior to PICU transfer. ED physicians indicated that receiving patient outcome feedback
is relevant to their clinical practice, since it helps them identify whether they
have appropriately triaged patients (i.e., to the PICU vs. general wards) and provides
them with a venue to reflect on potential improvements in their diagnostic reasoning
and clinical management.
Iterative Changes
After the first cycle, we disseminated a navigation guide to PICU physicians, this
supported the learnability of the process and served as a quick reference while the
task was performed. We also sent e-mails to PICU physicians at the beginning of each
clinical service week to remind them to review and send reports. This was phased out
after 5 weeks when the process was well-integrated into the workflow. After the second
cycle, we responded to questions from PICU physicians regarding specific clinical
information that ED physicians found helpful (additional diagnoses since the ED visit,
evolution of the clinical course, and patient disposition) by reporting back results
of our interviews with ED physicians. We also revised the EHR access log reports we
created to track feedback report throughput, so we can more clearly delineate time
intervals between tasks. Our IT team also identified and addressed the software error
which caused feedback reports to be inappropriately deleted, as described earlier.
After the third cycle, we added ED resident physicians as feedback recipients at the
request of the pediatric ED medical director. We also noted that significant time
and effort was needed to manually identify eligible patients and their respective
referring physicians and PICU physicians to ensure correct routing and delivery of
feedback reports, suggesting the need for more automation in this process.
Discussion
In this case study, we showed that a semiautomated EHR-supported system to deliver
timely and relevant PICU patient outcome feedback to referring physicians is technically
feasible, usable, and relevant to both ED and PICU physicians at a single institution,
sharing a common EHR platform. Although our system is distinct from prior published
work, deploying electronic interventions to improve subspecialty referrals have been
used in the past with varying success.[23 ]
[24 ]
The main challenges we encountered included the need for PICU physician review of
feedback reports (due to limitations in the EHR's ability to accurately summarize
clinical data[25 ]), PICU physicians' clinical workload affecting timely completion of feedback reports,
and the significant effort in manually identifying eligible patients and their corresponding
ED and PICU physicians for the purpose of generating feedback reports for routing
to PICU physicians. Expanding the electronic process to deliver feedback beyond our
institution has also been challenging due to the limited interoperability of EHR platforms,
making it difficult to send feedback reports between institutions with different EHR
systems and even between those with the same but differently configured EHR system.
Pending progress on this issue, delivering feedback outside of our health care organization
will necessitate a hybrid electronic-paper (faxed reports) feedback process.
Limitations and Strengths
Limitations and Strengths
Our case study has limitations. The feedback system was tested in only one institution
and only between two groups of physicians, although efforts are underway in our institution
to expand the process outside of our health care organization in different clinical
settings. We also only determined the technical performance and relevance of the feedback
system and did not measure impact on clinical practice and patient outcomes which
will be essential next steps.
Despite these limitations, we expect that the system we have developed can be adapted
to other institutions and/or other clinical settings across the continuum of patient
care. Although we used the Epic EHR platform to develop and test the feedback process,
we anticipate that other EHR platforms have similar functions that can be used to
build a similar feedback system. We recommend that institutions follow a similar step-wise
human factors approach to development to ensure that the system is feasible in their
setting and fulfills their specific feedback goals. The following are the key lessons
we learned:
Create a sense of serious urgency (a “burning platform”) to highlight the importance
and need for feedback between clinicians in the target settings.
Involve all stakeholders from the beginning, especially referring clinicians, to achieve
consensus and clarity around feedback goals and the processes to be implemented to
attain these goals.
Understand the context of the health care system and the environment where the feedback
intervention will be embedded in both clinical settings sending and receiving feedback,
not just the IT or EHR platform to be used.
Collaboratively design the feedback process involving all stakeholders.
Identify important and feasible metrics to evaluate the performance and impact of
the feedback process.
Conclusion
In conclusion, an EHR-supported system to deliver timely and relevant PICU patient
outcome feedback to referring ED physicians was technically feasible, usable, and
important to physicians. Next steps include wider implementation to referring clinicians
from other institutions, evaluation of the feedback system's impact on clinical practice
and patient outcomes, and investigation of potential applicability to other clinical
settings involved in similar care transitions.
Clinical Relevance Statement
Clinical Relevance Statement
Patient outcome feedback is an important way by which clinicians can improve their
performance and the care they provide to patients. Leveraging the EHR to support an
electronic semiautomated feedback system that can deliver timely and relevant patient
outcome information to clinicians is a feasible way to operationalize consistent clinician
feedback. The feedback system we developed can potentially be adapted for implementation
in other clinical settings involved in similar care transitions.
Multiple Choice Questions
Multiple Choice Questions
Why is it important to conduct stakeholder interviews and focus groups before implementing
a new information technology-supported process?
To determine perceived facilitators and barriers to the process to inform development
To inform stakeholders of the new workflow and specific tasks that they will now need
to do
To convince stakeholders that implementation of the new process will be of benefit
to them
To emphasize the importance of adhering to the new process
Correct Answer: The correct answer is option a. It is important to determine stakeholders' perspectives
regarding how proposed new processes can be facilitated or hindered within their specific
health care environments to proactively leverage facilitators and develop solutions
to minimize barriers for more successful implementation.
What data can usability testing provide that are useful to inform the development
and evaluation of a new electronic process?
Functionality of electronic applications
Problems in the human–computer interface
Impact on clinicians' workload
All of the above
Correct Answer: The correct answer is option d. Usability testing is important to (1) determine how
well electronic programs perform their specified functions, (2) identify problems
in the program interface that may interfere with human operation, and (3) measure
the workload conferred by the new process (e.g., time to completion of tasks, effort
required to complete tasks, and others).
