Identifying Barriers to The Implementation of Communicating Narrative Concerns Entered by Registered Nurses, An Early Warning System SmartApp

Keywords

qualitative - nursing informatics - toolkit - implementation - early warning system

Background and Significance

In the United States, more than 200,000 deaths occur annually in the inpatient setting as a result of cardiac arrest or sepsis.[1] It is suggested that 25% of cardiac arrests[2] and 13% of sepsis events[3] are due to preventable causes. Research shows that patients exhibit symptoms and signs indicating clinical decompensation at least 6 to 8 hours prior to deterioration,[4] with nurses often being the first to detect and respond to these events.[5] Nurses' assessments and clinical actions are predictors of patient deterioration, and when nurses are concerned about a patient this translates to increased nursing surveillance and subsequently increased documentation in the electronic health record (EHR).[6] Thus, our team developed and implemented an early warning system that leverages the signal resulting from changes in nursing documentation to identify patients who are at risk for clinical deterioration and negative patient outcomes such as cardiac arrest and sepsis in the acute care setting.

The Communicating Narrative Concerns Entered by Registered Nurses Early Warning System

To accomplish this, we built Communicating Narrative Concerns Entered by Registered Nurses (CONCERN), an early warning system that generates a risk prediction score utilizing nursing data and machine learning.[7] CONCERN is novel because it uses both nursing structured (i.e., vital sign entry frequency, medication administration record frequency, and demographics) and unstructured documentation (i.e., nursing note frequency, vital sign comment frequency, and nursing note content) to stratify patients at risk for clinical deterioration into three categories, red: high-risk, yellow: medium-risk, green: low-risk. In addition, CONCERN analyzes data routinely documented rather than imposing additional documentation requirements on nurses.[8] The CONCERN early warning system consists of a (1) clinician-facing SMART-on-Fast Healthcare Interoperability Resources (FHIR) app integrated into the EHR system and (2) a server-based, custom-built FHIR decision engine that uses patient data to generate hourly deterioration risk predictions. Targeted users of CONCERN include clinicians such as prescribing providers and registered nurses in the inpatient setting.

Communicating Narrative Concerns Entered by Registered Nurses Implementation

CONCERN was implemented on 37 intervention units (intensive care and medical surgical) during a randomized controlled trial at two health systems in the Northeastern United States, including one academic medical center and one community hospital at each site.[5] Our implementation process included a pilot and then a multisite randomized controlled trial. ⁵Site A was implemented in October 2020, and Site B was implemented in October 2021.

Given the timing of implementation and the coronavirus disease (COVID-19) pandemic, there were a few differences in the implementation process at each site. Site A included 24 intervention units and conducted their clinical end-user training during the COVID-19 surge in the Spring of 2020. Since CONCERN was already configured in the EHR and functioned to help identify deteriorating patients, the implementation continued to progress through the pandemic at Site A. Trainings were offered in person and virtually to accommodate learning needs and specific unit and health system restrictions. Site B's implementation experienced more delays due to the stalling of research activities during the pandemic and the simultaneous transition in EHR systems. The education campaign with clinical end-users began in late Spring 2021. By that time, the COVID-19 surge had declined. Operations were beginning to return to prepandemic normalcy, allowing the CONCERN team to present in person (and virtually) in the 13 intervention units to conduct educational sessions and attend nursing-shared decision-making council meetings.

The Communicating Narrative Concerns Entered by Registered Nurses Toolkit

Following the implementation of CONCERN at Sites A and B, our team sought to develop the CONCERN Implementation Toolkit[9] to enable two other hospital systems to adopt CONCERN. Toolkits are a collection of adaptable resources or documents such as readiness checklists, education tools, and evaluation criteria to guide implementation at different study sites.[10] The development of the α version of the CONCERN Implementation Toolkit included team members who had participated in the original implementation process and a subject matter expert and nurse informatician (P.D.) who had previous experience creating a toolkit, which is widely used in the inpatient setting.[11] An outline of the α version of the CONCERN Implementation Toolkit is included in [Appendix A].[12] Aligned with prior implementation methodology,[10] [12] [13] we validated the α version of the CONCERN Implementation Toolkit with qualitative interviews to identify the best implementation strategies. The Toolkit serves as a modality to disseminate our best implementation strategies to future stakeholders. Therefore, as implementation strategies were validated or revised based on the interviews, the Toolkit was modified.[9] [11] [12] The results of this iterative process can inform future research teams interested in implementing and scaling early warning systems across multiple hospital sites through the use of toolkits.

Appendix A: Validated Alpha CONCERN Implementation Toolkit

Toolkit components	Category	Individual resources
Clinical practice and documentation tool	Unit champion (UC) resources for peers	• Unit champion overview sheet and orientation checklist[a] • Slide decks for unit-based clinicians (abbreviated and long versions) • Nursing assistant slides[a] • Study overview and information sheet • Frequently asked questions (FAQ) for nurses[a] • FAQ for prescribing providers[a] • Introductory email text • Tip sheet—how to add CONCERN to the epic patient list
	Pocket card	• Power Point (PPT) and Portable Document Format (PDF) pocket card templates
	Poster	• PPT and PDF poster template[b]
	Stakeholder presentations	• Unit champion slides • Clinical end-users (nurses, prescribing providers) slides • Unit-based clinical leaders slides • Shared-decision making council slides
	YouTube video link and QR code	• Narrated PPT slides • MP4
Evaluation measure tools	Outcomes definitions and exclusions	• CONCERN outcomes tracking spreadsheet
	Configuration and hospital evaluation plan	• Study cohort and outcome flow diagram
Executive leader engagement tools	Stakeholder presentations	• Information technology (IT) leadership slides[b] • Nurse executive slides • Prescribing providers leadership slides
	Communication template	• Email text for executives and leaders about implementation
Financial considerations	Cost-benefit analysis	• Customizable cost-benefit tool • CONCERN cost-benefit analysis paper using the cost-benefit tool • Cost-benefit projection slides
Governance tools	IRB guidance	• Approved protocol for study • Approved recruitment flyer for postimplementation usability evaluation • Approved focus group interview guide for postimplementation usability evaluation
	Advisory board structure	• Minutes template and best-practice recommendations. (e.g., meeting cadence, roles of Advisory Board members)
Model calibration and bias mitigation tools	Model calibration	• Model calibration slides
Readiness assessment tools	Readiness assessment	• Local site checklist[a]
Sustainability and spead tools	Postimplementation (first 2-weeks post)	• Email text for post-go live support • Best practices on post-go live rounding schedule[a] • Sign in sheet template[a]
	CONCERN refresh tools (longer term postimplementation)	• Best practice and recommendations on CONCERN refresh rounding schedule[a] • CONCERN refresh slides[b]
Technical implementation tools	Technical documentation	• App orchard documentation for FHIR • FHIR implementation guide • Docker Bundle[a]
	IT staff support guidance	• IT technical support and troubleshooting document for end-users

Abbreviation: CONCERN, Communicating Narrative Concerns Entered by Registered Nurses.

^a Resource created after the interviews.

^b Resources modified after the interviews.

Objectives

The aim of this study was to identify the optimal resources needed to implement CONCERN and package these resources into the CONCERN Implementation Toolkit to enable the spread of CONCERN to other hospital sites.

Methods

Sample

Participants of this study included nurses, prescribing providers, and information technology (IT) experts. Site A included two academic medical hospitals located in the greater Boston, MA with 1,019 beds. Site B included two large academic medical hospitals with 2,696 beds located in New York City, New York. To be eligible for this study, clinicians were required to be either a prescribing provider (i.e., medical doctor, nurse practitioner, and physician assistant) or a registered nurse who had access to or encountered CONCERN on one of the intervention units at our study sites, given that CONCERN was implemented as part of a clinical trial.[5] In addition, eligible clinicians had to have used CONCERN within the last year as part of the clinical trial. Eligible IT experts had to support the implementation of CONCERN at one of the hospital sites.

Recruitment

We conducted focus groups and interviews at our two study sites from July 2022 to January 2023. Robust recruitment strategies were utilized to engage clinicians and included a range of methods such as (1) emailing CONCERN unit champions (e.g., clinical nurses on implementation units who served as peer trainers) and nurse unit managers, (2) posting flyers, (3) snowball sampling (i.e., asking participants to refer peers), (4) presenting at change of shift huddles, (5) presenting at shared decision-making council meetings, (6) leveraging our personal network, and (7) rounding on intervention units. Participants were provided with a $20 Amazon electronic gift card for their participation. We used convenience sampling to recruit IT experts who were involved in the implementation of CONCERN at one of the hospital sites. We received approval from the Institutional Review Boards at both study sites.

Data Collection

We created a semistructured focus group interview guide that was iteratively refined among all coauthors. Focus groups and interviews were led by P.D. or S.C. at Site A and M.H. at Site B; interviewers all had previous experience conducting interviews. Focus groups included two research team members, and at least one member was involved in the development and deployment of CONCERN at their respective institutions. Interviews were conducted if there were conflicts with scheduling focus group meetings. Interviews or focus groups typically lasted from 30 to 45 minutes and were conducted online via Zoom (Zoom Video Communications; San Jose, CA, United States) or in person on the unit. Verbal consent was provided prior to beginning the interview. Only audio was recorded via Zoom and participant names were deidentified. Participants were instructed to keep their cameras turned off, and to change their names on the Zoom display. The audio mp4 files were saved on a secure, Health Insurance Portability and Accountability Act compliant drive, and transcribed using Word transcription services (Microsoft; Redmond, WA, United States). Participants were recruited until data saturation was achieved by our thirteenth interview (i.e., no new information emerged).[14]

Data Analysis

Our analysis was guided by the Donabedian model[15]: (1) Structure: how CONCERN was implemented, (2) Process: barriers to implementation, and (3) Outcome: recommendations for future implementation. M.H. cleaned and de-identified the transcripts prior to analysis. We used NVivo (Lumivero; Denver, CO, United States) to conduct deductive content analysis. MH has experience as a nurse and informatician and served as the primary coder. The secondary coder, J.W., has experience in nursing education and reviewed the data and codebook to assess the completeness of analysis and codes identified. Coding discrepancies were discussed between J.W. and M.H. until consensus was met. Themes were presented to the research team and iteratively refined.

Toolkit Mapping

As implementation strategies were validated or revised based on the interviews, we updated the CONCERN Toolkit content to reflect those revisions. The resultant themes were cross-walked with existing resources in the Toolkit to evaluate any gaps in availability of resources. New Toolkit resources were created or added to the Toolkit to address the gaps, while other resources were modified. The mapping was initially conducted by J.W. and verified by the research team. Any discrepancies were iteratively discussed and resolved. The mapping of the Toolkit resources was guided by the emerging themes in the analysis.

Results

Our study included a total of 32 participants who used or were involved in the implementation of CONCERN. We conducted a total of six focus groups and seven interviews. A majority of the sample (87%; n = 20) were registered nurses, nine were IT experts (28%), and three were prescribing providers (13%). Most participants were from Site A (n = 19; 59%); 13 participants were from Site B (41%). [Table 1] presents characteristics of the participants included in the study.

Table 1 Participant characteristics

Demographics	n (%)
Participant roles (n = 32):
Prescribing providers
Physician assistant	2 (6%)
Nurse practitioner	1 (3%)
Registered nurses	20 (63%)
IT experts
Senior software engineer	2 (6%)
Systems designer	2 (6%)
Application coordinator	2 (6%)
Integration architect	2 (6%)
Chief medical information officer	1 (3%)
Clinician practice setting (n = 23):
Acute care	12 (80%)
Intensive care	3 (20%)
Not reported	8 (35%)
Years in current profession (n = 32):
< 5 y	5 (16%)
5–20 y	3 (9%)
> 20 y	7 (22%)
Not reported	17 (53%)

Abbreviation: IT, information technology.

The results of this study converged around two major ideas: the importance of employing a multimodal approach (e.g., posters, unit presentations, and unit champion) to educate clinicians on CONCERN and engaging with hospital leadership teams early in the implementation process. In total, 12 themes emerged with four themes mapping to each domain of Donabedian's model. Exemplar quotes are presented in [Table 2]. Following the table, each theme is narratively described. In addition, [Table 2] contains a column notating all new Toolkit resources created (resources added to the Toolkit) or modified (resources modified in the Toolkit). Eight new resources were added to the Toolkit including: unit champion overview sheet and orientation checklist, frequently asked questions (FAQs) for nurses, FAQ for prescribing providers, Docker Bundle, sign in sheet template, best practices on post-go live rounding schedule, nursing assistant slide, and the local site checklist.

Table 2 Summary of thematic analysis

Structure
Themes	Exemplar quotes	Changes to Alpha Toolkit resources:
Theme 1: Implementation strategies used to educate clinical end users on CONCERN	“We have a large [poster] board that we referred to… when we weren't sure…So I believe questions were minimal” (RN, Site A)	Created:  • Unit champion overview sheet and orientation checklist  • Frequently asked questions (FAQ) for nurses  • FAQ for prescribing Providers  • Nursing assistant Slide Modified:  • PPT and PDF poster template
	“[As a unit champion,] I was part of the education with my unit…, I was doing weekly education to reiterate everything that CONCERN was trying to roll out…I think just reiterating it has been helpful” (RN, Site B)
Theme 2: Leveraged multi-level stakeholder support to disseminate information on CONCERN integration	“[We] talked to nursing leadership as soon as the research project got funded, so they had a sense of what would be coming down the line in four years. Within two years we started talking to the IT leadership as we got closer to implementation. About a year out, we started going out to units and talking to unit leaders and staff, so they could know that it was coming and to recruit champions that could be there on the frontline with users” (IT expert, Site B)	Modified:  • Stakeholder presentations to clinical end users slides  • Information technology (IT) leadership slides
Theme 3: Highlighted the impact of CONCERN for nurses	“I think if [CONCERN] starts to become a tool that is widely accepted within the hospital, it can give more satisfaction … [to nurses] that their work is important because it's nurse driven and it's really just showing that your work will pay off in helping this patient” (RN, Site B)	Created:  • FAQ for nurses Modified:  • Stakeholder presentations to clinical end users slides
	“[CONCERN] helps to figure out where your focus [should be] at the beginning of your shift, so I like that idea, especially with new grads that are coming on that aren't confident in their own assessment making skills yet... It gives them a starting point” (RN, Site A)
Theme 4: Facilitators to technical implementation	“Interacting with the EHR vendor … [to] get updates saying this piece of software is no longer being supported or you know this piece of software needs to be patched to this version” (IT expert, Site B)	Created:  • Docker Bundle
	“First, we understand the business ask…and what exactly we're trying to accomplish in the EHR vendor…[then], we review that with our leadership teams [and] clinicians on the business side… once that approval is done, we go to the next phase, which is the technical feasibility assessment. and once that form is complete, we meet with the requestors as well as our development managers” (IT expert, Site A)
Process
Theme 5: Managing the mismatch in the function of CONCERN	“Sometimes if my patients [CONCERN scores] are yellow or red, I am like this patient is completely stable. I guess I must look into the reason why [CONCERN scores] are a certain color” (RN, Site A)	Created:  • FAQ for nurses Modified:  • Stakeholder Presentations to clinical end users slides
Theme 6: Managing expectations related to policy requirements	“There is stuff in policy that they feel like they're held accountable, so I think [CONCERN] is just more information” (RN, Site A)	Modified:  • Stakeholder presentations to clinical end users slides
Theme 7: Staffing and workload considerations when devising implementation strategies	“Our unit specifically had so much turnover and new nurses… [are] learning the basics, experienced nurses… [are] fatigued... So, adding something new to the plate while it's super useful, it just seemed like one more thing to focus on when we are just trying to get everyone to focus on the basics” (RN, Site B)	Created:  • Sign in sheet template  • Best practices on post-go live rounding schedule
	“In this [EHR] system everybody just throws stuff at us... It's too much charting, too many things all over the place and you never get a chance to absorb anything you know” (RN, Site A)
Theme 8: Navigating the complexities of a “first of its kind” innovative system	“There was a reorganization of committees… folks within the organization just trying to figure out the right channels to go through and that led to some obstacles” (IT Expert, Site B)	Created:  • Local site checklist
	“This was a research project, so it was studying an informatics intervention which is a bit different than a production enhancement to improve care and quality within the healthcare organization. It's for studying a particular intervention – so it makes it a little bit unique” (IT Expert, Site B)
Outcome
Theme 9: Increase the visibility of CONCERN	“A big barrier is… I personally use the Brain in the EHR vendor, and I find that a lot of younger nurses like to use the brain and [CONCERN] isn't present” (RN, Site B)	Created:  • FAQ for nurses
Theme 10: Suggestions for future CONCERN education	“If part of the core team came to huddle to do an in service and talk to the nurses on the unit and that could be helpful too. Because I feel like the unit champions have kind of a foundation of what CONCERN is but having somebody who knows more of the details, like the nitty gritty stuff coming by and doing an in service could be helpful” (RN, Site B)	Created:  • FAQ for nurses  • Best practices on post-go live rounding schedule  • Nursing assistant slide Modified:  • CONCERN refresh slides
	“Maybe create a module to add on to Workday [education training] that can kind of be released to mass people. That way it's kind of required, like annual training” (RN, Site B)
Theme 11: Future technical optimizations	“We did not get a clear picture if PMO [project management office] process was done or not, so I think it would have been helpful if we all knew [what] planning phase” (IT expert, Site B)	Created: • Local site checklist
Theme 12: Implementation science consideration	“There are all these factors we've talked about, like the pandemic and regulatory or governance bodies being stood up and there are lots of other potential things that cause implementations not to be cookie cutter events. So, it's important in implementation to sit down with the whole team like we did and say OK, these are the things we want to accomplish. How are we going to? And what's actually going to work? It affects what you end up rolling out and how you do it, so that's just something I think is important to keep in the background.” (IT Expert, Site B)	Created: • Local site checklist

Abbreviations: CONCERN, Communicating Narrative Concerns Entered by Registered Nurses; PDF, Portable Document Format; PPT, PowerPoint; RN, Registered Nurse.

Structure: Current Implementation Strategies

Theme 1: Implementation Strategies Used to Educate End Users on CONCERN

Implementation strategies included publicly displaying information about CONCERN on wall posters and digitally on the TV screen, distributing pocket cards to each clinician, and sending out informational emails describing CONCERN. These strategies allowed clinicians to easily access information on CONCERN during their clinical workflows. At the unit level, our team conducted educational sessions at change-of-shift huddles, created tailored presentations for each end-user group, and partnered with Unit Champions to disseminate information. In addition, CONCERN was automatically added to the “Patient List” in the EHR of each clinician's dashboard to support seamless integration and visibility. Following conversations with the participants, we identified a need to create additional training materials for the unit champion, a FAQ sheet for nurses and providers, and slides discussing the role of CONCERN for nursing assistants. We also modified the PowerPoint and PDF poster template to include the FAQ from the clinical end users.

Theme 2: Leveraged Multilevel Stakeholder Support to Disseminate Information on CONCERN Integration.

Partnering with multilevel hospital staff including leadership teams and unit level staff (e.g., nurse educators) early in the planning process facilitated the success of the implementation. Early engagement in governance and shared decision-making councils were established. IT experts stated that forming an interdisciplinary team with varied technical expertise that was readily available was important to the progression of the project goals. Partnerships with IT leadership helped to ensure that software was current and security assessments were met. Utilizing implementation checklists was encouraged to help ensure proactive stakeholder involvement. From this theme, we recognized the need to modify the stakeholder presentation slides to clinical end users and the IT leadership slides to include recommendations on required project team members at each stage and project timelines for leadership engagement.

Theme 3: Highlighted the Impact of CONCERN for Nurses

Nurses discussed the benefits of CONCERN in making patient assignments, amplifying nurses' expertise, increasing nurse recognition, and supporting new nurses in critical thinking and escalation of clinical issues. Clinicians also stated that CONCERN helped to increase situational awareness of the interdisciplinary team to promote the more efficient utilization of clinical resources. From this theme, we identified key questions that nurses prioritize in understanding CONCERN, allowing us to incorporate these questions into the FAQ and presentations for clinical end users.

Theme 4: Facilitators to Technical Implementation

IT stakeholders expressed the importance of developing an application that was EHR agnostic. Participants stated that choosing a standard (i.e., custom built bulk FHIR) that would mature overtime helped to support multisite integration. At Site A, the technical implementation followed established project management guidelines facilitating the process of governance approvals. To help facilitate this, we created the Toolkit resource “Docker Bundle.”

4.2 Process: Challenges Related to Implementation

Theme 5: Managing the mismatch in the function of CONCERN

Some clinicians expressed a mistrust in the clinical accuracy of the CONCERN risk score, with one clinician wondering how CONCERN accounts for specific situations that may deviate from the normal documentation inputs to adhere to a policy such as orthostatic blood pressure checks or blood transfusion protocols that would require more frequent blood pressure entries. Further education and explanation are needed around CONCERN's intended function to identify deterioration without adding to documentation requirements or altering existing workflows. This led us to make modifications to the stakeholder presentations for clinical end users and ensure that these questions were answered in the FAQ for nurses.

Theme 6: Managing Expectations Related to Policy Requirements

Clinicians also noted that the use of CONCERN not being tied to hospital policy requirements may influence adoption. This encouraged our team to reflect on how to incentivize the use of CONCERN apart from policy requirements. This theme prompted us to make modifications to the stakeholder presentations to ensure that our team was clearly setting expectations for clinicians on the intended use and application of the CONCERN early warning system.

Theme 7: Staffing and Workload Considerations when Devising Implementation Strategies

Participants discussed challenges related to CONCERN education and staffing stating that some clinicians might have missed the training due to the variability in the staffing schedule (e.g., night shift, rotating, travelers, per diem, and floating). The implementation of CONCERN at Site A also took place during the height of the COVID-19 pandemic, which caused disruptions to unit access and reassignment of implementation staff. Participants discussed workload challenges related to the constant introduction of new requirements and standards resulting in competing priorities. With the CONCERN implementation, IT experts stated that they had to navigate competing priorities related to transitioning EHR vendor. This theme prompted the creation of a sign in sheet template and post-go live rounding schedule to ensure that all clinicians were being familiarized to the CONCERN early warning system.

Theme 8: Navigating the Complexities of a “First of Its Kind” Innovative System

IT stakeholders discussed the challenges in navigating a first-of-its-kind early warning system built external to the EHR. Challenges noted related to mapping CONCERN to FHIR standards that would mature over time and having no template to follow related to governance approvals at Site B. One IT expert also described the uniqueness of CONCERN related to its origin as a research project rather than an EHR enhancement project. This theme highlighted the need for a local site checklist to provide flexible guidelines that could be modified to the differences in governance and structures of each health systems.

Outcome: Suggestions for Future Implementation

Theme 9: Increase the Visibility of CONCERN

Participants suggested increasing the visibility of CONCERN throughout the EHR to be aligned with additional workflow patterns that are site specific (e.g., activities including secure chat, brain). It was suggested that increasing the visibility of CONCERN to accommodate evolving clinician workflows and local site documentation policies, procedures, and standards could support future adoption. This theme prompted us to consider future optimizations for the CONCERN early warning system as well as making sure that the information in the FAQ for nurses was clear on the current intention and capabilities of CONCERN.

Theme 10: Suggestions for Future CONCERN Education

Participants suggested components to add into our educational sessions for CONCERN such as having a study team member conduct in-services at shift huddles and staff meetings, incorporating CONCERN education in standardized hospital education (e.g., annual education, orientation, continuing education credits), partnering with a unit-based educators to increase local dissemination, conducting ad hoc educational refreshers, increasing the visibility of the unit champion, providing case studies to guide clinicians on how to use CONCERN, and bringing tangible resources to help remember CONCERN (e.g., badge buddies). Staff also suggested broadening education to include other interdisciplinary care team members such as patient care assistants. This theme also helped to guide the creation of our FAQ for nurses, best practices for the post go-live rounding schedule, and the nursing assistant slide.

Theme 11: Future Technical Optimizations

Technical experts suggested fully leveraging cloud-based technologies to optimize the process of extracting patient data, transforming it, and inputting the risk score back into the EHR. One IT expert recommended the use of shared checklists to ensure that all team members were up to date with the project status and governance approvals received and needed. This theme helped to guide our local site checklist.

Theme 12: Implementation Science Consideration

The Toolkit is intended to guide future study sites in the adoption of CONCERN and ideally in the implementation of early warning systems, in general. Importantly, no two implementations are the same and each health system has unique operations, workflows, and policies that will influence the implementation process. Based on our experience, we developed a Toolkit to identify the suite of tasks, approvals, and actions typically required in the implementation process. We have also provided resources and materials that can be re-purposed for activities that may arise throughout the various stages of implementation. This theme also supported the need for creation of the local site checklist.

Discussion

Identifying methods and strategies that support sustainability and spread of evidence-based interventions is the cornerstone of implementation science. Our team employed an iterative approach to develop a Toolkit to help facilitate the implementation of one early warning system SmartApp, CONCERN. Validating the α version of the CONCERN Implementation Toolkit by eliciting feedback from clinician end-users and IT stakeholders is aligned with implementation methodology and is foundational to identifying best practices for future implementation at different hospital sites.[10] Through our qualitative results, we were able to identify barriers that prompted us to create or modify Toolkit resources to address these gaps. Similar to prior findings,[16] the results of this study were centered on the importance of employing a variety of approaches to engage and educate clinicians on the early warning system and secure buy-in of hospital leadership teams early.

Participants discussed barriers in the implementation process related to different levels of awareness of the CONCERN application. This is unsurprising given transient staffing and variable workflow in the acute care setting.[17] This is also true for training provider positions, such as residents, that frequently rotate through different services or float shift nurses who rotate through different units. Following the COVID-19 pandemic, the nursing workforce experienced high turnover rates with novice nurses rising in seniority quicker than in the past.[17] The evolving workforce prompted us to revisit our implementation strategies through modifying Toolkit resources (e.g., “CONCERN Refresh Slides”) and recognizing the importance of multimodal engagement strategies to educate clinicians on CONCERN.

Workforce changes have also caused senior nurses to be tasked with precepting more frequently than in previous years.[18] In this study, nurses commented on the challenge of juggling competing priorities daily, making it difficult to dedicate time to learn about new tools added to the EHR. Nurses reiterated the importance of emphasizing the scientific rationale behind CONCERN and how risk scores are calculated in educational sessions which led us to create the new Toolkit resource, “FAQ for nurses.” This resource provides a document for nurses to quickly scan to address frequently asked questions. In addition, participants recognized the potential of CONCERN to help with novice training and develop critical thinking skills. To address these comments, our team recognized the need to recommend the incorporation of CONCERN into onboarding and annual compliance training.

Developing early partnerships with hospital leadership was imperative to the progression of the implementation process and the success of the “CONCERN Go-Live.” Technical stakeholders reiterated that having an interdisciplinary core team with varied IT expertise readily available helped to ensure routine safety and compliance integration. Given that CONCERN is one of the first applications to be implemented and built externally to the EHR, there were various technical challenges in mapping it to FHIR standards and receiving the correct governance approvals. As our team reflected on our process navigating these challenges, we created a stakeholder checklist to help guide facilities in stakeholder engagement. In addition, our team developed a “Docker Bundle” (https://www.docker.com) to ensure a state-of-the-art and scalable installation solution allowing sites to download and setup the CONCERN system and all its dependencies regardless of the configuration of the existing target environment (i.e., EHR agnostic).

End-users, at both sites, discussed their experiences during the CONCERN implementation process, highlighting the local differences that each implementation site encountered. This further reinforced a need for the Toolkit to be rooted in best-practices and expert recommendations, while also allowing for the flexibility of local configurations. This led our team to create the Toolkit resource, “Local Site Checklist.” To further illustrate this point, consider the primary premise that engaging end-users is essential to ensuring uptake and use of CONCERN. In our interviews, one participant suggested that nurse educators and unit leaders in conjunction with clinical registered nurse (RN) unit champions disseminate CONCERN information to clinicians. While the Unit Champion model for communication and training was the approach that we originally used, future sites may choose to engage additional health care professionals to achieve this. Another practical detail would be the frequency in which study team members follow-up with clinician's postimplementation. At Site B, our team held virtual open Zoom drop-in sessions for unit-based clinicians daily for the 2 weeks immediate postimplementation, followed by 3 and 6 month in-person rounding. While each site will need to determine what is most effective based on resource allocation and hospital culture, we have created a Toolkit resource to help guide sites in this decision, “Best Practices on Post-Go Live Rounding Schedule.”

CONCERN is intended to increase the situational awareness of the interdisciplinary team regarding individual patient risk for clinical deterioration. CONCERN can be used as a supplement to escalate communication, as with RNs to prescribing providers, or a mechanism to clarify clinical thinking and reasoning, as is the case with expert preceptors and novice RNs. A RN also suggests educating nursing assistants on the utility of CONCERN as a mechanism to promote clinical communication and escalation (nursing assistant slide). Each user has a slightly different ways of incorporating CONCERN into their workflow, and the training materials need to reflect the nuances in application for different audiences.

A prior study demonstrated the effectiveness of qualitative methodology to validate toolkit resources.[12] This methodology helped our team to validate the resources in the α CONCERN Implementation Toolkit and create new resources to address gaps noted in our qualitative work. Future studies will focus on returning our findings in the α version of the Toolkit to the hospital sites (i.e., member checking) to validate the β version of the CONCERN Implementation Toolkit prior to dissemination.[19] Our findings will help us to scale and spread one of the first early warning systems driven by nursing data. While CONCERN was the use case for this study, we believe our findings could be generalized to the implementation process of other early warning systems in the acute care setting alongside further validation testing.[6] [10]

While this study did meet saturation at the conclusion of analysis, there was an imbalance in recruitment numbers with most clinicians being registered nurses employed at Site A. There were challenges with recruiting prescribing providers (e.g., nurse practitioners and physicians) limiting our ability to assess their perspectives on the implementation and the CONCERN Toolkit. In addition, while all participants had a knowledge of CONCERN, the level of engagement clinicians had with CONCERN and the implementation process was not measured. This study was conducted at two academic teaching hospitals. Future studies should explore the generalizability of these findings in alternative healthcare settings (e.g., rural hospitals and nonacademic medical centers)

Conclusions

This study aimed to validate resources in the α version of the CONCERN Implementation Toolkit which will enable the sustainability and spread of CONCERN to other hospital sites. Future studies will focus on validating the β version of the Toolkit through returning the results of the α version of the Toolkit to the hospital sites. Our findings highlight strengths and areas for future growth in our implementation process. As the development of early warning systems continues to increase, the results of this study provide considerations for research teams interested in implementing early warning systems in the acute care setting.

Clinical Relevance Statement

This study highlights the rigorous work conducted and needed to engage clinicians in the implementation process. Utilization of early warning systems will be at its strongest in partnership with clinicians. Our results highlight strategies and challenges to consider among future stakeholders interested in implementing early warning systems into practice.

Multiple Choice Questions

1. Stakeholders interested in implementing early warning systems should consider the:

   • Governance approvals required

   • Technical safety and security requirements needed

   • Education of unit clinicians

   • All of the above

   The correct answer is D. All of the above because implementation requires a multimodal approach partnering with unit staff, hospital leadership, and IT stakeholders.

2. What did clinicians in this study highlight as a benefit of the clinical decision support tool CONCERN? CONCERN's ability to…

   • Help with developing critical thinking skills for novice clinicians

   • Generate risk scores based on physician inputted data

   • Predict patients at risk for pressure injuries and falls

   • Create an acuity score for bed placement

   The correct answer is A. Help with developing critical thinking skills for novice clinicians. Concern is built using nursing data to predict the deterioration events sepsis and mortality.

• References

• 1 Morgan RW, Fitzgerald JC, Weiss SL, Nadkarni VM, Sutton RM, Berg RA. Sepsis-associated in-hospital cardiac arrest: epidemiology, pathophysiology, and potential therapies. J Crit Care 2017; 40: 128-135
• 2 Song MJ, Lee DS, Choi YY. et al. Incidence of preventable cardiopulmonary arrest in a mature part-time rapid response system: a prospective cohort study. PLoS One 2022; 17 (02) e0264272
• 3 Rhee C, Jones TM, Hamad Y. et al; Centers for Disease Control and Prevention (CDC) Prevention Epicenters Program. Prevalence, underlying causes, and preventability of sepsis-associated mortality in US acute care hospitals. JAMA Netw Open 2019; 2 (02) e187571-e187571
• 4 Schnock KO, Kang MJ, Rossetti SC. et al. Identifying nursing documentation patterns associated with patient deterioration and recovery from deterioration in critical and acute care settings. Int J Med Inform 2021; 153: 104525
• 5 Rossetti SC, Dykes PC, Knaplund C. et al. The Communicating Narrative Concerns Entered by Registered Nurses (CONCERN) clinical decision support early warning system: protocol for a cluster randomized pragmatic clinical trial. JMIR Res Protoc 2021; 10 (12) e30238
• 6 Davis MM, Howk S, Spurlock M, McGinnis PB, Cohen DJ, Fagnan LJ. A qualitative study of clinic and community member perspectives on intervention toolkits: “Unless the toolkit is used it won't help solve the problem”. BMC Health Serv Res 2017; 17 (01) 497
• 7 CONCERN Study. Columbia Department of Biomedical Informatics. Published 2023. Accessed December 5, 2023 at: https://www.dbmi.columbia.edu/concern-study/
• 8 Kang MJ, Rossetti SC, Knaplund C. et al. Nursing documentation variation across different medical facilities within an integrated healthcare system. Comput Inform Nurs 2021; 39 (12) 845-850
• 9 Implementation Toolkit Logic Model CONCERN. American Nurses Foundation. Accessed December 5, 2023 At: https://www.nursingworld.org/foundation/rninitiative/technology-enabled-nursing-practice/concern-implementation-toolkit/concern-implementation-toolkit-logic-model/
• 10 Thoele K, Ferren M, Moffat L, Keen A, Newhouse R. Development and use of a toolkit to facilitate implementation of an evidence-based intervention: a descriptive case study. Implement Sci Commun 2020; 1 (01) 86
• 11 Fall TIPS. A Patient-Centered Fall Prevention Toolkit. Agency for Healthcare Research and Quality. Published 2021. Accessed December 5, 2023 at: https://www.ahrq.gov/patient-safety/settings/hospital/fall-tips/index.html
• 12 Dykes PC, Khasnabish S, Burns Z. et al. Development and validation of a fall prevention efficiency scale. J Patient Saf 2022; 18 (02) 94-101
• 13 Murray E, May C, Mair F. Development and formative evaluation of the e-Health Implementation Toolkit (e-HIT). BMC Med Inform Decis Mak 2010; 10 (01) 61
• 14 Saunders B, Sim J, Kingstone T. et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant 2018; 52 (04) 1893-1907
• 15 Donabedian A. Evaluating the quality of medical care. 1966. Milbank Q 2005; 83 (04) 691-729
• 16 Moorman LP. Principles for real-world implementation of bedside predictive analytics monitoring. Appl Clin Inform 2021; 12 (04) 888-896
• 17 Kurtzman ET, Ghazal LV, Girouard S. et al. Nursing workforce challenges in the postpandemic world. J Nurs Regul 2022; 13 (02) 49-60
• 18 McDermott C. Reimagining the preceptor role. Nurs Adm Q 2023; 47 (03) 227-233
• 19 Birt L, Scott S, Cavers D, Campbell C, Walter F. Member checking: a tool to enhance trustworthiness or merely a nod to validation?. Qual Health Res 2016; 26 (13) 1802-1811