Shared-Task Worklists Improve Clinical Trial Recruitment Workflow in an Academic Emergency Department

Abstract

Background Clinical trials performed in our emergency department at Barnes-Jewish Hospital utilize a centralized infrastructure for alerting, screening, and enrollment with rule-based alerts sent to clinical research coordinators. Previously, all alerts were delivered as text messages via dedicated cellular phones. As the number of ongoing clinical trials increased, the volume of alerts grew to an unmanageable level. Therefore, we have changed our primary notification delivery method to study-specific, shared-task worklists integrated with our pre-existing web-based screening documentation system.

Objective To evaluate the effects on screening and recruitment workflow of replacing text-message delivery of clinical trial alerts with study-specific shared-task worklists in a high-volume academic emergency department supporting multiple concurrent clinical trials.

Methods We analyzed retrospective data on alerting, screening, and enrollment for 10 active clinical trials pre- and postimplementation of shared-task worklists.

Results Notifications signaling the presence of potentially eligible subjects for clinical trials were more likely to result in a screen (p < 0.001) with the implementation of shared-task worklists compared with notifications delivered as text messages for 8/10 clinical trials. The change in workflow did not alter the likelihood of a notification resulting in an enrollment (p = 0.473). The Director of Research reported a substantial reduction in the amount of time spent redirecting clinical research coordinator screening activities.

Conclusion Shared-task worklists, with the functionalities we have described, offer a viable alternative to delivery of clinical trial alerts via text message directly to clinical research coordinators recruiting for multiple concurrent clinical trials in a high-volume academic emergency department.

Keywords

alert fatigue - electronic health records - emergency department - clinical trials - workflow - clinical trial alerts - clinical trial recruitment

Background and Significance

While clinical trials are an important step to improving the quality and safety of health care delivery, participant recruitment is a difficult barrier for many studies.[1] The success of a clinical trial is linked to effective early enrollment.[2] In the 114 studies analyzed by Campbell et al in 2007, less than a third successfully recruited their target volume within the original timeframe.[3]

The use of commercial electronic health records (EHRs) facilitates opportunities to identify potential subjects for clinical trials through various automated means.[4] [5] [6] [7] EHR-based clinical trial alerts (CTAs), which notify clinical personnel when a patient's EHR data meet presumptive criteria for enrollment in a clinical trial, have been utilized for many years.[8] [9] [10] CTAs are associated with higher recruitment yield and have improved cost-efficiency of clinical trials by reducing the time research staff spend on patient identification and screening.[11] [12] [13] EHR-based CTAs may be delivered to various recipients by various methods and can be used in combination with other recruitment strategies.[14] [15] More recently, significant progress has been made using natural language processing and machine learning on unstructured text in EHRs to recognize study eligibility.[16] [17] [18] [19]

Recruitment for clinical trials conducted in the emergency department (ED) presents unique challenges due to the diversity of eligibility criteria, variable acuity, time constraints on clinical teams, concurrent recruitment for multiple clinical trials, and the need to identify subjects before the initiation of emergent interventions.[20] As clinicians and clinical staff rush between patient-care responsibilities, manual screening for eligibility for several ongoing clinical trials is quite impractical. Text-message alerts offer advantages of timeliness and automation. In the ED setting, text-message-based CTAs to dedicated cellular phones carried by treating physicians have improved recruitment.[21] [22] Screening by dedicated research staff, rather than by treating clinicians, has been shown to improve the screening and recruiting process for clinical trials within the ED.[23]

We have delivered EHR-based CTAs to clinical research coordinators (CRCs) for several years in our ED at an academic medical center. These alerts notify our CRCs when patients in the ED meet presumptive criteria for enrollment into any one of many ongoing clinical trials. Until recently, the alerts were sent as text messages directly to phones carried by our CRCs. As the frequency of alerts increased, the CRCs reported that it was difficult to manage the volume of text-message alerts and that they were often unsure which alerts were already being addressed by other CRCs working at the same time. In addition, the Director of Research (coauthor S.L.H.) dedicated increasing time and effort to monitoring the ED track board and notifications' list to redirect the CRCs' activities. It was realized that as the number of active clinical trials had grown, the volume of text-message CTAs had become overwhelming, and some of the CRCs resorted to searching for patients on the ED track board rather than sorting through and prioritizing the many individual text messages. Monitoring the screening log, the ED track board, and some of the alerts (which she received as e-mails) on her desktop PC, the Director increasingly noticed potential subjects for studies who were not being screened. She frequently found herself redirecting CRCs to promising potential subjects for whom there had been no CRC screening.

As a result, we converted to sending alerts to structured study-specific shared-task worklists within our existing screening-documentation system. Multiple users on different desktops may access the web-based system and complete tasks “live” from the shared set of lists. The alerts are organized by clinical trial—each trial having its own list. We avoid duplication of effort by allowing users to mark tasks as “handled” when screening is in progress. We describe the implementation and the impact of the worklist functionality on our clinical trial recruitment workflow.

Objective

To evaluate the effects on screening and recruitment workflow of replacing text-message delivery of CTAs with study-specific shared-task worklists in a high-volume academic ED supporting multiple concurrent clinical trials.

Methods

Setting and Population

This study was conducted at the ED of Barnes-Jewish Hospital, an academic medical center located in St. Louis, Missouri, United States. In 2019, Barnes-Jewish recorded 82,964 ED visits. Our hospital has used Epic EHR (Verona, Wisconsin, United States) since June 2018. Through the time of this study, the CRCs of the Washington University School of Medicine Emergency Care Research Core collectively worked (excluding paid time off), on average, a total of 37.7 hours/day.

Pre-existing Functionality of Rule-Based Alerts for Clinical-Trial Recruitment

Our Emergency Care Research Core has had a system to generate rule-based CTAs to notify research teams of potentially eligible subjects for enrollment in clinical trials in place since March 2010. Initially, this was a stand-alone system using a data feed from a dedicated emergency medicine information system (HMED by Allscripts, Chicago, Illinois, United States), but since June 2018 we have used Best Practice Advisories from within the Epic EHR. The CTAs typically include a visit identifier to allow CRCs to look up potential subjects in the medical record, patient location in the ED at the time of the alert, the name of the study for which the alert was generated, and a chief complaint. CTA rules evaluate the data in EHR records and send an alert when appropriate. When CRCs (registered nurses or paramedics) receive an alert, they typically first review the patient's EHR information and then, if it appears the patient may be eligible, they proceed to talk to the patient and/or physician to further screen the patient. The occurrence of a screening, whether or not the subject was enrolled, and the reason for exclusion if not, are documented through a custom-built, web-based clinical research screening-log application, external to the EHR. Clinical research staff may also screen patients for whom an alert was not generated but who were identified from the ED track board.

In the design of CTA rules, alert criteria are selected for each study. Triage data, laboratory and imaging orders, consultation orders, provider treatment orders, admission orders, and/or provider diagnostic assessments can all be used (in any combination) to initiate alerts. Many of these provider actions act as surrogates for the provider's differential diagnosis and assessment of severity, which are often not explicitly documented until later in the course of patient care.

Until recently, all notifications were sent in the form of text messages through an approved messaging application (Spok Paging Services, Eden Prairie, Minnesota, United States) to designated cellular phones carried by CRCs.

Shared-Task Worklist Implementation and Functionality

On January 8, 2020, a shared-task worklist format for message delivery was implemented ([Table 1]). Alerts from the EHR are sent as e-mails to a dedicated mailbox and are picked up by a custom e-mail listening service on our application server. The messages are formatted by the EHR to include a study code as well as a patient ID and patient location. The e-mail listener extracts this information from the e-mailed messages and sends it to our previously existing web-based screening-log application. The screening-log application has been modified to display the alerts in study-specific worklists, meaning that each clinical trial has its own list. The worklists include several highly useful functionalities for the ED setting. The list is part of our screening log application which uses a FHIR-like Epic API to pull in name and demographics on screened subjects. The CRC can enter the screening/enrollment function directly from an alert in a list. A CRC can select the “handle” button next to an alert to signal to other research team members that the notification is being addressed—thus avoiding duplication of effort. Multiple alerts can be marked as handled at once, which is useful for studies with quick screening and recruitment procedures. Once a screening is completed, it drops from the list. A feed from our HL7 ADT messaging system allows the application to regularly update potential subjects' locations within the department or hospital. When a patient leaves the ED, any unhandled notifications for that individual are either retained or removed based on how that particular study is configured.

Table 1 Key features of pre- and post-worklist functionality

Pre-worklist	Post-worklist
• High volume • Text alerts to individual cell phones • Study indicated in individual alerts • Patient location only at time of alert • Alerting/screening on different platforms	• High volume • Shared-task list • Alerts organized by study • Patient location updated in real time • Integration of alerts and screening log • In-progress status (“handled”) • Accountability (incomplete tasks evident)

Data Collection

Alert data were queried from the EHR reporting database (Epic Clarity on MS SQL Server) and merged with screening and enrollment data from our screening-log application database.

Study Selection and Analysis

Of the 27 clinical trials conducted since June 2018, only those with active alerting and enrollment (N = 10) during a 57-day period prior to and following shared-task worklist implementation (January 8, 2020) were selected for statistical analysis. Just after this 57-day time period (early March), our ED volume and clinical trial enrollment were drastically affected by the SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2) pandemic. We retrospectively selected equal 57-day time periods before and after the intervention to avoid any confounding effects on volumes related to SARS-CoV-2. During this selected time period, there were no changes to the criteria for these CTAs. Data were analyzed and tabulated as counts and percentages, and visualizations were created using ggplot2 in Rstudio software (Vienna, Austria).[24] [25] Fisher exact tests were used to compare the fractions of subjects with an alert who were screened or enrolled. p-Value significance thresholds for multiple comparisons were adjusted with Bonferroni correction. Pre/postcomparisons as odds ratios with 95% confidence intervals (CIs) were also calculated.

Results

Aggregate Alert, Screening, and Enrollment Data

The daily percentages of alerts with screenings and alerts with enrollments are shown for all studies beginning in June 2018 ([Fig. 1]). The 10 studies with active alerts and enrollment during the pre- and post-periods ([Appendix A]) were analyzed. These studies included five observational studies and five interventional studies. The topics of study included influenza (3), trauma (1), venous thrombus embolism (1), abscess (1), cellulitis (1), opioid use disorder (1), congestive heart failure (CHF, 1), and infection (1). The overall percentage of alerts with subsequent screenings increased from 2,465/6,394 (38.6%) to 4,942/6,239 (79.2%). Post/pre odds ratio = 2.05 (95% CI = 1.94–2.18; p < 0.001), while the percentage of alerts with subsequent enrollments was statistically unchanged at 221/6,394 (3.5%) and 231/6,239 (3.7%). Post/pre odds ratio = 1.07 (95% CI = 0.88–1.30; p = 0.473).

Fig. 1 Daily percentage of alerts with subsequent screening and enrollment for all studies. Worklist implementation is indicated with a solid vertical line, and the analysis period is bounded by dashed lines.

Appendix A Alerts, screenings, and enrollments over time for the 10 clinical trials included in this analysis. The density of the dots indicates the frequency of the events. Worklist implementation is indicated with a solid vertical line, and the analysis period is bounded by dashed lines.

Individual Studies

For 8/10 studies, there was a statistically significant increase in the percentage of alerts with screenings ([Table 2]). The percentages of alerts with enrollments were unchanged during the analysis period ([Table 3]).

Table 2 Screening percentages by study pre- and post-worklist implementation

Study	Screened/alert (pre)	Screened/alert (post)	p-Value	Odds ratio (95% CI)
O–Trauma	1,435/2,136 (67.2%)	1,615/1,994 (81.0%)	<0.001	1.21 (1.09–1.32)
O–VTE	359/1,080 (33.2%)	750/870 (86.2%)	<0.001	2.59 (2.22–3.04)
I–Flu 1	12/12 (100.0%)	87/93 (93.5%)	NA	0.936 (0.36–2.41)
I–Abscess	202/629 (32.1%)	430/549 (78.3%)	<0.001	2.44 (1.98–3.00)
O–Flu	9/71 (12.7%)	260/443 (58.7%)	<0.001	4.62 (2.25–10.70)
I–Cellulitis	111/695 (16.0%)	522/651 (80.2%)	<0.001	5.02 (3.97–6.38)
I–Flu 2	15/16 (93.8%)	44/44 (100.0%)	NA	1.07 (0.43–2.63)
O–Opioid	87/567 (15.3%)	388/599 (64.8%)	<0.001	4.22 (3.24–5.53)
I–CHF	41/141 (29.1%)	131/149 (87.9%)	<0.001	3.02 (1.95–4.72)
O–Infection	194/1,047 (18.5%)	715/847 (84.4%)	<0.001	4.55 (3.78–5.50)

Abbreviations: CI, confidence interval; I, interventional; O, observational.

Table 3 Enrollment percentages by study pre- and post-worklist implementation

Study	Enrolled/alert (pre)	Enrolled/alert (post)	p-Value	Odds ratio (95% CI)
O–Trauma	28/2136 (1.3%)	37/1,994 (1.9%)	0.171	1.41 (0.84–2.41)
O–VTE	60/1,080 (5.6%)	55/870 (6.3%)	0.501	1.14 (0.77–1.69)
I–Flu 1	0/12 (0.0%)	0/93 (0.0%)	NA	NA
I–Abscess	4/629 (0.6%)	2/549 (0.4%)	0.691	0.57 (0.05–4.01)
O–Flu	3/71 (4.2%)	14/443 (3.2%)	0.718	0.75 (0.20–4.16)
I–Cellulitis	0/695 (0.0%)	0/651 (0.0%)	NA	NA
I–Flu 2	0/16 (0.0%)	1/44 (2.3%)	NA	NA
O–Opioid	31/567 (5.5%)	28/599 (4.7%)	0.595	0.86 (0.49–1.49)
I–CHF	0/141 (0.0%)	0/149 (0.0%)	NA	NA
O–Infection	95/1,047 (9.1%)	94/847 (11.1%)	0.193	1.22 (0.90–1.67)

Abbreviations: CI, confidence interval; I, interventional; O, observational.

Discussion

To address the problems we experienced with increasing volume of alerts sent to cell phones, we changed our alert delivery method to shared worklists integrated into our existing screening documentation system. When we compared the 57-day period prior to and following implementation of the shared-task worklists, we saw an increase in the utilization of alerts for 8/10 clinical trials (see [Table 2]). The lack of statistical significance in the remaining two (I–Flu 1 and I–Flu 2) was likely due to a ceiling effect, as the screening rates for these studies were relatively high preimplementation. On the other hand, we did not see a change in the rates of enrollment.

The Director's anecdotal feedback and observations help place our quantitative results in perspective and provide additional insights. Since the implementation of the worklists, there has been significant turnover of the CRCs and some are no longer available, but the Director was able to provide anecdotal observations of her own as well as informal feedback from CRCs regarding the worklists. She retrospectively noted a significant change in the administrative time needed overseeing the screening activities of the CRCs from hours per day to minutes per day. Prior to the implementation of worklists, it was common that identification of subjects eligible for enrollment, especially in more complicated observational or interventional studies, occurred only after the potential subjects were identified by the Director and brought to the attention of CRCs. Some of the CRCs still managed to use the alerts by phone, but other CRCs reported that the presentation of the alerts on the phones was overwhelming and very difficult to manage. The Director agreed with this assessment and did not use a phone for her monitoring activities—rather she regularly scanned the track board and supplemented this with e-mailed alert notifications (identical to those sent to the phones) for those studies for which she had requested them. Some CRCs often screened mostly for simple, straightforward studies from the track board rather than using the alerts. They tended to avoid evaluating potential subjects for more complex studies. Those studies often require repeated follow-up on clinical progress prior to an enrollment decision. Additionally, CRCs reported hesitance to work alerts due to uncertainty whether another CRC, working simultaneously, was already addressing them. Although it was somewhat dependent upon which CRCs were working, it was common for the Director to rapidly prescreen the patients in the ED and then delegate lists of alerts to the CRCs for which to complete and document screenings. She did this many times throughout the day. Although as an emergency physician, the Director is very good at this, it was obviously not the ideal workflow and was a relative waste of her effort. Also, her prescreening effort was not being documented; although she had quickly scanned many patients for whom there were alerts, for the sake of efficiency she did not delegate all of these to the CRCs for full screening. Immediately after implementation of the worklists, the need for this effort markedly decreased. She now monitors the shared worklists and can very quickly determine how well the CRC team is addressing alerts. She has observed that the CRCs are consistently utilizing the lists to prompt screening, are more efficient with screening activities, and are enrolling many more subjects without her intervention. Anecdotally, the CRCs reported increased ease of screening, better communication among the research team regarding screening, and feeling less overwhelmed by the alert volume. Given these observations, we believe that the reason we did not see an increase in enrollment rates upon implementation of the worklists is that, preimplementation, the Director kept enrollment rates up despite the broken workflow. This supplementation of the CRC workflow came at the high and unsustainable cost of significant time and effort of the Director.

In our experience, individual alert delivery did not scale well as our research program grew to its current level. The presentation of alerts as shared worklists, with the additional functionalities as described, has resulted in a tremendous improvement in workflow. Despite the lack of prescreening and delegation activities of the Director, documented screening rates are up and enrollments have not decreased. The timing of this implementation was quite fortuitous in that the SARS-CoV-2 pandemic brought the opportunity and need for a rapid increase in the number of studies our group was recruiting for, many being SARS-CoV-2-related. With the shared worklists, the CRCs have a manageable workflow and the Director has had time to perform all of the study-design and management activities required to navigate the expansion and lead the team.

When first contemplating how to implement shared-task worklists, we did consider Epic EHR messaging to “in-basket pools.” This platform shares some but not all of the functionalities of our shared-task worklists. With Epic in-basket pools, the pool of team members shares an in-basket from which messages are eliminated as any member completes them. We opted to develop shared worklist functionality within our external system rather than using an Epic in-basket pool for several reasons, only some of which relate directly to alerting. First, we have built flexible management options for alerts on patients who leave the ED prior to being screened. For some studies, we drop discharged or admitted patients from the worklist when they leave (based on enterprise ADT messages), while for other studies we keep these messages on the worklist. This is useful for studies that allow for enrollment even after the ED visit. Second, we have incorporated the ability to mark an alert as “handled” while a CRC is working through screening one or more patients, letting other CRCs know that the alerts are being addressed. Epic in-basket messages are dropped from the shared list when completed but cannot be marked as being in-process. When first implementing Epic as our EHR, we considered performing all screening and enrollment within Epic, but decided against this. We have several functionalities built into our external screening log that would not be available in Epic. One issue, important to some of our ED studies, is that nonpatients who are eligible for participation in clinical trials cannot be easily entered into a study in the Epic EHR. We have had several studies in which visitors or health care providers are eligible to participate as controls. Maintaining our screening-log outside of the EHR allows screening/enrollment of nonpatients without initiating a time-consuming registration process in the EHR.

Limitations

There are many confounding variables in these data including variability in patient presentation, studies in which recruitment goals change over time (e.g., temporarily stopping recruitment for a study for various reasons, ending recruitment for one arm), and seasonal variation in some studies. Studies vary dramatically in terms of enrollments per screening. Some interventional studies included in this analysis recorded no enrollments during either the pre- or postimplementation time period (I–Flu 1, I–Cellulitis, I–CHF) which is not surprising given complex eligibility requirements and the relatively short study period. [Appendix A] demonstrates the seasonal variations in the prevalence of the influenza-related studies (I–Flu 1, I–Flu 2, O–Flu).

The study was unexpectedly limited by the onset of the SARS-CoV-2 pandemic. We had expected to select approximately equal before-and-after time periods around the implementation and planned to include all of the studies that were recruiting throughout those periods. ED census and screening/enrollment rates were abruptly and markedly affected as the pandemic began in our locality. We therefore limited the study period to an approximately symmetrical time period (before and after the implementation) limited by the onset of the pandemic. By the time we performed the data analyses of this study, there had been much turnover in the CRCs, in part related to the disruptions of the SARS-CoV-2 pandemic. When we discovered the unexpected findings of increased screening without an increase in enrollment, our attention turned to unmeasured workflow issues. However, we were then unable to obtain subjective data directly from the CRCs and had to rely upon the observations of the Director and her recollections of informal feedback from the CRCs. Finally, these findings are limited to a single academic ED.

Conclusion

Shared-task worklists, with the functionalities we have described, offer a viable alternative to unorganized, individual delivery of CTAs via text message to a team of CRCs in a high-volume academic ED supporting multiple concurrent clinical trials.

Clinical Relevance Statement

Our experience with delivery of CTAs by means of shared-task worklists in a high-volume academic ED demonstrates an efficient, scalable, and vendor independent alternative to direct messaging of CRCs or providers. For other similar clinical research settings, this approach may provide improved workflow for clinical trial screening and recruitment, enhancing research program effectiveness.

Multiple Choice Questions

1. With regard to clinical trial alerts in the emergency department,

   • The best way to avoid alert fatigue is to deliver the alerts directly to clinicians.

   • Clinical trial alerts can improve cost-efficiency of clinical trials.

   • Daily reports of potential subjects are better than real-time clinical trial alerts.

   • Clinical trial alerts always indicate a patient is eligible for a study.

   Correct Answer: The correct answer is option b. Responses to alerts sent directly to clinicians in the ED have been shown to decrease over time. In the emergency department, real-time clinical trial alerts improve cost-efficiency and result in higher recruitment yields. However, they cannot generally be specific enough to always indicate an eligible subject without losing adequate sensitivity.

2. Which of the following metric changes occurred with implementation of shared worklists in this implementation?

   • Enrollment in most studies increased.

   • The screening/alert rate decreased for most studies.

   • The screening/alert rate increased for most studies.

   • Enrollment in most studies decreased.

   Correct Answer: The correct answer is option c. Although we did not observe a change in enrollment, screening became much better aligned with alerts. Thus, screening of patients for whom there was an alert significantly increased.

Conflict of Interest

None declared.

Acknowledgments

We would like to thank the clinical research teams for their hard work and dedication in subject recruitment to the clinical trials discussed in this manuscript.

Protection of Human and Animal Subjects

This study was reviewed and deemed exempt by the Institutional Review Board (IRB) of Washington University School of Medicine.

• References

• 1 Ross S, Grant A, Counsell C, Gillespie W, Russell I, Prescott R. Barriers to participation in randomised controlled trials: a systematic review. J Clin Epidemiol 1999; 52 (12) 1143-1156
• 2 Haidich AB, Ioannidis JP. Patterns of patient enrollment in randomized controlled trials. J Clin Epidemiol 2001; 54 (09) 877-883
• 3 Campbell MK, Snowdon C, Francis D. et al; STEPS group. Recruitment to randomised trials: strategies for trial enrollment and participation study. The STEPS study. Health Technol Assess 2007; 11 (48) iii , ix-105
• 4 Köpcke F, Prokosch HU. Employing computers for the recruitment into clinical trials: a comprehensive systematic review. J Med Internet Res 2014; 16 (07) e161
• 5 Pressler TR, Yen PY, Ding J, Liu J, Embi PJ, Payne PR. Computational challenges and human factors influencing the design and use of clinical research participant eligibility pre-screening tools. BMC Med Inform Decis Mak 2012; 12: 47
• 6 Weng C, Batres C, Borda T. et al. A real-time screening alert improves patient recruitment efficiency. AMIA Annu Symp Proc 2011; 2011: 1489-1498
• 7 Effoe VS, Katula JA, Kirk JK. et al; LIFT Diabetes Research Team. The use of electronic medical records for recruitment in clinical trials: findings from the Lifestyle Intervention for Treatment of Diabetes trial. Trials 2016; 17 (01) 496
• 8 Embi PJ, Jain A, Clark J, Harris CM. Development of an electronic health record-based clinical trial alert system to enhance recruitment at the point of care. AMIA Annu Symp Proc 2005; 2005: 231-235
• 9 Butte AJ, Weinstein DA, Kohane IS. Enrolling patients into clinical trials faster using RealTime Recuiting. Proc AMIA Symp 2000; 2000: 111-115
• 10 Devoe C, Gabbidon H, Schussler N. et al. Use of electronic health records to develop and implement a silent best practice alert notification system for patient recruitment in clinical research: quality improvement initiative. JMIR Med Inform 2019; 7 (02) e10020
• 11 Lai YS, Afseth JD. A review of the impact of utilising electronic medical records for clinical research recruitment. Clin Trials 2019; 16 (02) 194-203
• 12 Embi PJ, Jain A, Clark J, Bizjack S, Hornung R, Harris CM. Effect of a clinical trial alert system on physician participation in trial recruitment. Arch Intern Med 2005; 165 (19) 2272-2277
• 13 Ni Y, Kennebeck S, Dexheimer JW. et al. Automated clinical trial eligibility prescreening: increasing the efficiency of patient identification for clinical trials in the emergency department. J Am Med Inform Assoc 2015; 22 (01) 166-178
• 14 Treweek S, Pearson E, Smith N. et al. Desktop software to identify patients eligible for recruitment into a clinical trial: using SARMA to recruit to the ROAD feasibility trial. Inform Prim Care 2010; 18 (01) 51-58
• 15 Beauharnais CC, Larkin ME, Zai AH, Boykin EC, Luttrell J, Wexler DJ. Efficacy and cost-effectiveness of an automated screening algorithm in an inpatient clinical trial. Clin Trials 2012; 9 (02) 198-203
• 16 Ni Y, Wright J, Perentesis J. et al. Increasing the efficiency of trial-patient matching: automated clinical trial eligibility pre-screening for pediatric oncology patients. BMC Med Inform Decis Mak 2015; 15: 28
• 17 Meystre SM, Heider PM, Kim Y, Aruch DB, Britten CD. Automatic trial eligibility surveillance based on unstructured clinical data. Int J Med Inform 2019; 129: 13-19
• 18 Shivade C, Hebert C, Lopetegui M, de Marneffe MC, Fosler-Lussier E, Lai AM. Textual inference for eligibility criteria resolution in clinical trials. J Biomed Inform 2015; 58 (Suppl): S211-S218
• 19 Ni Y, Bermudez M, Kennebeck S, Liddy-Hicks S, Dexheimer J. A real-time automated patient screening system for clinical trials eligibility in an emergency department: design and evaluation. JMIR Med Inform 2019; 7 (03) e14185
• 20 Cofield SS, Conwit R, Barsan W, Quinn J. Recruitment and retention of patients into emergency medicine clinical trials. Acad Emerg Med 2010; 17 (10) 1104-1112
• 21 Simon LE, Rauchwerger AS, Chettipally UK. et al. Text message alerts to emergency physicians identifying potential study candidates increase clinical trial enrollment. J Am Med Inform Assoc 2019; 26 (11) 1360-1363
• 22 Jegzentis K, Nowe T, Brunecker P. et al. Automated real-time text messaging as a means for rapidly identifying acute stroke patients for clinical trials. Trials 2014; 15: 304
• 23 Isaacman DJ, Reynolds EA. Effect of a research nurse on patient enrollment in a clinical study. Pediatr Emerg Care 1996; 12 (05) 340-342
• 24 Wickham H. ed. Elegant Graphics for Data Analysis. New York, NY: Springer-Verlag; 2016
• 25 R Core Team. A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2020

Publication History

Received: 10 November 2020

Accepted: 18 February 2021

Article published online:
07 April 2021