CC BY-NC-ND 4.0 · Appl Clin Inform 2021; 12(03): 417-428
DOI: 10.1055/s-0041-1730033
Review Article

Use, Impact, Weaknesses, and Advanced Features of Search Functions for Clinical Use in Electronic Health Records: A Scoping Review

Jordan R. Hill
1   Department of Medicine, Indiana University School of Medicine, Indianapolis, Indiana, United States
,
Shyam Visweswaran
2   Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pennsylvania, United States
,
Xia Ning
3   Department of Biomedical Informatics, The Ohio State University, Columbus, Ohio, United States
4   Department of Computer Science and Engineering, The Ohio State University, Columbus, Ohio, United States
5   Translational Data Analytics Institute, The Ohio State University, Ohio, United States
,
Titus K. Schleyer
1   Department of Medicine, Indiana University School of Medicine, Indianapolis, Indiana, United States
6   Center for Biomedical Informatics, Regenstrief Institute, Indianapolis, Indiana, United States
› Author Affiliations
Funding This project was made possible, in part, by support from the National Library of Medicine (Grant Nos. R01LM012605–01A1 and R01LM012095), the Agency for Healthcare Research and Quality (Grant No. 1R01HS027185–01A1), with support from the Indiana Clinical and Translational Sciences Institute (funded in part by Award Number UL1TR002529 from the National Institutes of Health, National Center for Advancing Translational Sciences) Clinical and Translational Sciences Award, and the Lilly Endowment, Inc. Physician Scientist Initiative. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors, and do not necessarily reflect the views of the funding agencies.
 

Abstract

Objective Although vast amounts of patient information are captured in electronic health records (EHRs), effective clinical use of this information is challenging due to inadequate and inefficient access to it at the point of care. The purpose of this study was to conduct a scoping review of the literature on the use of EHR search functions within a single patient's record in clinical settings to characterize the current state of research on the topic and identify areas for future study.

Methods We conducted a literature search of four databases to identify articles on within-EHR search functions or the use of EHR search function in the context of clinical tasks. After reviewing titles and abstracts and performing a full-text review of selected articles, we included 17 articles in the analysis. We qualitatively identified themes in those articles and synthesized the literature for each theme.

Results Based on the 17 articles analyzed, we delineated four themes: (1) how clinicians use search functions, (2) impact of search functions on clinical workflow, (3) weaknesses of current search functions, and (4) advanced search features. Our review found that search functions generally facilitate patient information retrieval by clinicians and are positively received by users. However, existing search functions have weaknesses, such as yielding false negatives and false positives, which can decrease trust in the results, and requiring a high cognitive load to perform an inclusive search of a patient's record.

Conclusion Despite the widespread adoption of EHRs, only a limited number of articles describe the use of EHR search functions in a clinical setting, despite evidence that they benefit clinician workflow and productivity. Some of the weaknesses of current search functions may be addressed by enhancing EHR search functions with collaborative filtering.


#

Background and Significance

Electronic information systems are essential to improving the health and health care of Americans.[1] Inherent in this statement is the belief that health care providers should have as much information as possible about their patients. The rapid adoption of electronic health records (EHRs)[2] [3] and health information exchanges[4] [5] has made patient information more available and accessible through the progressive conversion from paper-based to electronic formats. More than 95% of U.S. hospitals[6] and 79% of office-based physicians[7] use certified EHR technology,[6] which implies that most patient information is generated and recorded electronically. However, the efficient and effective use of this information remains challenging because of the limitations of tools used to access it at the point of care.

Although it is reasonable to assume that the amount of data captured per patient in EHRs is increasing, it is challenging to quantify electronic patient information growth precisely. In 2011, Beth Israel Deaconess Medical Center estimated generating 80 MB in text and image data per patient per year.[8] In data-rich environments like critical care, it is estimated that a patient generates an average of 1,460 new data points daily and a health care provider is exposed to an average of 4,380 data points during a shift of 12 hours.[9] [10]

As EHR usage expands to facilities beyond hospitals and outpatient clinics—such as nursing homes[11]—the amount of patient data recorded in EHRs is only expected to increase. Digital pathology data and images[12] and results from laboratories external to health systems[13] are examples of data that are increasingly captured by EHRs. Synergistic with this trend, interoperability among EHRs across institutions via health information exchanges adds to the volume of information per patient that clinicians can access.[14]

The increasing amount of patient information available to clinicians typically translates to more time spent in the EHR, which is compounded by the fact that accessing and reviewing patient information is taxing and time-consuming.[15] [16] [17] Clinicians spend a considerable amount of time searching, filtering, and collating a large volume of data to identify important details and ascertain clinically meaningful patterns.[18] Reviewing information in the EHR has been shown to take up a third of the duration of a patient encounter,[15] 12% of each physician hour of work,[16] and 20% of the total time physicians spend using EHRs.[19] This is a source of considerable frustration[18] [20] and often-cited criticism of EHRs,[21] [22] [23] [24] [25] which is exacerbated by confusing EHR layouts and workflows. These challenges in accessing relevant data impede the delivery of care, for instance, through clinicians' having to rely on their memory or the patient to obtain information instead of searching the EHR,[21] increased time pressure and stress,[22] loss of clinician productivity,[23] increased cognitive burden,[24] and addition of time to a physician's daily workflow.[23] [24] Although EHRs have increased the capture and availability of patient information, their full potential to support effective and efficient clinical decision-making is far from realized. One of the primary sources of frustration for EHR users is the difficulty of accessing data efficiently.[18] [20] [21] [22] [23] [24] [25]

One potential solution to the challenge of inadequate and inefficient accessibility of patient information is the development of search capability within EHRs. Zalis and Harris found that EHR searching consumes 20% of a radiologist's diagnostic effort for each case, and that having to remember many concepts and terms when searching patient records imposes a heavy cognitive burden.[24] The authors discuss how search functions could support multiple clinical decision-making scenarios effectively and efficiently.

The cognitive requirements mentioned by Zalis and Harris[24] to ensure all relevant concepts are searched within a patient's EHR are nontrivial. It follows that advanced features within an EHR search function could alleviate some of this cognitive load by, for example, providing search term recommendations to clinicians. One method to generate such recommendations is collaborative filtering, the process of filtering or evaluating items and making suggestions using the opinions or behavior of the user and/or others.[26] The widespread, long-term, and successful use of collaborative filtering on Web sites such as Amazon[27] and YouTube[28] suggests that it could be adapted to help clinicians retrieve relevant patient data in EHRs.

We found no existing literature reviews on search functions within EHRs in clinical settings. This lack of summarized knowledge on the topic was the primary rationale for this review.


#

Objective

The purpose of this study was to conduct a scoping review of the literature on the use of EHR search functions within a single patient's record in clinical settings to characterize the current state of research on the topic and identify areas for future study. We were particularly interested in research on the process and impact of clinicians' use of search functions, and any applications of collaborative filtering.


#

Methods

Due to the lack of existing literature reviews on this topic and our desire to broadly summarize the clinical effects of EHR search functions, we determined that a scoping review was appropriate. Based on the methodology outlined by Arksey and O'Malley,[29] we performed this scoping review in four steps: (1) identify relevant articles, (2) select articles, (3) identify themes, and (4) summarize and report findings. We describe the first two steps here. In the “Results” section, in addition to the results from the first two steps, we provide descriptions of the qualitative syntheses of the selected articles organized by themes. Additionally, we report the information outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Checklist developed by Liberati et al[30] when applicable.

Identification of Relevant Articles

We searched for relevant journal articles and conference proceedings in four databases: Applied Science and Technology, IEEE Xplore, PubMed, and Web of Science. Since we wanted to capture as many relevant articles as possible, we developed a search strategy focused on high recall at the expense of precision. The search query included the term “search” appearing in the title or abstract along with one or more of the following terms: “EHR,” “electronic health record(s),” “EMR,” “electronic medical record(s),” or “computerized medical record(s).” The Web of Science database does not have a title or abstract search field, so we used the “Topic” search field that included the title, abstract, and article keywords. The database-specific fields we used are summarized in [Table 1].

Table 1

Databases, fields, and query used in the search

Database

Fields

Query

Applied Science & Technology

Title/abstract

Search AND [EHR OR electronic health record(s) OR EMR OR electronic medical record(s) OR computerized medical record(s)]

IEEE Xplore

Title/abstract

PubMed

Title/abstract

Web of Science

Topic

We included only English-language journal articles and conference proceedings, published on or before December 2020. One of the authors (JRH) used the Mendeley Reference Manager to identify and remove duplicates.


#

Selection of Articles

We reviewed the titles and abstracts of unique articles obtained in the first step to identify articles for a full-text review (see [Fig. 1]). We selected articles that studied within-EHR search functions and their implementations, or examined the use of EHR search functions in the context of clinical tasks. We included articles that described studies:

  • focusing on qualitative, subjective user data;

  • using quantitative or objective measures of user performance;

  • comparing different designs or features of search functions; and/or

  • describing how users interact with search functions to identify user needs.

Zoom Image
Fig. 1 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram of the article screening process.

We excluded articles that described the following:

  • theoretical search systems or algorithms that were proposed but not evaluated in an actual or simulated clinical setting with users;

  • systems or algorithms that were only validated in a nonclinical way (e.g., statistical measures of accuracy);

  • studies that focused on using search functions to search across large numbers of patient records as opposed to searching within a single patient's EHR (e.g., research cohort identification and data mining); and/or

  • use of search functions to provide users with information from external information sources, such as PubMed.

Due to the broad search criteria used, many of the returned articles were not about the use of search functions within EHRs (e.g., an abstract that contains both “search” and “electronic health record” is not often about search functions within EHRs). Other articles on information retrieval in EHRs described new search algorithms without clinical validation, functionality for searching across large numbers of patient records, and/or search functions that were not developed to retrieve patient data. Articles that were selected for full-text review but were ultimately rejected are listed, along with reasons for rejection, in [Appendix Table A1].

Appendix Table A1

Articles that were screened but excluded from the review

Study

Reason(s) for exclusion

Alafaireet et al 2017

Focuses on providing medical information resources to health care providers instead of searching within a single patient's record during the clinical workflow

Biese et al 2013

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Cairns et al 2011

A system is proposed and validated but not evaluated in a clinical or clinical-like setting. Focuses on providing medical information resources to health care providers instead of searching within a single patient's record during the clinical workflow

Costumero et al 2014

An algorithm is proposed but not evaluated in a real or simulated clinical setting

Daumke et al 2010

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Divita et al 2016

A system is proposed but not evaluated in a real or simulated clinical setting

Dos Reis et al 2016

An algorithm is proposed but not evaluated in a real or simulated clinical setting

Edinger et al 2017

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow. An algorithm is proposed and validated but not evaluated in a real or simulated clinical setting

Epstein et al 2010

Focuses on providing medical information resources to health care providers instead of searching within a single patient's record during the clinical workflow

Farfan et al 2009

An algorithm is proposed but not evaluated in a real or simulated clinical setting

Garcelon et al 2017

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow. An algorithm is proposed and validated but not evaluated in a real or simulated clinical setting

Garcelon et al 2020

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Gong and Zhang 2009

Focuses on health data presentation to the user instead of using a search function

Goodwin and Harabagiu 2018

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Gubanov and Pyayt 2012

A system is proposed and validated but not evaluated in a real or simulated clinical setting

Hammond et al 2013

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Hanauer et al 2009

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Hanauer et al 2017

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Hultman et al 2017

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Kreuzthaler et al 2015

An algorithm is proposed but not evaluated in a real or simulated clinical setting

Kukhareva et al 2017

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Lelong et al 2017

A system is proposed and validated but not evaluated in a real or simulated clinical setting

Liu et al 2009

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Lopez-Garcia et al 2016

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Martinez et al 2014

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow. An algorithm is proposed and validated but not evaluated in a real or simulated clinical setting

Masseroli and Marchente 2008

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow. An algorithm is proposed and validated but not evaluated in a real or simulated clinical setting

Moen et al 2015

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Rinner et al 2012

A system is described but not evaluated in a real or simulated clinical setting

Schulz et al 2008

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Seyfried et al 2010

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Sonntag and Profitlich 2019

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Tsai et al 1999

Focuses on providing medical information resources to health care providers instead of searching within a single patient's record during the clinical workflow. A system is described but not evaluated in a real or simulated clinical setting

Vasanthakumar and Bond 2018

An algorithm is proposed and validated but not evaluated in a real or simulated clinical setting

Wu et al 2018

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Yu and Yilayavilli 2009

A system is described but not evaluated in a real or simulated clinical setting

Zalis and Harris 2010

Discusses usefulness of search features within electronic health records (EHRs) but does not test or propose a specific search function

Zampi et al 2012

Focuses on searching across large numbers of records instead of within a single patient's record during the clinical workflow

Zhang et al 2013

An algorithm is proposed and validated but not evaluated in a real or simulated clinical setting

Zhao et al 2012

An algorithm is proposed but not evaluated in a real or simulated clinical setting

Zhu and Carterette 2012

An algorithm is proposed and validated but not evaluated in a real or simulated clinical setting

We also searched the four databases for collaborative filtering interventions within the context of clinical tasks. No articles were found.


#
#

Results

We identified a total of 5,002 entries from all databases that decreased to 2,966 unique articles after duplicates were removed (see [Fig. 1]). After screening the titles and abstracts, we selected 57 articles for in-depth review. After full-text review, we excluded 40 articles because they did not describe search functions within an EHR and/or did not investigate the use of search functions for clinical tasks. The 17 articles selected for inclusion in this scoping review are listed and summarized in [Table 2].

Table 2

Articles included in analysis

Study

Details

Bamnote and Agrawal[32]

Analyzed user logs of an electronic health record (EHR) search function and classified queries as informational, navigational, or transactional; looked at how different types of users use different kinds of queries

Biron et al[34]

Implemented EHR search function in a cancer treatment hospital in France; discusses details of the tool and its use for both medical care and research

Duftschmid et al[36]

Evaluated a within-document search function with diabetes specialists to see if they could find all required pieces of information in a patient's record within a time limit (with and without search)

Duke et al[35]

Describes usage of an EHR search function that allows searching of different orders to place

Garcelon et al[33]

Describes usage of a data warehouse that allows searching within medical documents (within and across records) and its reception among users

Hanauer[39]; Hanauer et al[42]; Yang et al[43]; Zheng et al[44]; Zheng et al[45]

Relate to clinical aspects of EMERSE (collaborative search bundles, query analysis, personalization usage)[a]

Hasan et al[38]

Brazilian Portuguese semantic search engine evaluated on clinical summarization tasks to determine effect on the time taken to perform tasks by two medical students and two nursing students

Kovacs et al[40]

Noninstantaneous search engine for radiologists to search the EHR for retrieving data for clinical follow-up

Moen et al[41]

Finnish-language search tool assessed by three subject matter experts regarding whether it made finding information in the EHR easier/quicker, etc.

Natarajan et al[31]

Examined search queries used in EHR search functions to understand user information needs

Ruppel et al[18]

Assessed EHR users' searches within individual patient records over 13 mo to understand information needs

Tawfik et al[37]

Gave participants realistic search scenarios in an EHR to complete with and without a search function to evaluate its usefulness

Ye and Fabbri[46]

Created and evaluated an algorithm to highlight “related terms” to expand keyword search within the EHR (compared with exact match only highlighting)

a The Electronic Medical Record Search Engine (EMERSE) has both clinical and research applications; for this review, we excluded articles discussing only the research applications of EMERSE.


Based on findings reported in the 17 articles, we delineated four themes to describe the features and effects of search functions in clinical use: (1) how clinicians use search functions, (2) impact of search functions on clinical workflow, (3) weaknesses of current search functions, and (4) advanced search features.

How Clinicians Use Search Functions

How search functions are used varies depending on the user and the clinical task.[18] [31] [32] Multiple articles explored how EHR search functions are used, by whom, and how often.

Ruppel et al analyzed EHR search data from a large integrated health care system to characterize clinicians' information needs within an EHR.[18] In this study, 35.7% of all users and 75.6% of physician users conducted at least one search using the EHR search function, and physicians and pharmacists were the most active search function users.

Several articles assessed and classified common types of searches performed with an EHR. Most searches performed within a patient's record in a French children's hospital were to identify specific medical events, treatments, and laboratory test results.[33] Biron et al observed similar search behavior in their EHR full-text search tool at a cancer treatment facility.[34] Common queries comprised medications, processes or stages involved in medical care (e.g., chemotherapy, radiotherapy, etc.), comorbidities, and laboratory test results.

Bamnote and Agrawal analyzed user log files of a general search utility (CISearch) to classify types of queries.[32] Types included navigational (e.g., searching for a patient's medical record number), transactional (an action, e.g., “add note”), or informational (searching for specific data on a patient). Their results showed that 91.8% of searches made by physicians were informational. Administrative users performed a balance of navigational and informational searches. Of note, not all search engines provide both navigational and transactional functions described in this study, so information on these functions may have limited relevance to other types of search systems.

Regenstrief Institute's Gopher EHR system documented in Duke et al[35] differs slightly from some of the other search functions described as it allows users not only to search within a patient's EHR for mention of a phrase but also to place orders (e.g., a laboratory test) via a search box. This article describes how the Gopher system was used in a clinical setting. The most common types of orders input into the search box were for medications (38%), consultations (14%), call orders (12%), and laboratory tests (8%). These orders coincide with some of the commonly searched-for terms in previously mentioned studies.[33] [34]


#

Impact of Search Functions on Clinical Workflow

Studies investigating the impact of search functions on clinical workflow employed several data collection methods, including objective and subjective, qualitative, and quantitative methods.

Laboratory studies that simulated a clinical setting and provided clinical users tasks to evaluate a search function's time savings were the most common types of evaluations. In three studies, search functions that identified keywords within EHR documents were shown to increase the speed at which participants found relevant data (compared with not having a search function or having only a document-level search function),[36] [37] [38] decrease the number of clicks required to find data,[37] and reduce participant cognitive load.[37]

To elicit opinions of users of their data warehouse, Garcelon et al[33] used a survey designed to evaluate the Electronic Medical Record Search Engine (EMERSE).[39] Of the 59 respondents, 96.6% said the data warehouse enabled effective searching, 94.9% said it solves problems or facilitates tasks, 96.6% said it saves time, 78% found answers given by the search function were accurate, 86.4% said the search function found data they may have otherwise missed, and 79.7% said it expanded their ability to do chart reviews (the article did not distinguish between reviews for a single patient and for identifying a study cohort).

Biron et al provided only general impressions from clinical users of their full-text search tool.[34] Their study found that the clinicians strongly approved of the tool, and said they found the searches quick and time-saving. Kovacs et al described an integrated search function in a clinical setting and focused mainly on the system's performance; 82% of the 170 queries that returned results over 22 months were clinically relevant.[40]


#

Weaknesses of Current Search Functions

Several articles identified weaknesses and challenges with search functions and explored approaches to improving them. Two articles described mapping terms in queries to Unified Medical Language System terms to develop concept-based search terms for better retrieval.[31] [32] However, the complexity of EHR language and the presence of acronyms, abbreviations, and incomplete words limited the concept-based approach. One suggested approach is to use a combination of free-form and concept-based terms for querying.[32] Other challenges to implementation of search functions are the multitude of synonyms for medical concepts, that negations or mentions in family history are often presented (false positives),[34] that it is difficult for users to know something is not in a patient's record—despite negative search results (false negatives),[41] and that some users want a summary of medication changes over time, which is not adequately presented in many search results.[18]

Hanauer et al[42] and Yang et al[43] discuss how it is difficult for clinicians (even those with significant experience) to generate a set of minimally necessary search terms to ensure a reasonably inclusive search of the patient's record and to properly formulate their data needs with appropriate queries.


#

Advanced Search Features

To address the cognitive load imposed on clinicians when developing queries for an EHR search engine, Hanauer et al[42] and Zheng et al[44] integrated collaborative “search bundles” into University of Michigan's EMERSE system. These bundles were created by users to save a collection of search terms under a certain label—for instance, a search bundle labeled “animal bites” including terms such as “dog bite,” “cat bite,” “puncture wound,” and different permutations of “irrigate.”[42] This would allow a search containing many terms to be executed quickly and reduce the cognitive load on users. The “collaborative” aspect of collaborative search bundles is derived from users' ability to share the search bundles they created. Similarly, Duftschmid et al provided predefined queries for users to reduce cognitive load when searching.[36]

Whereas Hanauer et al stated that users embraced the EMERSE collaborative search feature more than anticipated,[42] Zheng et al performed a more in-depth analysis of the usage of collaborative search bundles over 4 years.[44] During this time, there were 451 registered EMERSE users (62.7% academic researchers and 21.6% practicing clinicians) and nearly 1 million searches performed (searches across patient records were not separated from those within a single patient's record). Approximately half of these searchers were assisted by search bundles, and of those bundle-based searches, 35.8% used a search bundle made available by another user. Most of the bundle sharing happened privately (73.8%), likely within individual research groups. More than half of the 702 search bundles created were made available to other users (34.5% to private groups and 20.5% publicly). Of all the EMERSE users, 188 created at least one search bundle, and half of those shared their bundles with others. Seventy-seven bundle users, the so-called bundle leechers, never created a bundle or shared one with others. Zheng et al also identified some problems with collaborative search bundles used in EMERSE.[44] Most search bundles were shared only with others in the same department, and there was evidence that users entered search terms in a search bundle but did not use the available bundle. This implied a lack of awareness regarding existing search bundles and their applicability to users in other departments, which in turn limited the potential of the system. Additionally, creating search bundles was time-consuming and there was no way of ensuring that bundles were of high quality.

Another study by Zheng et al examined whether EMERSE users took advantage of the software personalization options available to them (text highlighting options, color palettes, etc.).[45] They found that the vast majority of system users did not use the personalization options available, implying that most customization options were not worth including in a search feature.

Two studies describe the effects of implementing an advanced feature in a search function that presents exact matches to keywords used and results that are close to or related to the keyword. In Moen et al's study, evaluators said the tool made it easier and faster to find information within the EHR; whereas some results were not relevant, the evaluators found they were easy to ignore.[41] Ye and Fabbri's study timed how long it took two medical researchers to determine if patients had had dialysis within 2 weeks of surgery.[46] They found that highlighting exact matches and related terms reduced the time required compared with highlighting only exact matches to the search terms.


#
#

Discussion

Given the relatively small number of articles we found (17 papers from 2006 to 2020), the first implication of our findings is that the development, use, and study of search functions in EHRs remain in their infancy. Considering the burden imposed on users by poor accessibility of patient information and the potential of search functions to alleviate it,[24] this is a significant gap. Only a small number of articles in our review studied within-EHR search functions or examined their use in the context of clinical tasks, despite evidence that clinicians generally support integrating a search function in an EHR system, and doing so improves clinician speed and ability to retrieve relevant patient information. However, despite this evidence and the widespread adoption of EHRs, both in the United States and globally, few articles describe within-EHR search function implementations and effects in actual or simulated clinical settings.

Our review identified four themes: (1) how clinicians use search functions, (2) impact of search functions on clinical workflow, (3) weaknesses of current search functions, and (4) advanced search features. Different types of users use EHR search functions differently. This reality highlights the importance of search functions being designed to accommodate the needs of different kinds of users, including those who are not physicians, or physicians who work in various specialties and have differing needs. It also highlights the need for objective assessment of the effectiveness of a search function with comparisons across different tools, different users, and different specialties. The development of a standardized method for assessing the effects (positive and negative) of a search function would be valuable in this area of research. The publication of desiderata for EHR search functions is also an important area of future work to better inform the design of future systems.

Despite the differences in how search functions are used, there are common categories of most searched-for terms. Medical events, patient treatment/medications, laboratory test results, and allergies were found to be among the most common types of searches performed by users, implying that these areas are where clinical users perceive the most benefit from a search function. It is therefore important that developers of EHR search functions understand the needs of their users and ensure that, at minimum, search systems effectively enable searching for terms in these categories. If a search system is being developed for a specific group of users, it would be beneficial that system designers and developers take the time to fully understand the needs of their users.

In the articles we reviewed, challenges were identified with more basic, keyword-identifying search functions. For example, different permutations (spelling, acronyms, synonyms, etc.) of a term or concept carry the danger of missing relevant information. Negations can be presented to the user as false positives. Integration of artificial intelligence and natural language processing approaches in EHR search functions may help decrease false positives and should be considered for future research in this area. Additionally, although some articles specifically mentioned free-text searching and others discussed search functions using more structured data, it was not an explicit focus of the majority of the articles. The types of data being searched would likely affect the strategies required to ensure better search function performance.

Many EHR systems do not allow for search results to be presented as values changing over time (e.g., medication dosage), a feature that would be beneficial in understanding a patient's medical history. This demonstrates a need to optimize not only the design and implementation of search functions within EHRs but also the means by which information is presented in EHRs generally. Enabling users to view information quickly and in a way that is most useful to their clinical needs (e.g., variables changing over time) could eliminate arduous searches of a patient's record. For example, simply having clinical notes organized consistently can minimize the time clinicians spend reviewing them.[47]

Interoperability of EHR systems (e.g., data collected from different health care institutions) is also an important factor to consider when looking at designing an EHR search function. If the search function can mine both data collected within the native-EHR system and data collected in another system, it may be useful to differentiate between those different data types when provided search results to users.

Many of the difficulties with existing search functions in real and simulated clinical settings can be summarized by the fact that creating an inclusive search that will identify all necessary information in a patient's record while filtering out false positives is nontrivial. Different sources discussed and experimented with different ways of overcoming this problem.

Multiple studies explored advanced search function features that returned exact matches to search terms as well as related terms.[41] [46] Although this has been shown to be beneficial in alleviating some of the cognitive load on clinicians to developing inclusive searches, this strategy would require a significant amount of initial effort to map terms to others that are related, and these mappings would need to be updated regularly. Providing predefined queries to clinical users, as suggested by Duftschmid et al,[36] would present similar challenges. However, existing work on mapping clinical concepts can help inform the development of search term clusters.[48]

The most widely known system attempting to address the challenge of creating inclusive searches is collaborative search bundles developed for EMERSE.[42] [44] Whereas the publications on these search bundles have demonstrated their effectiveness, there were still challenges in implementation, such as users who used bundles made by others but did not contribute their own (“bundle leechers”), lack of awareness of bundles among EMERSE users, the time-consuming nature of developing bundles, and the lack of quality control to ensure bundles were effective.

Collaborative search bundles in EMERSE, however, demonstrate that collaborative searching has significant potential: it allows the expertise of multiple individuals to be shared, improved upon, and used by others to streamline searching through a patient's record and reduce cognitive load. At the same time, collaborative searching, ideally, should be implemented to ensure the following:

  • users know that collaborative searching resources are available;

  • resources are easy for users to find and access;

  • resources are effective; and

  • resources do not require significant setup time from busy clinicians.

The potential of collaborative search led us to search for collaborative filtering algorithms implemented within EHR systems in clinical settings. However, this search yielded no results. Collaborative filtering algorithms, properly implemented, could address many of the challenges identified. An item-based collaborative filtering algorithm could recommend search terms based on a term already entered by a user (therefore identifying potential synonyms or acronyms for the same or related concepts), and a user-based collaborative filtering algorithm could adjust those recommendations based on user characteristics (e.g., type of clinician, field of work). Essentially, such an algorithm could replace collaborative search bundles in alleviating some of the cognitive load associated with creating an inclusive search. Also, the algorithm would not require any explicit input from users as it would gather information and make recommendations automatically based on the actions of other users. An algorithmic strategy that makes recommendations based on the searches of other users of the same system can be extended to other kinds of EHR search functions (e.g., document-level searches vs. keyword searches). If properly implemented, the function could be integrated within the search field (similar to suggested searches on Google or Amazon), potentially saving time and effort for users.

Collaborative filtering algorithms have been proposed for medical and health care settings,[49] [50] [51] [52] [53] [54] [55] [56] [57] [58] although none have been integrated into a clinical setting. Some proposed collaborative filtering interventions use patient EHR data (e.g., to support rare disease diagnosis[51]). Recently, an order recommender system using collaborative filtering was tested with physicians in a laboratory setting and positively received by participants.[57] [58] Additionally, one article and a brief conference proceedings paper propose using collaborative filtering to support EHR search and discuss the methods by which such a system could be developed.[55] [56] We believe that implementing collaborative filtering within EHR search functions is a promising direction for future work.

This study had some limitations. By limiting our search to four databases, it is possible that articles relevant to this topic were not found and included in this review. It is also possible that relevant articles that used keywords other than those used were missed in our search. We did not limit article inclusion in this review by the language used in the EHR search function (though the article had to be available in English), though we acknowledge that language can affect search function use, impact, and efficiency. An investigation into the effects of different languages and countries on EHR search functions is an interesting topic for future investigation. Additionally, it is clear from our results that search functions have been developed for use within EHRs for over a decade. We did not, however, investigate how search functions have evolved over this time period, and this is a useful topic for a future review.

Though we focused only on clinical applications of search functions within EHRs, many articles identified by our search described theoretical EHR search systems or assessed search systems' performance outside of clinical settings. Whereas the information contained in these sources is likely valuable, it is beyond the scope of this review. Additionally, we only searched for academic publications describing clinical implementations and results of EHR search functions. We did not look beyond academic databases to determine which specific EHR systems had implemented search functions, how they work, and how they affected users' workflow.


#

Conclusion

Considering the widespread adoption of EHR systems, a surprisingly limited number of research articles have analyzed the use, impact, and weaknesses of existing search functions for clinical tasks. The articles that explored this topic showed that when a search function is available, it is generally positively received by users and usually improves clinician workflow by reducing the time required to find information or review a record. Future research is needed in the development, implementation, and evaluation of efficient search functions in EHRs.


#

Clinical Relevance Statement

There are several major clinical implications of this study's results. According to the literature analyzed, search functions are used differently depending on user and task, and users find it difficult to construct an inclusive search. The complexity of the language (e.g., acronyms, abbreviations, and partial-word searches) in medical records makes it challenging to standardize search queries that are useful and efficient to improve the performance of search functions in EHRs. Integrating collaborative filtering is a potential solution to some of the challenges in using EHR searches in clinical settings. Collaborative filtering has been used in medical settings and using EHR data, but we found no article demonstrating its use in an EHR search function. EHR search functions would likely benefit from combining aspects of both item- and user-based collaborative filtering.


#

Multiple Choice Questions

  1. Which of the following is not among the categories of most searched-for terms within EHRs?

    • Medical events.

    • Allergies.

    • Family medical history.

    • Treatment/medications.

    Correct Answer: The correct answer is option c. Medical events, patient treatment/medications, laboratory test results, and allergies were found to be among the most common types of searches performed by users.

  2. To whom does the term “bundle leechers” refer when considering EMERSE collaborative search bundles?

    • Users who use bundles made by those in other departments.

    • Users who use bundles made by others without contributing their own.

    • Users who slowly and continuously create search bundles over a period of time.

    • Users who share their bundles with others.

    Correct Answer: The correct answer is option b. EMERSE collaborative search bundle users who use bundles made by others but did not contribute their own were referred to as “bundle leechers.”


#
#

Conflict of Interest

J.R.H. has nothing to disclose. S.V. and X.N. report grants from NIH/NLM during the conduct of the study. T.K.S. reports grants from NIH/NLM and AHRQ during the conduct of the study. In addition, T.K.S. and X.N. have a patent related to collaborative filtering pending.

Acknowledgments

We gratefully acknowledge the efforts of Lynn Whittaker, Emily Fortier, and researchers affiliated with Regenstrief's Learning Health Informatics Program in the preparation of this manuscript.

Protection of Human and Animal Subjects

No human or animal subjects were involved in this study.


  • References

  • 1 Blumenthal D. Stimulating the adoption of health information technology. N Engl J Med 2009; 360 (15) 1477-1479
  • 2 The Office of the National Coordinator for Health Information Technology. Non-Federal Acute Care Hospital Electronic Health Record Adoption. Healthit.gov. 2017 . Accessed December 28, 2020 at: https://dashboard.healthit.gov/quickstats/pages/FIG-Hospital-EHR-Adoption.php
  • 3 The Office of the National Coordinator for Health Information Technology. Office-Based Physician Electronic Health Record Adoption. Healthit.gov. 2017 . Accessed December 28, 2020 at: https://dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php
  • 4 Pylypchuk Y, Johnson C, Patel V. State of Interoperability among U.S. Non-Federal Acute Care Hospitals in 2018. 2020 . Available at: https://www.healthit.gov/sites/default/files/page/2020-03/State-of-Interoperability-among-US-Non-federal-Acute-Care-Hospitals-in-2018.pdf
  • 5 The Office of the National Coordinator for Health Information Technology. Connecting Health and Care for the Nation: A Shared Nationwide Interoperability Roadmap. Available at: https://www.healthit.gov/sites/default/files/hie-interoperability/nationwide-interoperability-roadmap-final-version-1.0.pdf
  • 6 Ross MK, Wei W, Ohno-Machado L. “Big data” and the electronic health record. Yearb Med Inform 2014; 9 (01) 97-104
  • 7 Centers for Disease Control and Prevention. Electronic Medical Records/Electronic Health Records (EMRs/EHRs). National Center for Health Statistics. 2010 . Accessed March 12, 2021 at: https://www.cdc.gov/nchs/fastats/electronic-medical-records.htm
  • 8 Halamka J. The Cost of Storing Patient Records. GeekDoctor. 2011 . Accessed December 14, 2020 at: http://geekdoctor.blogspot.com/2011/04/cost-of-storing-patient-records.html
  • 9 Gal DB, Han B, Longhurst C, Scheinker D, Shin AY. Quantifying electronic health record data: a potential risk for cognitive overload. Hosp Pediatr 2021; 11 (02) 175-178
  • 10 Donovan F. Organizations See 878% Health Data Growth Rate Since 2016. HIT Infrastructure. 2019 . Accessed December 14, 2020 at: https://hitinfrastructure.com/news/organizations-see-878-health-data-growth-rate-since-2016
  • 11 Munyisia E, Yu P, Hailey D. The effect of an electronic health record system on nursing staff time in a nursing home: a longitudinal cohort study. Australas Med J 2014; 7 (07) 285-293
  • 12 Rojo MG, Castro AM, Gonçalves L. COST action “EuroTelepath”: digital pathology integration in electronic health record, including primary care centres. Diagn Pathol 2011; 6 (01, Suppl 1): S6
  • 13 Wilkerson ML, Henricks WH, Castellani WJ, Whitsitt MS, Sinard JH. Management of laboratory data and information exchange in the electronic health record. Arch Pathol Lab Med 2015; 139 (03) 319-327
  • 14 Politi L, Codish S, Sagy I, Fink L. Balancing volume and duration of information consumption by physicians: the case of health information exchange in critical care. J Biomed Inform 2017; 71: 1-15
  • 15 Overhage JM, McCallie Jr D. Physician time spent using the electronic health record during outpatient encounters: a descriptive study. Ann Intern Med 2020; 172 (03) 169-174
  • 16 Hill Jr RG, Sears LM, Melanson SW. 4000 clicks: a productivity analysis of electronic medical records in a community hospital ED. Am J Emerg Med 2013; 31 (11) 1591-1594
  • 17 Sinsky C, Colligan L, Li L. et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann Intern Med 2016; 165 (11) 753-761
  • 18 Ruppel H, Bhardwaj A, Manickam RN. et al. Assessment of electronic health record search patterns and practices by practitioners in a large integrated health care system. JAMA Netw Open 2020; 3 (03) e200512
  • 19 Holmgren JA, Downing NL, Bates DW. et al. Assessment of electronic health record use between US and non-US health systems. JAMA Intern Med 2020; 18 (02) 251-259
  • 20 Mazur LM, Mosaly PR, Moore C, Marks L. Association of the usability of electronic health records with cognitive workload and performance levels among physicians. JAMA Netw Open 2019; 2 (04) e191709
  • 21 Christensen T, Grimsmo A. Instant availability of patient records, but diminished availability of patient information: a multi-method study of GP's use of electronic patient records. BMC Med Inform Decis Mak 2008; 8 (12) 12
  • 22 Vehko T, Hyppönen H, Puttonen S. et al. Experienced time pressure and stress: electronic health records usability and information technology competence play a role. BMC Med Inform Decis Mak 2019; 19 (01) 160
  • 23 Smelcer JB, Miller-Jacobs H, Kantrovich L. Usability of electronic medical records. J Usability Stud 2009; 4 (02) 70-84
  • 24 Zalis M, Harris M. Advanced search of the electronic medical record: augmenting safety and efficiency in radiology. J Am Coll Radiol 2010; 7 (08) 625-633
  • 25 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
  • 26 Ben Schafer J, Frankowski D, Herlocker J, Sen S. Collaborative filtering recommender systems. In: Brusilovsky P, Kobsa A, Nejdl W. eds. The Adaptive Web. Berlin, Heidelberg: Springer; 2007: 291-324
  • 27 Smith B, Linden G. Two decades of recommender systems at Amazon.com. IEEE Internet Computing 2017; 21: 12-18
  • 28 Davidson J, Liebald B, Liu J, Nandy P, Van Vleet T. The YouTube video recommendation system. . Proceedings of the 4th ACM Conference on Recommender Systems. Barcelona, Spain, September 26–30, 2010: 293-296
  • 29 Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005; 8 (01) 19-32
  • 30 Liberati A, Altman DG, Tetzlaff J. et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009; 339: b2700
  • 31 Natarajan K, Stein D, Jain S, Elhadad N. An analysis of clinical queries in an electronic health record search utility. Int J Med Inform 2010; 79 (07) 515-522
  • 32 Bamnote GR, Agrawal SS. A review of clinical queries for electronic health record. Int J Adv Res Comput Sci 2012; 3 (03) 722-725
  • 33 Garcelon N, Neuraz A, Salomon R. et al. A clinician friendly data warehouse oriented toward narrative reports: Dr. Warehouse. J Biomed Inform 2018; 80: 52-63
  • 34 Biron P, Metzger MH, Pezet C, Sebban C, Barthuet E, Durand T. An information retrieval system for computerized patient records in the context of a daily hospital practice: the example of the Léon Bérard Cancer Center (France). Appl Clin Inform 2014; 5 (01) 191-205
  • 35 Duke JD, Morea J, Mamlin B. et al. Regenstrief Institute's Medical Gopher: a next-generation homegrown electronic medical record system. Int J Med Inform 2014; 83 (03) 170-179
  • 36 Duftschmid G, Rinner C, Kohler M, Huebner-Bloder G, Saboor S, Ammenwerth E. The EHR-ARCHE project: satisfying clinical information needs in a Shared Electronic Health Record system based on IHE XDS and Archetypes. Int J Med Inform 2013; 82 (12) 1195-1207
  • 37 Tawfik AA, Kochendorfer KM, Saparova D, Al Ghenaimi S, Moore JL. Using Semantic Search to Reduce Cognitive load in an Electronic Health Record. IEEE 13th International Conference on E-Health Networking, Applications and Services. Columbia, MO, June 13–15, 2011: 181-184
  • 38 Hasan SA, Zhu X, Liu J, Barra CM, Oliveira L, Farri O. Ontology-driven semantic search for Brazilian Portuguese clinical notes. Studi Health Technol Inform 2015; 216: 1022
  • 39 Hanauer DA. EMERSE: The Electronic Medical Record Search Engine. AMIA Annu Symp Proc 2006; 2006: 941
  • 40 Kovacs MD, Mesterhazy J, Avrin D, Urbania T, Mongan J. Correlate: a PACS- and EHR-integrated tool leveraging natural language processing to provide automated clinical follow-up. Radiographics 2017; 37 (05) 1451-1460
  • 41 Moen H, Peltonen L, Suhonen H, Koivumäki M, Salakoski T, Salanterä S. Unsupervised phrase-level query rewriting for assisting search in clinical free text. Stud Health Technol Inform 2019; 264 (03) 1550-1551
  • 42 Hanauer DA, Mei Q, Law J, Khanna R, Zheng K. Supporting information retrieval from electronic health records: a report of University of Michigan's nine-year experience in developing and using the Electronic Medical Record Search Engine (EMERSE). J Biomed Inform 2015; 55: 290-300
  • 43 Yang L, Mei Q, Zheng K, Hanauer DA. Query log analysis of an electronic health record search engine. AMIA Annu Symp Proc 2011; 2011: 915-924
  • 44 Zheng K, Mei Q, Hanauer DA. Collaborative search in electronic health records. J Am Med Inform Assoc 2011; 18 (03) 282-291
  • 45 Zheng K, Chen Y, Adler-Milstein J. et al. How do healthcare professionals personalize their software? A pilot exploration based on an electronic health records search engine. Stud Health Technol Inform 2019; 264: 1408-1412
  • 46 Ye C, Fabbri D. Extracting similar terms from multiple EMR-based semantic embeddings to support chart reviews. J Biomed Inform 2018; 83: 63-72
  • 47 Hultman GM, Marquard JL, Lindemann E, Arsoniadis E, Pakhomov S, Melton GB. Challenges and opportunities to improve the clinician experience reviewing electronic progress notes. Appl Clin Inform 2019; 10 (03) 446-453
  • 48 National Quality Forum. Submitting eCQMs for NQF Endorsement. Accessed March 12, 2021 at: https://www.qualityforum.org/Electronic_Quality_Measures.aspx
  • 49 Huang Z, Lu X, Duan H, Zhao C. Collaboration-based medical knowledge recommendation. Artif Intell Med 2012; 55 (01) 13-24
  • 50 Menon AK, Jiang X, Kim J, Vaidya J, Ohno-Machado L. Detecting inappropriate access to electronic health records using collaborative filtering. Mach Learn 2014; 95 (01) 87-101
  • 51 Shen F, Liu S, Wang Y, Wen A, Wang L, Liu H. Utilization of electronic medical records and biomedical literature to support the diagnosis of rare diseases using data fusion and collaborative filtering approaches. JMIR Med Inform 2018; 6 (04) e11301
  • 52 Klann JG, Szolovits P, Downs SM, Schadow G. Decision support from local data: creating adaptive order menus from past clinician behavior. J Biomed Inform 2014; 48: 84-93
  • 53 López-Nores M, Blanco-Fernández Y, Pazos-Arias JJ, Gil-Solla A. Property-based collaborative filtering for health-aware recommender systems. Expert Syst Appl 2012; 39 (08) 7451-7457
  • 54 Shen F, Liu S, Wang Y, Wang L, Afzal N, Liu H. Leveraging collaborative filtering to accelerate rare disease diagnosis. AMIA Annu Symp Proc 2018; 2017: 1554-1563
  • 55 Ren Z, Peng B, Schleyer TK, Ning X. Collaborative filtering methods for recommending search terms to clinicians. J Biomed Inform 2020
  • 56 Fan Z, Burgun E, Schleyer T, Ning X. Improving Information Retrieval from Electronic Health Records Using Dynamic and Multi-collaborative Filtering. 2019 IEEE International Conference on Healthcare Informatics (ICHI). Xi'an, China, June 10–13, 2019
  • 57 Chiang J, Kumar A, Morales D. et al. Physician usage and acceptance of a machine learning recommender system for simulated clinical order entry. AMIA Jt Summits Trans Sci Proc 2020; 2020: 89-97
  • 58 Kumar A, Aikens RC, Hom J. et al. OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases. J Am Med Inform Assoc 2020; 27 (12) 1850-1859

Address for correspondence

Titus Schleyer, DMD, PhD
Regenstrief Institute
1101 W 10th Street, Indianapolis, IN 46202
United States   

Publication History

Received: 22 January 2021

Accepted: 30 March 2021

Article published online:
14 July 2021

© 2021. The Author(s). This is an open access article published by Thieme under the terms of the Creative Commons Attribution-NonDerivative-NonCommercial License, permitting copying and reproduction so long as the original work is given appropriate credit. Contents may not be used for commercial purposes, or adapted, remixed, transformed or built upon. (https://creativecommons.org/licenses/by-nc-nd/4.0/)

Georg Thieme Verlag KG
Rüdigerstraße 14, 70469 Stuttgart, Germany

  • References

  • 1 Blumenthal D. Stimulating the adoption of health information technology. N Engl J Med 2009; 360 (15) 1477-1479
  • 2 The Office of the National Coordinator for Health Information Technology. Non-Federal Acute Care Hospital Electronic Health Record Adoption. Healthit.gov. 2017 . Accessed December 28, 2020 at: https://dashboard.healthit.gov/quickstats/pages/FIG-Hospital-EHR-Adoption.php
  • 3 The Office of the National Coordinator for Health Information Technology. Office-Based Physician Electronic Health Record Adoption. Healthit.gov. 2017 . Accessed December 28, 2020 at: https://dashboard.healthit.gov/quickstats/pages/physician-ehr-adoption-trends.php
  • 4 Pylypchuk Y, Johnson C, Patel V. State of Interoperability among U.S. Non-Federal Acute Care Hospitals in 2018. 2020 . Available at: https://www.healthit.gov/sites/default/files/page/2020-03/State-of-Interoperability-among-US-Non-federal-Acute-Care-Hospitals-in-2018.pdf
  • 5 The Office of the National Coordinator for Health Information Technology. Connecting Health and Care for the Nation: A Shared Nationwide Interoperability Roadmap. Available at: https://www.healthit.gov/sites/default/files/hie-interoperability/nationwide-interoperability-roadmap-final-version-1.0.pdf
  • 6 Ross MK, Wei W, Ohno-Machado L. “Big data” and the electronic health record. Yearb Med Inform 2014; 9 (01) 97-104
  • 7 Centers for Disease Control and Prevention. Electronic Medical Records/Electronic Health Records (EMRs/EHRs). National Center for Health Statistics. 2010 . Accessed March 12, 2021 at: https://www.cdc.gov/nchs/fastats/electronic-medical-records.htm
  • 8 Halamka J. The Cost of Storing Patient Records. GeekDoctor. 2011 . Accessed December 14, 2020 at: http://geekdoctor.blogspot.com/2011/04/cost-of-storing-patient-records.html
  • 9 Gal DB, Han B, Longhurst C, Scheinker D, Shin AY. Quantifying electronic health record data: a potential risk for cognitive overload. Hosp Pediatr 2021; 11 (02) 175-178
  • 10 Donovan F. Organizations See 878% Health Data Growth Rate Since 2016. HIT Infrastructure. 2019 . Accessed December 14, 2020 at: https://hitinfrastructure.com/news/organizations-see-878-health-data-growth-rate-since-2016
  • 11 Munyisia E, Yu P, Hailey D. The effect of an electronic health record system on nursing staff time in a nursing home: a longitudinal cohort study. Australas Med J 2014; 7 (07) 285-293
  • 12 Rojo MG, Castro AM, Gonçalves L. COST action “EuroTelepath”: digital pathology integration in electronic health record, including primary care centres. Diagn Pathol 2011; 6 (01, Suppl 1): S6
  • 13 Wilkerson ML, Henricks WH, Castellani WJ, Whitsitt MS, Sinard JH. Management of laboratory data and information exchange in the electronic health record. Arch Pathol Lab Med 2015; 139 (03) 319-327
  • 14 Politi L, Codish S, Sagy I, Fink L. Balancing volume and duration of information consumption by physicians: the case of health information exchange in critical care. J Biomed Inform 2017; 71: 1-15
  • 15 Overhage JM, McCallie Jr D. Physician time spent using the electronic health record during outpatient encounters: a descriptive study. Ann Intern Med 2020; 172 (03) 169-174
  • 16 Hill Jr RG, Sears LM, Melanson SW. 4000 clicks: a productivity analysis of electronic medical records in a community hospital ED. Am J Emerg Med 2013; 31 (11) 1591-1594
  • 17 Sinsky C, Colligan L, Li L. et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann Intern Med 2016; 165 (11) 753-761
  • 18 Ruppel H, Bhardwaj A, Manickam RN. et al. Assessment of electronic health record search patterns and practices by practitioners in a large integrated health care system. JAMA Netw Open 2020; 3 (03) e200512
  • 19 Holmgren JA, Downing NL, Bates DW. et al. Assessment of electronic health record use between US and non-US health systems. JAMA Intern Med 2020; 18 (02) 251-259
  • 20 Mazur LM, Mosaly PR, Moore C, Marks L. Association of the usability of electronic health records with cognitive workload and performance levels among physicians. JAMA Netw Open 2019; 2 (04) e191709
  • 21 Christensen T, Grimsmo A. Instant availability of patient records, but diminished availability of patient information: a multi-method study of GP's use of electronic patient records. BMC Med Inform Decis Mak 2008; 8 (12) 12
  • 22 Vehko T, Hyppönen H, Puttonen S. et al. Experienced time pressure and stress: electronic health records usability and information technology competence play a role. BMC Med Inform Decis Mak 2019; 19 (01) 160
  • 23 Smelcer JB, Miller-Jacobs H, Kantrovich L. Usability of electronic medical records. J Usability Stud 2009; 4 (02) 70-84
  • 24 Zalis M, Harris M. Advanced search of the electronic medical record: augmenting safety and efficiency in radiology. J Am Coll Radiol 2010; 7 (08) 625-633
  • 25 Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA 2018; 319 (12) 1276-1278
  • 26 Ben Schafer J, Frankowski D, Herlocker J, Sen S. Collaborative filtering recommender systems. In: Brusilovsky P, Kobsa A, Nejdl W. eds. The Adaptive Web. Berlin, Heidelberg: Springer; 2007: 291-324
  • 27 Smith B, Linden G. Two decades of recommender systems at Amazon.com. IEEE Internet Computing 2017; 21: 12-18
  • 28 Davidson J, Liebald B, Liu J, Nandy P, Van Vleet T. The YouTube video recommendation system. . Proceedings of the 4th ACM Conference on Recommender Systems. Barcelona, Spain, September 26–30, 2010: 293-296
  • 29 Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005; 8 (01) 19-32
  • 30 Liberati A, Altman DG, Tetzlaff J. et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009; 339: b2700
  • 31 Natarajan K, Stein D, Jain S, Elhadad N. An analysis of clinical queries in an electronic health record search utility. Int J Med Inform 2010; 79 (07) 515-522
  • 32 Bamnote GR, Agrawal SS. A review of clinical queries for electronic health record. Int J Adv Res Comput Sci 2012; 3 (03) 722-725
  • 33 Garcelon N, Neuraz A, Salomon R. et al. A clinician friendly data warehouse oriented toward narrative reports: Dr. Warehouse. J Biomed Inform 2018; 80: 52-63
  • 34 Biron P, Metzger MH, Pezet C, Sebban C, Barthuet E, Durand T. An information retrieval system for computerized patient records in the context of a daily hospital practice: the example of the Léon Bérard Cancer Center (France). Appl Clin Inform 2014; 5 (01) 191-205
  • 35 Duke JD, Morea J, Mamlin B. et al. Regenstrief Institute's Medical Gopher: a next-generation homegrown electronic medical record system. Int J Med Inform 2014; 83 (03) 170-179
  • 36 Duftschmid G, Rinner C, Kohler M, Huebner-Bloder G, Saboor S, Ammenwerth E. The EHR-ARCHE project: satisfying clinical information needs in a Shared Electronic Health Record system based on IHE XDS and Archetypes. Int J Med Inform 2013; 82 (12) 1195-1207
  • 37 Tawfik AA, Kochendorfer KM, Saparova D, Al Ghenaimi S, Moore JL. Using Semantic Search to Reduce Cognitive load in an Electronic Health Record. IEEE 13th International Conference on E-Health Networking, Applications and Services. Columbia, MO, June 13–15, 2011: 181-184
  • 38 Hasan SA, Zhu X, Liu J, Barra CM, Oliveira L, Farri O. Ontology-driven semantic search for Brazilian Portuguese clinical notes. Studi Health Technol Inform 2015; 216: 1022
  • 39 Hanauer DA. EMERSE: The Electronic Medical Record Search Engine. AMIA Annu Symp Proc 2006; 2006: 941
  • 40 Kovacs MD, Mesterhazy J, Avrin D, Urbania T, Mongan J. Correlate: a PACS- and EHR-integrated tool leveraging natural language processing to provide automated clinical follow-up. Radiographics 2017; 37 (05) 1451-1460
  • 41 Moen H, Peltonen L, Suhonen H, Koivumäki M, Salakoski T, Salanterä S. Unsupervised phrase-level query rewriting for assisting search in clinical free text. Stud Health Technol Inform 2019; 264 (03) 1550-1551
  • 42 Hanauer DA, Mei Q, Law J, Khanna R, Zheng K. Supporting information retrieval from electronic health records: a report of University of Michigan's nine-year experience in developing and using the Electronic Medical Record Search Engine (EMERSE). J Biomed Inform 2015; 55: 290-300
  • 43 Yang L, Mei Q, Zheng K, Hanauer DA. Query log analysis of an electronic health record search engine. AMIA Annu Symp Proc 2011; 2011: 915-924
  • 44 Zheng K, Mei Q, Hanauer DA. Collaborative search in electronic health records. J Am Med Inform Assoc 2011; 18 (03) 282-291
  • 45 Zheng K, Chen Y, Adler-Milstein J. et al. How do healthcare professionals personalize their software? A pilot exploration based on an electronic health records search engine. Stud Health Technol Inform 2019; 264: 1408-1412
  • 46 Ye C, Fabbri D. Extracting similar terms from multiple EMR-based semantic embeddings to support chart reviews. J Biomed Inform 2018; 83: 63-72
  • 47 Hultman GM, Marquard JL, Lindemann E, Arsoniadis E, Pakhomov S, Melton GB. Challenges and opportunities to improve the clinician experience reviewing electronic progress notes. Appl Clin Inform 2019; 10 (03) 446-453
  • 48 National Quality Forum. Submitting eCQMs for NQF Endorsement. Accessed March 12, 2021 at: https://www.qualityforum.org/Electronic_Quality_Measures.aspx
  • 49 Huang Z, Lu X, Duan H, Zhao C. Collaboration-based medical knowledge recommendation. Artif Intell Med 2012; 55 (01) 13-24
  • 50 Menon AK, Jiang X, Kim J, Vaidya J, Ohno-Machado L. Detecting inappropriate access to electronic health records using collaborative filtering. Mach Learn 2014; 95 (01) 87-101
  • 51 Shen F, Liu S, Wang Y, Wen A, Wang L, Liu H. Utilization of electronic medical records and biomedical literature to support the diagnosis of rare diseases using data fusion and collaborative filtering approaches. JMIR Med Inform 2018; 6 (04) e11301
  • 52 Klann JG, Szolovits P, Downs SM, Schadow G. Decision support from local data: creating adaptive order menus from past clinician behavior. J Biomed Inform 2014; 48: 84-93
  • 53 López-Nores M, Blanco-Fernández Y, Pazos-Arias JJ, Gil-Solla A. Property-based collaborative filtering for health-aware recommender systems. Expert Syst Appl 2012; 39 (08) 7451-7457
  • 54 Shen F, Liu S, Wang Y, Wang L, Afzal N, Liu H. Leveraging collaborative filtering to accelerate rare disease diagnosis. AMIA Annu Symp Proc 2018; 2017: 1554-1563
  • 55 Ren Z, Peng B, Schleyer TK, Ning X. Collaborative filtering methods for recommending search terms to clinicians. J Biomed Inform 2020
  • 56 Fan Z, Burgun E, Schleyer T, Ning X. Improving Information Retrieval from Electronic Health Records Using Dynamic and Multi-collaborative Filtering. 2019 IEEE International Conference on Healthcare Informatics (ICHI). Xi'an, China, June 10–13, 2019
  • 57 Chiang J, Kumar A, Morales D. et al. Physician usage and acceptance of a machine learning recommender system for simulated clinical order entry. AMIA Jt Summits Trans Sci Proc 2020; 2020: 89-97
  • 58 Kumar A, Aikens RC, Hom J. et al. OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases. J Am Med Inform Assoc 2020; 27 (12) 1850-1859

Zoom Image
Fig. 1 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram of the article screening process.