Advances in Sharing Multi-sourced Health Data on Decision Support Science 2016-2017

Summary

Introduction: Clinical decision support science is expanding to include integration from broader and more varied data sources, diverse platforms and delivery modalities, and is responding to emerging regulatory guidelines and increased interest from industry.

Objective: Evaluate key advances and challenges of accessing, sharing, and managing data from multiple sources for development and implementation of Clinical Decision Support (CDS) systems in 2016-2017.

Methods: Assessment of literature and scientific conference proceedings, current and pending policy development, and review of commercial applications nationally and internationally.

Results: CDS research is approaching multiple landmark points driven by commercialization interests, emerging regulatory policy, and increased public awareness. However, the availability of patient-related “Big Data” sources from genomics and mobile health, expanded privacy considerations, applications of service-based computational techniques and tools, the emergence of “app” ecosystems, and evolving patient-centric approaches reflect the distributed, complex, and uneven maturity of the CDS landscape. Nonetheless, the field of CDS is yet to mature. The lack of standards and CDS-specific policies from regulatory bodies that address the privacy and safety concerns of data and knowledge sharing to support CDS development may continue to slow down the broad CDS adoption within and across institutions.

Conclusion: Partnerships with Electronic Health Record and commercial CDS vendors, policy makers, standards development agencies, clinicians, and patients are needed to see CDS deployed in the evolving learning health system.

Keywords

Informatics - decision making, computer assisted - decision support techniques

Introduction

Clinical decision support (CDS) is a well-established and evolving field that “provides clinicians, staff, patients, or other individuals with knowledge and person-specific information, intelligently filtered or presented at appropriate times, to enhance health and health care”[1] [2]. The increasingly varied settings where CDS is applied range from solving problems that arise during day-to-day clinical care, to addressing patient's health maintenance and preventative care, or to focus on enhancing patient/ provider education and training[3] [4]. CDS systems (CDSS) have been recognized in roughly three main groups: (i) Relevant information retrieval at the point-of-care based on the clinical context (e.g., Infobuttons), (ii) Error checks, alerts, order sets, and other recommendations that are patient- and situation-specific, and (iii) Reducing cognitive burden and support problem solving/decision making by structuring and organizing information such as structured reports, graphical representations, dashboards, structured templates for documentation, preference lists, and order sets. These groups can further be considered to be within four implementation phases: (i) Standalone CDSSs, (ii) CDSSs that are built into the clinical systems, (iii) Standardizing the CDS content for sharing, and (iv) Web service models.

This increasingly broad domain of CDS is influenced by the access to and adoption of more powerful and cheaper technology, current and anticipated forces of regulation, and evolving commercial market enthusiasm. The scope of this survey is to highlight significant CDS developments and innovations during 2016-2017 that build on recent IMIA and community reviews, with an emphasis on data access, data sharing, and data ownership issues that cross and extend the groups and phases described above.

Material and Methods

TWe assessed literature and guidelines from PubMed, Google Scholar, and Web of Sciences from 2016 to 2017, including presentations at the 2017 American Medical Informatics Association (AMIA) annual symposium, the 2017 Society for Medical Decision Making meeting (SMDM 2017), public/private groups (CDS coalition), and federal guidelines. Literature search keywords included “privacy”, “clinical decision support”, “decision support”, “data sharing”, “policy”, “interoperability”, and “standards”. We manually excluded publications and sources that did not explicitly involve decision support systems or methods in human health, such as patient privacy and autonomy, or data access and data sharing technology. As this area is dynamic and rapidly evolving, we have included seminal papers and initiatives that fell slightly outside the search daterange for context. We used extensive personal communications and recommendations regarding international work as well as in industry settings.

These literature and resources were evaluated for innovation and evidence of implementation beyond the 2016 IMIA Yearbook's 10 thematic areas of CDS[5]. Throughout, we looked for evidence of change or progress towards the next generation of anticipated interoperability models: (i) Sharing knowledge artifacts: developing computable standard knowledge artifacts that could be integrated and interpreted by others, (ii) Knowledge service providers with standards-based patient evaluation services: computable standard-based knowledge (e.g., clinical care guidelines), written once and incorporated anywhere, and (iii) Sharing standard applications: applications are developed and shared based on standard EHR Application Programming Interfaces (API) [6] . In the following review, we outline new developments that either extend or reinforce these interoperability models, reflect changes to practice, or provide new evidence that address data sharing, data access, and privacy issues for CDS.

Results

Based on our reviews, there were several inter-related advancing themes: regulatory developments for CDS commercialization, decision support data sharing architectures, access to data and standards development, decision support application sharing, large data environments, patient-directed decision support, population health, and ethical and privacy issues in CDS ([Table 1]).

Table 1 Number of final consensus statements for each of the Institute of Medicine domains of health care quality

Themes	Summary Comments
Regulatory developments for CDS commercialization	FDA draft guidance released to clarify CDS software that will and that will not remain under regulatory monitoring
Decision support data sharing architectures	Wider implementation and testing of Service Oriented Architectures (SOA) for data sharing and services in broader multi-system environments
Access to data and standards development	Progress with many CDS-related standards: Clinical Quality Framework (CQF), FHIR compatible QUICK – the QUality Improvement and Clinical Knowledge model, Clinical Quality Language (CQL) expression language and Event-Condition-Action (ECA) rules
Decision support application sharing	Advances in HL7 FHIR standards development for generalizable data access and SMART (Substitutable Medical Applications & Reusable Technologies) platform to embed web applications into electronic health record (EHR) workflows
Large data environments	Movement of commercial vendors into big data analytics and steady progress with exchanging genetic and genomic data across laboratories, EHRs, and others
Patient-directed decision support	Regulatory development in mHealth CDS, and expanded focus on evaluating multi-stakeholder involvement in shared decision making through patient-portals
Population health	Initiatives by public health institutions such as VA and CDC addressing technical challenges in linking disparate public and private data sources and CDSS
Data sharing and ethics in CDS	Efforts by agencies such as Global Alliance for Genomics and Health, American College of Medical Genetics, research efforts by National Institutes of Health (NIH)'s ‘All of US’ program and industry (Verily's Project Baseline), and patient advocacy groups addressing ethical concerns, including personal benefit expectations of patients separate from the goals of the research communities

Regulatory Developments for CDS Commercialization

The Food and Drug Administration (FDA) Center for Devices and Radiological Health (CDRH) is responsible for the development and public release of guidance on CDS software. CDS guidelines have been under CDRH consideration since 2011 and CDS vendors as well as industry were anticipating new guidelines during 2017. However, CDRH has provided draft guidance for public comments in December 8, 2017 [7]. The released policy draft guidance to Health Care Professionals for Clinical and Patient Decision support defines and categorizes CDS software functions that are not devices from CDS functions that remain devices. It further clarifies both of these definitions as being distinct from Patient Decision Support software. The implications are that the draft guidance provides formalized definitions of locations where CDSSs are used in healthcare settings, and for the development and compliance components of future commercial CDSSs. According to the new guidance, products that have a sole intent to transfer, store, convert formats, and display medical device data and results, including medical images, waveforms, signals, or other clinical information, are not considered as devices and are not subject to FDA regulatory requirements. But, if the products also analyze or interpret the acquired data, help with diagnosis, cure, mitigation, prevention, or treatment of a disease or condition, or if the products generate, based on the acquired data, alarms or alerts to inform healthcare providers to take an immediate clinical action, they will be under the FDA regulatory oversight. In parallel to this guidance, the FDA Software Precertification program was launched in September 26, 2017 with nine selected industry vendors including those with known CDS interests such as Apple, Roche, Johnson and Johnson, and Verily, with a stated intent to foster innovation while protecting public health through an accelerated approval process[8].

To prepare for anticipated FDA enhanced transparency and evaluation requirements, the industry and health-system-oriented CDS Coalition has offered voluntary industry guideline recommendations for CDS design[9]. The coalition stipulates that CDS software developers address two fundamental questions: whether healthcare users of the software can independently review the basis of the software recommendations, and whether healthcare users, in the intended circumstances, actually need to rely on the software. Since the 2012 FDA Safety and Innovation Act (FDASIA), the FDA along with the NIH Office of the National Coordinator (ONC) and the Federal Communications Commission (FCC) have been required to develop and publish in their websites “a report that contains a proposed strategy and recommendations on an appropriate, risk-based, regulatory framework pertaining to health information technology, including mobile medical applications, that promotes innovation, protects patient safety, and avoids regulatory duplication“. These requirements and the potential for central coordination of future guidelines development have seen recommendations to form a Health Information Technology Safety Center to collect error reports, review CDS and EHR knowledge resources, and provide models for reviewing EHR and CDS safety-related issues at participating centers and institutions[10] [11].

Starting in 2016, the Joint Commission mandated organizations to prioritize implementation of specific predictive analytics policies, procedures, and staff education about important alarms in healthcare settings[12]. Environments including Intensive Care Units (ICUs) were further required to address issues of CDS design and implementation in order to minimize cognitive overload and reduce alert fatigue in practitioner settings. Critical care settings are data-, information-, and knowledge-intensive and real-time decision support based on high-volume data analytics is increasingly a key requirement. Work that involves overcoming very high volume of streaming data coupled with rapid decision support recommendations has been reported both nationally and internationally in critical care and pediatric settings[13] [14] [15] [16].

Decision Support Data Sharing Architectures

Approaches that use Service Oriented Ar-chitectures (SOA) to share common encoded structured rules and services have been tested in broader multi-system environments. CDS using SOA access requires patient data to be abstracted from EHRs and shared with the CDS services that provide knowledge resources from outside of EHR systems[17]. CDS efforts involving SOA are testing the standardization of interoperability between EHR systems, based in part on established meaningful use requirements such as Conti-nuity of Care Documents (CDA)/Consolidat-ed CDA and in some instances Virtual Medi-cal Record (vMR) clinical entities[18]. SOA CDSSs seek to overcome the requirement that all EHRs be configured to implement complicated CDS knowledge encoding. Data access and data sharing via SOA are required to be secure, vendor diverse, reusable, and implementable as web-accessible services. Two international standards agencies, HL7 and Object Management Group (OMG) jointly support SOA by developing service specification standards. Several countries have adopted SOA including USA and Canada as part of their Health Information Technology (HIT) strategies [19]. However, SOA approaches continue to require tuning for specific systems, and are not yet scalable to provide generalized CDS services to mul-tiple applications - one overarching goal of standards development agencies.

Other technical factors of the SOA architecture are concerns, such as the delay in response time due to the excessive time required for data retrieval, or the balance between availability and security requirements for ensuring protection of potentially sensitive patient data. Requirements for hosting SOAs on commercially-owned cloud computing environments that allow for scalability and rapid deployment are being proposed and evaluated but may not yet be cost effective until they demonstrate substantial shared usage[19] [20]. Work reporting a privacy-preserving framework for implementing real-time monitoring proposed new data protocols that support patient health status monitoring without privacy leakage in simulation settings [21]. One example model of a SOA implementation describes a HealthCare Decision Support System (HCDSS) in Taiwan supporting both design and registration of CDS over a SOA architecture among hospital groups [22]. Another study evaluated web-based CDS in a multi-emergency department implementation using a shared Enterprise Clinical Rules Service (ECRS), where recommendations were proposed for inclusion of children with minor blunt head trauma in clinical trials through real-time and near-real time recommendations [23] [24].

Access to Data and Standards Development

Providing access and enabling the sharing of patient's health data through HIT is foundational to CDS. Organizations weigh cost, access, and security of patient information against CDSS implementation efforts that require data sharing and access from external sources[25]. However, in many cases, only a subset of patient's data is all that is required for CDS (and clinical quality measures), based on the objective of the CDS.

A multi-institutional dedicated collaboration called OpenCDS continues to work on open-source CDS standards-based tools that focus on developing and managing knowledge executable platforms that work with different clinical systems[26]. As CDS and Electronic Clinical Quality Measures (eCQM) can be considered two approaches to a similar challenge, the Clinical Quality Framework (CQF) was developed as part of the Clinical Quality Improvement (CQI) efforts to promote standards of access to EHR clinical content under a common goal to reduce duplication of efforts. Components of the framework include a common metadata standard, the QUICK (QUality Improvement and Clinical Knowledge) model to replace vMR, and a new expression language, the Clinical Quality Language (CQL) that attempts to incorporate logic encoding features of GELLO (Guidelines Expression Language - Object Oriented), an Object-Oriented Query and Expression Language for CDS, Event-Condition-Action (ECA) rules, and Arden Syntax. These new standards are designed to be able to use the Fast Healthcare Interoperability Resources (FHIR), the emerging HL7 data access API standard, to provide a more direct access to EHR clinical data[27]. Many demonstration pilots have been successfully tested including the Center for Disease Control (CDC) Opioid CDS Demonstration Project that encodes CDC's guidelines for Opioid prescription for chronic pain management[28] [29]. Archetypes, the clinical information models developed by the international virtual standards development agency, openEHR, were found to contain more granular representational detail and structure compared to HL7's vMR, thus providing better access and integration to patient's data, and some potential for applications beyond CDS[30] [31].

Another role of enhanced standards-driven patient data access is that it may be an important solution to help reduce alert fatigue by designing and configuring alerts so that only accurate and relevant alerts are triggered. Other solutions and frameworks that have been recommended as to personalize and reduce alert fatigue include configuring alerts based on context and provider workflows, both of which require providing CDSSs increased and structured access to patient's data[32] [33] [34].

Decision Support Sharable Applications

Shareable CDS healthcare applications are being proposed to develop access and link to patient data from EHRs, provide CDS, and add additional representational features that cater to the diverse needs of healthcare sub- and super-specialty professionals and patients. The HL7 FHIR standard is being leveraged in support of generalizable data access approaches in a range of areas[35]. FHIR specifies a library of standard clinical “resources”, such as allergy, adverse reaction, family history, and procedures which can be readily mapped to existing EHR data models. FHIR resources roughly correspond to the data content of vMR templates but avoid the complexity of mapping through the HL7 V3 RIM intermediary. FHIR utilizes the ubiquitous hypertext transfer protocol HTTPS and creates a standard API to share the information encoded within resources among applications such as EHRs. This reflects a significant shift with regard to providing a remote CDS service through encapsulating full access to relevant patient's clinical data required for the CDS logic such as laboratory results, medications, and diagnosis data within vendor agnostic interfaces without requiring custom coding. This controlled and standardized access to clinical data for CDS provides new opportunities of creating CDS abstraction interfaces that were not possible with inflexible and custom access to localized and limited sets of patient data.

A similar significant shift is the expansion of implementation of CDS on the SMART (Substitutable Medical Applications & Reusable Technologies) platform, a standard specification to embed web applications into EHR workflows. The FHIR-based apps discussed above when combined with the SMART platform provide embeddable apps with graphical interfaces, paving the way for “SMART on FHIR” app stores. An increasing number of EHR vendors have announced capabilities of supporting SMART apps. Though still relatively modest, the SMART on FHIR application market appears to be a developing ecosystem with a range of EHR vendors and academic stakeholders. This encapsulation of standards-driven data access as well as presentation frameworks that can be linked to providers workflows at the point of care show progress towards using sharable and custom external CDS to address complex clinical scenarios [6] [35] [36] [37].

Large Data Environments and Decision Support

Many products of the emerging big data analytics innovations in healthcare meet broad criteria to the extent that they can be considered as CDSSs. Big data analytic capabilities based on previously described healthcare implementation examples have been categorized into four major capabilities: patterns of care analytics, unstructured data analytics, predictive analytics, and traceability. The objective of the “traceability” capability is to track the output data from all service units of organizations IT system components to check the data for consistency, visibility, and easy accessibility for analysis[38].

In the commercial world of big data ana-lytics, IBM Watson Health's Cognitive Com-puting platform has gained significant media attention though it has limited independent academic evaluation of utility. Adoption of IBM's proprietary DeepQA system to healthcare has been advertised as able to consume structured and unstructured pa-tient's clinical data, medical knowledge de-rived from medical textbooks, dictionaries, clinical guidelines, research literature, and publicly available information on the web, and to hypothesize an ordered list of possible evidence-based answers to a given patient's clinical condition [39] [40]. Using patient's longitudinal clinical data from EHRs, it is purported to be able to generate and summarize evidence about differential diagnoses, problem lists, most appropriate treatments, and to search for specific clinician's free text questions about the patient. The CDS applications of Watson have been predominantly within oncology with variable results of the system in choosing the appropriate treatment protocols for cancer patients as compared to tumor board cancer specialists[41] [42]. The reduced accuracy of this otherwise relatively opaque system has been attributed to local variations of cancer management given the available resources[43].

Expectations of integrating genomics and clinical data to provide Genomics CDS (GCDS) face four challenges related to interoperability: CDS data sharing architectures, patient data access and standards, application sharing, and CDS with large data environments [44] [45] [46]. Despite the broad and evolving availability of commercial and institutional genetic testing services, most current clinical genetic test results are predominantly self-contained reports and are not computationally interpretable. Actionable result data from genetic testing laboratories are often free text, lack consistent structure, indexing, or mapping to standard clinical vocabularies rendering them of limited traceable or interoperable value. This adds to the complexity of data sharing and the integration of genetic results with other EHR data structures, including their incorporation into CDS modules. Recent works report on the requirements for design, delivery, and presentation methods for delivering summarized multigene sequencing panel results to clinicians[47], the evaluation of the information structure and delivery context of variant reports to clinical stakeholders at the point of care [48], and approaches to providing “Infobuttons” for retrieving specific genetic information [49].

The active pharmacogenomics (PGx) application area of GCDS is identifying barriers and opportunities in workflow integration, standards alignment, and models to evaluate tailored genomic-based drug therapy [50] [51] [52]. A pediatrics pilot implementation of an automated physician-oriented view of pharmacogenomics variants, potentially actionable at the point of care for medication prescribing, reported positive results for scalability of the system, as well as the need to focus on clinician's alert response behaviors [50]. A review of user-design components of pharmacogenomics systems to date identified that evaluations of user-interface at the point of care were limited, with the majority work being dominated by prototype designs [51].

The Pharmacogenomic Resource for Enhanced Decisions in Care and Treatment (PREDICT) personalized medicine program implements and evaluates public health approaches to identify cohorts of patients with specific genotypes for precise medication prescriptions targeted to improve treatment effectiveness, patient outcomes, and reduce healthcare costs [52]. Recent PREDICT-associated studies reported on multidisciplinary implementation model approaches at the point of care [53], outcomes of clinician drug prescription patterns at the point of care based on preemptive pharmacogenomic risk [54], and attitudes of clinicians regarding notification timeliness and decision-making responsibility following implementation into standard practice environments[55].

The electronic Medical Records and Genomics (eMERGE) network, established in 2007 and now in phase three, focuses on developing and validating electronic phenotyping algorithms for large-scale, high-throughput genomics research. eMERGE enrolls over 25,000 participants across 10 national centers to identify best practices for identifying rare variants and assessing phenotype variant implications and impacts on patient and care processes. In 2015, the eMERGE consortia published a review of the practical considerations for adoption, based on the successful implementation of at least one PGx CDS rule in the clinical setting across 10 eMERGE sites. The majority of eMERGE sites opted for the creation of an Omic Ancillary System (OAS) to separately manage genetic and genomic data [56]. Recent eMERGE publications focused on CDSS specific implementation outcomes performed within the network, including GCDS for drug-based precision medicine[57] [58].

Patient-Directed Decision Support

The rapidly expanding mobile health ecosystem remains primarily driven by consumer demand for personal health tools, with the definition of where these technologies could be considered as CDSSs linked to clinically actionable health outcomes still unclear. Recent published works within clinically oriented personalized health that involve mobile or e-health technologies were predominantly research-oriented framework proposals. With the FDA Software precertification program enrolling major device and technology vendors with interests in consumer-oriented software and devices and embedded CDS, the landscape and expectations of personal CDS software tools and devices may change soon [8]. Other recent works showed a focus on design and evaluation methods, with proposals of models for evaluating the provisioning of evidence-based practices with patient involvement [59] [60], evaluation of patient education in chronic care management [61] [62], and requirements for capturing specific patient-provided mobile data in forms that link to physician-oriented decision support environments [18].

Patient portals that are integrated into EHR systems may be key environments for developing shared decision-making, through linking patient-provided data in context (including data acquired through mHealth as well as direct to consumer laboratory tests) with clinically acquired EHR data. Works focused on evaluating where patient portals contributed to communications and shared decision-making in intensive care settings [63], and consequences on clinical organizational behavior through augmented communications through patient portals [64]. Patient portals that provided results based on clinically acquired data reported on approaches for evaluating risks to patients through the presentation of summarized clinical lab results [65], and evaluation of impact of sharing genomic results through a clinically-oriented patient portal (vs. study specific genomic interfaces) [66].

In a separate reflection of the challenges of evaluating patient-directed decision support dependencies on location and stakeholders, the Center for Medicare and Medicaid Services (CMMS) cancelled the proposed testing of the Shared Decision Making (SDM) model evaluation due to the lack of participation of Accountable Care Organizations. CMMS will continue to evaluate the Direct Decision Support (DDS) model that explores how to enhance practitioner-patient relationships and support preference-sensitive patient decision-making through a focus on six conditions, including heart disease, osteoarthritis, and localized prostate cancer [67] [68].

Population Health

Expansion of CDS beyond clinical settings to include population health environments was considered from the perspectives of chronic care management and diagnostic support. A coordinated Chronic Care Model (CCM) implementation that linked multiple EHRs replaced manual patient characterization, and disconnected programs reported favorable health management outcomes over a four-year evaluation [69]. A study of interoperability requirements across clinical domains within the Veterans Administration (VA) characterized technical challenges in linking national VA infrastructures to local CDSSs, as well as data idiosyncrasies such as standardized medications [70]. The Public Health Activities and Services Tracking (PHAST) group, representing a multistate consortia of researchers and practice partners engaged in public health service delivery, proposed approaches that specified linking local and public health data sources [71]. A review of the requirements to link exposome-acquired data from public health sources to health outcomes that can support discovery suggests that it is early days for effective implementation of such data sources into CDSSs [72].

At the national level, in order to ensure that patients receive “the right immunization at the right time“, the CDC has developed Clinical Decision Support for Immunization (CDSi) logic specification that incorporates vaccine decision guidelines in an unambiguous way to enable automatic evaluation and forecasting of recommended immunizations for a given patient. Any Immunization Information System (IIS) and clinical systems could develop and implement the CDS engine and run the CDSi logic [73]. A recently announced collaborative, expansive in its stakeholder invitations (involving CDC programs, EHR vendors, developers, and public health partners, among others), is the CDC's Office of Public Health Scientific Services (OPHSS) initiative for “Adapting Clinical Guidelines for the Digital Age” [74].

Data Sharing and Ethics in CDS

In a review of the current ethical and legal issues surrounding computerized decision support, Goodman addresses the complex relationship that exists between protecting patient safety and opportunities at improving care [75]. As described, accountability, responsibility, and liability are at stake, as well as the role of government regulations, and more importantly the lack of empirical evidence to help address ethical concerns. Potential applications of CDS in medical genomics face perhaps the broadest array of data sharing, privacy, and ethical challenges, and they reflect the expansive data ownership and expected utility issues of multi-source data linkage that crosses individual, institutional, national, and international boundaries. The discussion of the potential benefits of sharing and integrating genetic and genomic data with clinical data as part of CDS systems for personalized medicine is a prominent focus with cancer and rare diseases communities, but unevenly implemented within individual national or state borders. International organizations such as the Global Alliance for Genomics and Health (GA4GH) or the US national American College of Medical Genetics (ACMG) have proposed recommendations for evaluation and adoption of fine-grained guidelines, policies, and procedures that encode patient preferences and that can maintain abilities to evaluate disease community health benefits through expanding trust relationships across borders[76] [77] [78]. Ongoing efforts to identify public needs and awareness about large scale genomic, epidemiological, clinical, and research data reflect the base challenges of engaging patient and stakeholder communities over the long-term as CDS systems and the economic market for CDS evolve [79] [80] [81]. The growing public awareness of the US national precision medicine program “All of Us” [82], the VA Million Veterans Program [83], as well as industry research efforts such as Verily's Project Baseline[84] that are collecting large amounts of data for predictive and diagnostic decision support may spur a more expansive discussion of personal benefit expectations separate of the research goals that are being promoted.

Discussion

The breadth and overlaps of the activity in CDS did not allow us to exhaustively identify or describe every important or significant study or initiative. Throughout our review, we sought to identify the expansion or change across the thematic foci described in the previous IMIA review[5], and to highlight policies and innovative efforts that cross boundaries of data sharing, technology, and application domains.

The opportunities for CDSSs to help deliver high quality, safe, and cost-effective healthcare remain immense, yet their implementation as reflected in the literature is still limited, uneven, and lacks consistency and uniformity across institutions or industry partners. The potential domains of influence for successful CDSS development and adoption include technical, organizational, financial, political, policy/regulatory, and sociocultural dimensions, but they are not yet clearly represented as strategic investment priorities within or across health systems. It is important to acknowledge that the implementation of commercial EHR systems now dominates clinical health systems nationally and internationally - within the US, Canada, Australia and Western Europe. EHR system implementation has been a relatively stable market in 2017, with over 50% of the market represented by systems from Cerner or Epic[85] [86]. These dominant market forces make the potential impact of CDS systems and approaches we describe difficult to separate from the dependencies on the ecosystems of those vendors and their many individual and unique implementations.

Current limitations for exchangeable CDSS development and adoption are attributable to the lack of formal specifications for CDSS components, the lack of alignment of policy and regulatory guidance, and the slow universal adoption of vocabulary and interoperable standards or uniform workflows. These factors contribute to quality concerns of the collected multi-source data, not just in support of the implementation and evaluation of CDSSs, but also to the overall usefulness of the data for other informatics-related applications such as registry development, data submission to insurers, regulatory and professional institutions, or quality and outcomes research efforts. To date, major vendors continue to develop their own proprietary vocabularies to represent clinical content while gradually incorporating standards-based vocabularies provided by third parties such as ICD codes, medication lists, and laboratory orders. The most relevant issue for CDS beyond variety in common terminologies is the widely heterogeneous implementation of clinician workflows across institutions, even within the same vendor of EHR systems, which impacts the capabilities to consistently encode clinical care guidelines. A recent taxonomy evaluation of CDS malfunctions identified that uneven code set updates, software updates, data migration between systems, and the difficulty of implementing CDS within local environments were consistent across distinct and separate evaluated environments [87]. Of the interoperability models discussed in the introduction, the challenge of heterogeneous implementations of workflow will continue to pose significant barriers to artifact sharing for CDS, and will influence the expansion of CDS applications that require standard EHR API's.

One promising new initiative for interoperable CDS development is the recently announced American Medical Association (AMA) Integrated Health Model Initiative (IHMI) partnership with major HIT vendors including Epic, Cerner, and IBM, as well as the standards organization SNOMED and the American Medical Informatics Association (AMIA). IHMI seeks to incorporate best practices to manage interoperability and care models to support continuous learning health environment [88]. This sponsorship by AMA is a significant recognition of the need to develop new models of partnership between vendors, researchers, and clinical professions.

In 2007, the National Academy of Medicine (formerly Institute of Medicine) proposed the Learning Health System (LHS), where learning from current state and iterative health improvements are a routine and continuous process [89]. CDS and HIT are at the heart of LHS where CDS guides practice, practice outcomes, and performance-derived new knowledge, which are further incorporated into CDS, iteratively learning and improving healthcare, and reducing costs. The LHS has gained momentum in many areas including CDS, yet it remains difficult to observe or demonstrate in practice. The architecture cycle of the current version of the LHS includes Data to Knowledge (D2K), where data is converted into knowledge, Knowledge to Performance (K2P), where knowledge influences performance, and Performance to Data (P2D), where changes in performance generate new data which triggers the next iterative cycle. The K2P is touted to provide recommendations to all stakeholders in the healthcare ecosystem and drive implementable changes from the individual level up to the system level, thus transcending the current CDS paradigm [90] [91]. The themes, policies, HIT infrastructures, and required competences (data science and implementation research) shared by LHS and precision medicine initiatives overlap, and suggestions are to synergize both initiatives [92].

Conclusions

Many biomedical informatics concerns involving access to data, security and privacy technologies, standards for interoperability, evaluation models, as well as data ownership and governance remain challenging and active research/implementation problems for CDS. In the review period, these challenges were not uniquely different between national or international research, though different motivators such as evolving federal regulatory guidance and commercialization may differentiate the activity in the US versus in other countries in the near future. Across these areas of innovation, we noted concerted but not well-connected efforts on how to define and engage patients and communities in the development of trusted and reciprocal data sharing that will be fundamental to CDS development. The field and the expanding stakeholders reflect the central importance of CDS system development to digital health and will be increasingly dependent on biomedical informatics collaborations to design and evaluate clinical effectiveness in these broad settings.

Acknowledgments

This work was supported in part by NCATS grant UL1TR001860 from the National Institutes of Health (USA).

• References

• 1 Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A Roadmap for National Action on Clinical Decision Support. J Am Med Inform Assoc 2007; Mar 1; 14 (02) 141-5
• 2 Osheroff J, Teich J, Levick D, Saldana L, Velasco F, Sittig D. , et al. Improving outcomes with clinical decision support: an implementer's guide. Chicago, IL: Healthcare Information and Management Systems Society (HIMSS); 2012
• 3 Greenes RA. Health Information Systems 2025. In: Weaver CA, Ball MJ, Kim GR, Kiel JM. , editors. Healthcare Information Management Systems: Cases, Strategies, and Solutions. Cham: Spring-er International Publishing; 2016: 579-600 . (Healthcare Information Management Systems: Cases, Strategies, and Solutions).
• 4 Middleton B, Sittig DF, Wright A. Clinical De-cision Support: a 25 Year Retrospective and a 25 Year Vision. Yearb Med Inform 2016; Aug 2; (Suppl. 01) S103-16
• 5 Jenders RA. Advances in Clinical Decision Support: Highlights of Practice and the Liter-ature 2015-2016. Yearb Med Inform 2017; Aug 19; 26 (01) 125-32
• 6 Mandl KD, Mandel JC, Kohane IS. Driving Innovation in Health Systems through an Apps-Based Information Economy. Cell Syst 2015; Jul 29; 1 (01) 8-13
• 7 Clinical and Patient Decision Support Software: Draft Guidance for Industry and Food and Drug Administration Staff. fda.gov Washington, D.C; Dec 8, 2017.
• 8 Fostering Medical Innovation: A Plan for Dig-ital Health Devices; Software Precertification Pilot Program [Internet]. Washington, D.C.: FDA. Available from: https://www.federalreg-ister.gov/documents/2017/07/28/2017-15891/fostering-medical-innovation-a-plan-for-digi-tal-health-devices-software-precertification-pilot
• 9 Voluntary Guidelines for the Design of Clinical Decision Support Software to Assure the Central Role of Healthcare Professionals in Clinical Decision- Making [Internet]. . Washington, DC 20037: CDS Coalition; 2017 Apr. Available from: http://cdscoalition.org/wp-content/uploads/2017/04/CDS-3060-Guidelines-032717-with-memo.pdf
• 10 Labkoff SE, Sittig DF. Who Watches the Watchers. Appl Clin Inform 2017; 8 (02) 680-5
• 11 Health IT Safety Center Roadmap. healthitsafety. org Jul 21, 2015
• 12 2017 Comprehensive Accreditation Manual for Hospitals (CAMH) | Joint Commission Resources. Joint Commission; 2016 Dec 1. Available from: https://www.jcrinc.com/2017-comprehensive-ac-creditation-manuals/2017-comprehensive-accred-itation-manual-for-hospitals-camh-/
• 13 Sow DM. Big Data Analytical Technologies and Decision Support in Critical Care. In: Healthcare Information Management Systems. Cham: Springer; 2016: 515-27 . (Health Informatics).
• 14 Bai Y, Sow D, Vespa P, Hu X. Real-Time Pro-cessing of Continuous Physiological Signals in a Neurocritical Care Unit on a Stream Data Analytics Platform. In: Intracranial Pressure and Brain Monitoring XV. Cham: Springer; 2016: 75-80 . (Acta Neurochirurgica Supplement; vol. 122)
• 15 Adams J, Lieng M, Kuhn B, Guo E, Simonian E. , et al. Automated Mechanical Ventilator Waveform Analysis of Patient-Ventilator Asynchrony. Chest 2015; 148 (04) 175A
• 16 Suresh S. Big Data and Predictive Analytics: Applications in the Care of Children. Pediatric Clin North Am 2016; Apr; 63 (02) 357-66
• 17 Marco-Ruiz L, Pedrinaci C, Maldonado JA, Panziera L, Chen R, Bellika JG. Publication, discovery and interoperability of Clinical Decision Support Systems: A Linked Data approach. J Biomed Inform 2016; Aug; 62: 243-64
• 18 Zini EM, Lanzola G, Bossi P, Quaglini S. An Environment for Guideline-based Decision Support Systems for Outpatients Monitoring. Methods Inf Med 2017; Aug 11; 56 (04) 283-93
• 19 Rodriguez-Loya S, Kawamoto K. Newer Architec-tures for Clinical Decision Support. In: Clinical Decision Support Systems. Cham: Springer; 2016: 87-97 . (Health Informatics; vol. 8).
• 20 Ali T, Hussain M, Ali Khan W, Afzal M, Hussain J, Ali R. , et al. Multi-model-based interactive au-thoring environment for creating shareable medical knowledge. Comput Methods Programs Biomed 2017; Oct; 150: 41-72
• 21 Liu X, Deng R, Choo K-KR, Yang Y. Privacy-Pre-serving Outsourced Clinical Decision Support Sys-tem in the Cloud. IEEE Trans Serv Comput 1-1
• 22 Huang Y-P. The service oriented architecture in decision support system. IEEE 2016; 868-72
• 23 Goldberg HS, Paterno MD, Grundmeier RW, Rocha BH, Hoffman JM, Tham E. , et al. Use of a remote clinical decision support service for a multicenter trial to implement prediction rules for children with minor blunt head trauma. Int J Med Inform 2016; Mar 1; 87: 101-10
• 24 Kuppermann N, Holmes JF, Dayan PS. Head in-jury decision rules in children. Lancet 2017; Sep 23; 390 (10101): 1487-8
• 25 Ohno-Machado L. Using health information technology for clinical decision support and pre-dictive analytics. J Am Med Inform Assoc 2017; Jan 1; 24 (01) 1-1
• 26 OpenCDS Home [Internet]. opencds.org. [cited 2017 Dec 12]. Available from: http://www.opencds.org/
• 27 HL7 Standards Product Brief - HL7 Fast Health-care Interoperability Resources Specification (FHIR®), DSTU Release 1. hl7.org
• 28 Dowell D, Haegerich TM, Chou R. CDC Guideline for Prescribing Opioids for Chronic Pain—United States, 2016. JAMA 2016; Apr 19; 315 (15) 1624
• 29 CQF Home - ONC Tech Lab Standards Coordi-nation - Confluence [Internet]. oncprojectracking. healthit.gov. [cited 2017 Dec 11]. Available from: https://oncprojectracking.healthit.gov/wiki/dis-play/TechLabSC/CQF+Home
• 30 Welcome to openEHR [Internet]. openehr.org. [cited 2017 Dec 11]. Available from: http://www.openehr.org/home
• 31 González-Ferrer A, Peleg M, Marcos M, Maldonado JA. Analysis of the process of representing clinical statements for decision-support applica-tions: a comparison of openEHR archetypes and HL7 virtual medical record. J Med Syst 2016; May 21; 40 (07) 744
• 32 Kane-Gill SL, O’Connor MF, Rothschild JM, Selby NM, McLean B, Bonafide CP. , et al. Technologic Distractions (Part 1): Summary of Approaches to Manage Alert Quantity With Intent to Reduce Alert Fatigue and Suggestions for Alert Fatigue Metrics. Crit Care Med 2017; Sep; 45 (09) 1481-8
• 33 Khalifa M, Zabani I. Improving Utilization of Clinical Decision Support Systems by Reducing Alert Fatigue: Strategies and Recommendations. Stud Health Technol Inform 2016; 226: 51-4
• 34 McCallie DP. Clinical Decision Support: History and Basic Concepts. In: Healthcare Information Management Systems. Cham: Springer; 2016: 3-19 . (Health Informatics).
• 35 Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. J Am Med Inform Assoc 2016; Aug 24; 23 (05) 899-908
• 36 Sinha S, Jensen M, Mullin S, Elkin PL. Safe Opioid Prescription: A SMART on FHIR Approach to Clinical Decision Support. Online J Public Health Inform 2017; 9 (02) e193
• 37 Warner JL, Rioth MJ, Mandl KD, Mandel JC, Kreda DA, Kohane IS. , et al. SMART precision cancer medicine: a FHIR-based app to provide genomic information at the point of care. J Am Med Inform Assoc 2016; Jun 29; 23 (04) 701-10
• 38 Wang Y, Kung L, Byrd TA. Big data analytics: Understanding its capabilities and potential benefits for healthcare organizations. Technological Forecasting and Social Change. North-Holland 2018; Jan 1; 126: 3-13
• 39 Ferrucci D, Levas A, Bagchi S, Gondek D, Mueller ET. . Watson: Beyond Jeopardy! Artif Intell 2013; Jun 1;199-200: 93-105
• 40 Devarakonda MV, Mehta N. Cognitive Computing for Electronic Medical Records. In: Healthcare In-formation Management Systems. Cham: Springer; 2016: 555-77 . (Health Informatics).
• 41 Zhou N, Lv H, Zhang C, Li T, Zhu J, Jiang M. , et al. P1.01-069 Clinical Experience with IBM Watson for Oncology (WFO) Cognitive System for Lung Cancer Treatment in China. Journal of Thoracic Oncology . Elsevier; 2017; Nov 1; 12 (11) S1921
• 42 Chen Y, Elenee Argentinis JD, Weber G. IBM Wat-son: How Cognitive Computing Can Be Applied to Big Data Challenges in Life Sciences Research. Clin Ther 2016; Apr 1; 38 (04) 688-701
• 43 Zhang XC, Zhou N, Zhang CT, Lv HY, Li TJ, Zhu JJ. , et al. 544PConcordance study between IBM Watson for Oncology (WFO) and clinical practice for breast and lung cancer patients in China. Ann Oncol 2017 Nov 1; 28:(suppl_10)
• 44 Aronson SJ, Williams MS. Genetics Aware Clinical Decision Support. In: Genomic and Precision Medicine. Elsevier 2017; 205-15
• 45 Masys DR. Electronic Health Records and Genomic Medicine. In: Genomic and Precision Medicine. Elsevier; 2017. p. 131–42
• 46 Ginsberg G, Willard H. , editors. Genomic and Precision Medicine: Foundations, Translation, and Implementation, 3rd Edition. Academic Press; 2017. 1 p.
• 47 Cutting E, Banchero M, Beitelshees AL, Cimino JJ, Fiol GD, Gurses AP. , et al. User-centered design of multi-gene sequencing panel reports for clinicians. J Biomed Inform 2016; Oct; 63: 1-10
• 48 Klinkenberg-Ramirez S, Neri PM, Volk LA, Samaha SJ, Newmark LP, Pollard S. , et al. Evaluation: A Qualitative Pilot Study of Novel Information Technology Infrastructure to Commu-nicate Genetic Variant Updates. Appl Clin Inform 2016; 7 (02) 461-76
• 49 Heale BSE, Overby CL, Del Fiol G, Rubinstein WS, Maglott DR, Nelson TH. , et al. Integrating Genomic Resources with Electronic Health Re-cords using the HL7 Infobutton Standard. Appl Clin Inform 2016; 7 (03) 817-31
• 50 Manzi SF, Fusaro VA, Chadwick L, Brownstein C, Clinton C, Mandl KD. , et al. Creating a scalable clinical pharmacogenomics service with automated interpretation and medical record result integration – experience from a pediatric tertiary care facility. J Am Med Inform Assoc 2016; Dec 30; 24 (01) 74-80
• 51 Hinderer M, Boeker M, Wagner SA, Lablans M, Newe S, Hülsemann JL. , et al. Integrating clinical decision support systems for pharmacogenomic testing into clinical routine - a scoping review of designs of user-system interactions in recent system development. BMC Med Inform Decis Mak 2017; Jun 6; 17 (01) 450
• 52 Pharmacogenomic Resource for Enhanced Decisions in Care and Treatment (PREDICT) [Internet]. mc.vanderbilt.edu. [cited 17AD Dec 12]. Available from: http://www.mc.vanderbilt.edu/documents/predictpdx/files/PREDICT%20Fact%20Sheet%20for%20Providers%202-15-12.pdf
• 53 Caraballo PJ, Hodge LS, Bielinski SJ, Stewart AK, Farrugia G, Schultz CG. , et al. Multidisciplinary model to implement pharmacogenomics at the point of care. Genet Med 2016; Sep 22; 19 (04) 421-9
• 54 O'Donnell PH, Wadhwa N, Danahey K, Borden BA, Lee SM, Hall JP. , et al. Pharmacogenom-ics-Based Point-of-Care Clinical Decision Support Significantly Alters Drug Prescribing. Clin Phar-macol Ther 2017; Jun 15; 102 (05) 859-69
• 55 Peterson JF, Field JR, Shi Y, Schildcrout JS, Denny JC, McGregor TL. , et al. Attitudes of clinicians following large-scale pharmacogenomics implementation. Pharmacogenomics J 2015; Aug 11; 16 (04) 393-8
• 56 Herr T, Denny J, Hakonarson H, Hartzler A, Hripcsak G, Kannry J. , et al. Practical considerations in genomic decision support: The eMERGE experience. J Pathol Inform 2015; 6 (01) 50
• 57 Freimuth RR, Formea CM, Hoffman JM, Matey E, Peterson JF, Boyce RD. Implementing Ge-nomic Clinical Decision Support for Drug-Based Precision Medicine. CPT Pharmacometrics Syst Pharmacol 2017; Mar 15; 6 (03) 153-5
• 58 Smoller J, Karlson E, Green R, Kathiresan S, MacArthur D, Talkowski M. , et al. An eMERGE Clinical Center at Partners Personalized Medicine. JPM 2016; Dec; 6 (04) 5
• 59 Kart Ö, Mevsim V, Kut A, Yürek İ, Altın AÖ, Yılmaz O. A mobile and web-based clinical decision support and monitoring system for diabetes mel-litus patients in primary care: a study protocol for a randomized controlled trial. BMC Med Inform Decis Mak 2017; Nov 29; 17 (01) 646
• 60 Seto E, Ware P, Logan AG, Cafazzo JA, Chapman KR, Segal P. , et al. Self-Management and Clinical Decision Support for Patients With Complex Chronic Conditions Through the Use of Smartphone-Based Telemonitoring: Randomized Controlled Trial Protocol. JMIR Res Protoc 2017; 6 (11) e229
• 61 Jackson BD, Con D, De Cruz P. Design consid-erations for an eHealth decision support tool in inflammatory bowel disease self-management. Intern Med J 2017 Nov 14
• 62 Guo Y, Chen Y, Lane DA, Liu L, Wang Y, Lip GYH. Mobile Health Technology for Atrial Fibrillation Management Integrating Decision Support, Edu-cation, and Patient Involvement: mAF App Trial. Am J Med 2017; Dec; 130 (12) 1388-1396.e6
• 63 Seaman JB, Arnold RM, Scheunemann LP, White DB. An Integrated Framework for Effective and Efficient Communication with Families in the Adult Intensive Care Unit. Ann Am Thorac Soc 2017; Jun; 14 (06) 1015-20
• 64 Otte-Trojel T, Rundall TG, de Bont A, van de Klundert J, Reed ME. The organizational dynamics enabling patient portal impacts upon organizational performance and patient health: a qualitative study of Kaiser Permanente. BMC Health Serv Res 2nd ed. 2015; Dec 16; 15 (01) 2034
• 65 Fraccaro P, Vigo M, Balatsoukas P, van der Veer SN, Hassan L, Williams R. , et al. Presentation of laboratory test results in patient portals: influence of interface design on risk interpretation and visual search behaviour. BMC Med Inform Decis Mak 2018; Feb 12; 18 (01) 707
• 66 Williams JL, Rahm AK, Zallen DT, Stuckey H, Fultz K, Fan AL. , et al. Impact of a Patient-Facing Enhanced Genomic Results Report to Improve Understanding, Engagement, and Communication. J Genet Counsel 2017; Dec 4; 23 (04) 281
• 67 Beneficiary Engagement and Incentives: Shared Decision Making (SDM) Model | Center for Medi-care & Medicaid Innovation [Internet]. CMS.gov; 2016 Dec. Available from: https://innovation.cms.gov/initiatives/Beneficiary-Engagement-SDM
• 68 Beneficiary Engagement and Incentives Models: Direct Decision Support Model (DDM) Center for Medicare & Medicaid Innovation [Internet]. CMS.gov; 2016 Aug. Available from: https://innovation.cms.gov/initiatives/Beneficiary-En-gagement-DDS/
• 69 Kawamoto K, Anstrom KJ, Anderson JB, Bosworth HB, Lobach DF, McAdam-Marx C. , et al. Long-Term Impact of an Electronic Health Record-En-abled, Team-Based, and Scalable Population Health Strategy Based on the Chronic Care Model. AMIA Annu Symp Proc 2016; 2016: 686-95
• 70 Rajeevan N, Niehoff KM, Charpentier P, Levin FL, Justice A, Brandt CA. , et al. Utilizing patient data from the veterans administration electronic health record to support web-based clinical decision support: informatics challenges and issues from three clinical domains. BMC Med Inform Decis Mak 2017; Jul 19; 17 (01) 135
• 71 Bekemeier B, Park S. Development of the PHAST model: generating standard public health services data and evidence for decision-making. J Am Med Inform Assoc 2017; Nov 2; 95 (06) 930
• 72 Manrai AK, Cui Y, Bushel PR, Hall M, Karakitsios S, Mattingly CJ. , et al. Informatics and Data Ana-lytics to Support Exposome-Based Discovery for Public Health. Annu Rev Public Health 2017; Mar 20; 38 (01) 279-94
• 73 Clinical Decision Support for Immunization (CDSi): Logic Specification for ACIP Recommen-dations. cdc.gov Aug 25, 2017
• 74 What We Do - Adapting Clinical Guidelines for the Digital Age | OPHSS | CDC [Internet]. cdc. gov. [cited 2017 Dec 11]. Available from: https://www.cdc.gov/ophss/WhatWeDoACG.html
• 75 Goodman KW. Ethical and Legal Issues in Deci-sion Support. In: Clinical Decision Support Sys-tems. Cham: Springer; 2016. . pp. 131–46. (Health Informatics; vol. 102).
• 76 GA4GH Strikes Formal Collaborations with 15 International Genomic Data Initiatives [Internet]. ga4gh.org. [cited 2017 Dec 11]. Available from: https://www.ga4gh.org/news/sAhZCeJjS96QHh-VPIYwwWA.article
• 77 Global Alliance for Genomics and Health: Privacyand Security Policy. 26 ed. ga4gh.org. 2015 May
• 78 Framework for responsible sharing genomic and health related data [Internet]. ga4gh.org. 2015. Available from: https://www.ga4gh.org/ga4gh-toolkit/regulatoryandethics/framework-for-re-sponsible-sharing-genomic-and-health-relat-ed-data/#%7B%22-%22:%7B%7D%7D
• 79 Salerno J, Knoppers BM, Lee LM, Hlaing WM, Goodman KW. Ethics, big data and computing in epidemiology and public health. Ann Epidemiol 2017; May; 27 (05) 297-301
• 80 Hollis KF. To Share or Not to Share: Ethical Acquisition and Use of Medical Data. AMIA Jt Summits Transl Sci Proc. Berlin, Heidelberg: American Medical Informatics Association; 2016. ;2016(Chapter 10): 420-7
• 81 Rosenbaum L. Bridging the Data-Sharing Divide — Seeing the Devil in the Details, Not the Other Camp. http://dxdoiorg/101056/NEJMp1704482 . Massachusetts Medical Society; 2017 Apr 26;376(23):NEJMp1704482–2203
• 82 allofus.nih.gov [Internet]. [cited 2017 Dec 13]. Available from: https://allofus.nih.gov
• 83 Million Veteran Program (MVP) [Internet]. re-search.va.gov. [cited 2017 Dec 12]. Available from: https://www.research.va.gov/mvp/
• 84 Verily Life Sciences [Internet]. [cited 2017 Dec 12]. Available from: https://verily.com/projects/precision-medicine/baseline-study/
• 85 Johnson III RJ. A Comprehensive Review of an Electronic Health Record System Soon to Assume Market Ascendancy: EPIC®. J Healthc Commun 2016 Sep 23; 1(4).
• 86 Holmgren AJ, Adler-Milstein J, McCullough J. Are all certified EHRs created equal? Assessing the relationship between EHR vendor and hospital meaningful use performance. J Am Med Inform Assoc 2017; Nov 24; 50 (06) 1751
• 87 Wright A, Ai A, Ash J, Wiesen JF, Hickman T-TT, Aaron S. , et al. Clinical decision support alert malfunctions: analysis and empirically derived taxonomy. J Am Med Inform Assoc 2017 Oct 16
• 88 AMA Integrated Health Model Initiative (IHMI) Collaboration Ecosystem [Internet]. [cited 2017 Dec 12]. Available from: https://ama-ihmi.org/groups/ihmi-community
• 89 The Learning Health Care System in America : Health and Medicine Division [Internet]. nation-alacademies.org. [cited 2017 Dec 13]. Available from: http://www.nationalacademies.org/hmd/ctivities/Quality/LearningHealthCare.aspx
• 90 Friedman CP, Rubin JC, Sullivan KJ. Toward an Information Infrastructure for Global Health Improvement. Yearb Med Inform 2017; Aug 19; 26 (01) 16-23
• 91 Lessard L, Michalowski W, Fung-Kee-Fung M, Jones L, Grudniewicz A. Architectural frame-works: defining the structures for implementing learning health systems. Implementation Sci 2017; 12 (01) e63
• 92 Nwaru BI, Friedman C, Halamka J, Sheikh A. Can learning health systems help organisations deliver personalised care?. BMC Med 2017; 15 (01) 12