Developing and Implementing a Clinical Informatics Curriculum

Abstract

Objectives We developed and implemented a customized internal clinical informatics (CI) curriculum for the UC Irvine CI Fellowship program. The goal was to transition from an externally sourced curriculum to a more focused, internally developed program, aligning with feedback from early fellows and the evolving practical needs of clinical informatics. By designing this curriculum in-house, we sought to provide a more efficient, cost-effective, and relevant educational experience for fellows.

Methods The curriculum was designed over a one-year period, beginning in July 2021 and launched in July 2022. The development process involved collaboration among core clinical informatics faculty, organizing the content into 13 thematic blocks, each spanning four weeks. Each block included a mix of pre-recorded lectures for asynchronous learning and interactive sessions for applied, synchronous learning. The curriculum was designed to cycle twice over the course of the two-year fellowship, with second-year fellows taking on greater teaching responsibilities to solidify their knowledge. Feedback was gathered from graduating fellows and used to iteratively refine the content and structure.

Results Feedback from fellows during the first year of implementation was overwhelmingly positive, with evaluations indicating high satisfaction regarding the relevance, focus, and practical application of the content. Quantitative and qualitative feedback suggested that fellows found the internal curriculum more aligned with their learning goals compared to the prior external curriculum. Modifications were made based on fellow input to adjust the number and structure of interactive sessions, ensuring high-yield learning.

Conclusion This case study highlights the successful development and implementation of a customized clinical informatics curriculum at UC Irvine. The curriculum offers a more tailored, responsive, and comprehensive educational model, addressing both financial constraints and the need for a practical, focused learning experience. This initiative provides valuable insights and a potential framework for other institutions seeking to transition to an internal informatics curriculum.

Keywords

training and education requirements - strategies for health IT training - accreditation and licensure - continuing professional development - continuing education

Background and Significance

The UC Irvine (UCI) Clinical Informatics (CI) Fellowship program was established to promote excellence in CI. Accredited in 2017, the program has sought to provide a comprehensive educational experience, leveraging both internal expertise and external collaborations. However, feedback from early cohorts highlighted the need for a curriculum more tailored to the practical and evolving needs of the field, leading to the development of an internal, modular curriculum. A survey of recent informatics fellow graduates from 2016 to 2024 found that almost a third had minimal or no informatics background upon entering the fellowship.[1] This speaks to the importance of a comprehensive curriculum to ensure fellowship graduates are prepared for clinical practice and the CI board exam.

Central to our program's success has been its diverse and expert faculty. Our program has eight core faculty that are board-certified physicians in CI, whose collective experience and knowledge form the backbone of the fellowship's educational and research endeavors. Complementing this group is extended faculty, which brings together an additional five physician informaticists and three PhD faculty, each contributing specialized expertise and perspectives to the program.

Prior literature on CI curriculum mainly focuses on the content provided during the curriculum and the benefits of an internal curriculum as described in McClintock et al[2] and Lingham et al.[3] Another article by Whitfill et al focuses on implementing a novel imaging informatics curriculum for fellows but this also focuses on content and curriculum format.[4] This paper is novel in that it not only reviews the important core informatics content, a proposed block schedule, and information delivery format but it also provides a framework, timeline, and guide for curriculum implementation. In addition, this article outlines common pitfalls to be aware of when attempting to implement an internal curriculum, so those attempting to implement a curriculum can have strategies in place to mitigate common risks.

Background

From its inception in 2017 until 2021, UCI's approach to education in CI included a collaboration with an outside academic organization. During this period, the program invested in an external lecture curriculum, providing fellows with a broad range of insights, and learning opportunities from leading experts in the field. This partnership was a crucial component of the fellowship's early years, laying a foundation upon which the program has continued to build and evolve. The external curriculum was primarily designed as a conventional university degree program. This degree spanned 24 months and required core classes as well as electives. This curriculum was perceived as less practical, often involving a substantial number of written assignments, such as essays and chapter summations. These tasks contributed to a sense of busy work among the fellows. Feedback was received that some of the required coursework contained significantly more technical depth than was likely to be employed in most informatics careers. It also featured a significant portion of elective courses, providing the fellows with a variety of learning options and the flexibility they appreciated. However, this format led to uncertainties for both the fellows individually and program leadership concerning the consistency and comprehensiveness of the content covered by each fellow.

Despite the breadth of learning opportunities, feedback from the first three classes of fellows indicated a mismatch between the curriculum's focus and the fellows' educational needs. They also expressed concern that certain aspects of the external curriculum had lower educational value relative to the time invested. This feedback, alongside considerations of content control, cost, and educational focus, prompted a reevaluation of the curriculum approach.

Another significant factor in the decision to make an internal curriculum was the continued financial impact of paying for an external curriculum. Across the country, there is quite a varied range of funding structures for CI fellowships.[5] At UCI, the ability to save funds with an internal curriculum was extremely helpful for our yearly budget.

Curriculum Design and Implementation

The development and implementation of the UCI curriculum involved a structured and phased approach. The curriculum was structured around 13 distinct blocks, with each block spanning 4 weeks. The curriculum's initial design included one pre-recorded hour lecture and a 1-hour interactive session each week. The curriculum was later revised to have fewer interactive sessions based on the fellows' feedback. The recent increase in asynchronous and hybrid education models that flourished after the coronavirus disease 2019 pandemic impacted many of our curriculum design decisions.[6] The pre-recorded lectures offered flexibility for fellows to access and review the material as needed, while the interactive sessions facilitated active learning. The interactive sessions could be in person, though most faculty, and fellows, preferred to have the interactive sessions remotely using video conference call technology.

The curriculum was designed to be phased with graduated responsibility. To deepen understanding and ensure thorough comprehension of the curriculum, the program is designed to have the fellows cycle through the entire curriculum twice over the course of their 2-year fellowship. This repetition, especially for second-year fellows, includes graduated responsibilities, encouraging them to engage more deeply and apply the knowledge they have acquired in their first year. Second-year fellows are given different assignments that are in-line with their level of training, and they are expected to take on a teaching role during the interactive sessions to further solidify their knowledge.

The curriculum's goal is to produce competent informaticists who will have the needed skills to function independently and pass their informatics boards. In addition, the goal of assignments and interactive sessions was to strengthen board-relevant knowledge, allow for practical application of core topics, and provide frequent opportunities for fellows to ask questions and clarify any knowledge gaps. Given the breadth of knowledge and skills required to complete an informatics fellowship, our goal was to make the curriculum as efficient and high-yield as possible. This meticulous structure aims to provide a comprehensive and immersive learning experience, tailored to the evolving needs of CI professionals.

Our curriculum's overarching framework is anchored on the board exam content outline published by the American Board of Preventative Medicine,[7] to guarantee comprehensive coverage across all relevant topics. In addition, we aimed to cover and supplement key learning milestones during the curriculum.[8] In particular, if there were areas where our fellows had fewer hands-on opportunities to meet specific milestones, we would incorporate those key learning points into the curriculum. We further enriched our curriculum by integrating the “Domains, tasks, and knowledge for CI subspecialty practice” as delineated by Howard Silverman et al[9] and “A Systematic Approach to the Design and Implementation of Clinical Informatics Fellowship Programs” by Lingham et al,[3] providing a structured overview of CI practice areas. The development process leveraged essential texts by Mankowitz,[10] Finnell and Dixon,[11] and the foundational “Biomedical Informatics” by Shortliffe et al,[12] ensuring our content is both relevant and grounded in the field's core knowledge.

Our curriculum's content creation process was systematic, collaborative, and spanned 1 year. We were fortunate to have many engaged faculty who agreed to contribute to the curriculum. Each UCI CI faculty member who agreed to contribute was assigned as the block lead for one or two blocks, depending on their areas of expertise and level of involvement in the fellowship. At UCI, we have 13 CI faculty, with 10 of them being core faculty. We had nine core faculty initially agree to participate in the curriculum. Four of the faculty were block leads for two blocks, while the remaining five faculty were block leads for one block. For each block's specific content, we assessed our faculty's strengths, aligning their expertise and interest in pertinent topics. This curriculum was organized into 4-week blocks, guided by two principal criteria: the interrelatedness of topics to ensure thematic coherence, and the alignment with faculty expertise to maximize instructional depth. This approach enabled us to craft a curriculum that is both comprehensive and deeply informed by the unique strengths of our faculty, and ensuring a broad yet appropriately detailed and practical exploration of CI. For the limited number of areas that fell outside of the expertise of our core faculty, we encouraged the block leads to source guest lectures from content experts from our Department of Information Technology Services, and from our PhD faculty at the UCI Donald Bren School of Information and Computer Sciences. Our program maintains close operational and research relationships with both divisions of UCI. For a detailed view of the curriculum design and block schedule, refer to the end of the article.

The curriculum development team provided each block lead with an outline highlighting key concepts to be covered, suggested readings and reference materials to support the content, and suggested guest lecturers. Block leads had the flexibility to invite guest speakers to enrich the content or to deepen their understanding of the topic. To ensure timely development, a detailed timeline was established, specifying deadlines for submitting outlines, recording lectures, and finalizing interactive session content. This structured approach spanned an entire year, beginning in July 2021, with the aim of launching the curriculum in July 2022. This timeline and structure facilitated the orderly development of the curriculum, ensuring that each block was thoroughly prepared and aligned with the overall educational goals. For a specific timeline, please refer to [Supplementary Appendix 1] (available in the online version only).

Fig. 1 (A–H) Curriculum survey questions.

To keep the recorded content updated, program leadership reviews aggregate written and verbal fellow feedback annually and requests directed changes from block leads. Block leads are also asked to update recorded content annually to reflect changes in the field and to support their corresponding interactive sessions.

Challenges and Solutions

During the curriculum design process, several challenges emerged, notably faculty accountability and delays in completing tasks. The curriculum's development spanned a year, with block leads assigned their sections by August 2021 for a planned July 2022 launch. Deadlines for various content creation stages, such as syllabus development, slide creation, and lecture recording, were set to maintain progress. Despite reminders and check-ins for missed deadlines, many leads struggled to meet their timelines, resulting in some needing to deliver lectures live. We estimate the time spent creating one full block of content ranged from 20 to 50 hours. This can be a significant amount of time, especially if the block lead is creating their own content. In addition, the pressures on academic faculty, who juggle clinical duties without additional time allocated for teaching, contributed to the content delays.

Additionally, faculty attrition posed a unique challenge. One of our faculty that was notably behind on our development timeline left shortly before the curriculum's launch, requiring urgent efforts to complete their block. Fortunately, we have a core group of motivated and dependable faculty. When we were informed that one of our block leads would be leaving for another institution, we were able to quickly pivot, assign a new block lead, and create content in time for the start of the block.

We had 10 block leads covering 13 blocks at the beginning of the year. One faculty member had a family emergency, so another faculty member covered his blocks. As previously mentioned, we had another faculty leave for another institution, so his blocks were also redistributed. Outside of these two faculty members, block content was late for 50% of the remaining faculty and 30% of the lectures that were supposed to be pre-recorded were live during the first block.

Another challenge was ensuring that the collective content covered the breadth of board-relevant and practical informatics material. During the curriculum design process, we outlined a comprehensive list of topics that were broken up and organized into blocks. Within each block, we listed topics for each block lead to cover, though we allowed some creative liberties regarding the exact content covered. Given some of the delays in content creation mentioned above, it was challenging to holistically review the content prior to the go-live date. To mitigate this, we reviewed the content as it was created to ensure it matched with the outlined content provided to the block lead. In addition, we adjusted the content based on feedback from the fellows after the completion of the first full year of content.

Outcomes and Feedback

Ultimately, the curriculum was launched in July 2022 and completed the first full year of content by June 2023. To ensure appropriate feedback and the ability to adjust as needed for year 2, fellows were required to fill out a survey for each piece of the curriculum. This meant we had documented quantitative and qualitative feedback for each lecture and interactive session for the entire content. To anonymize feedback with only three fellows, our fellowship coordinator aggregated the survey data into figures and compiled the comments. In addition, during the verbal feedback sessions with the fellows, when it was time to discuss a block, that block lead would step out of the meeting so the fellows could provide candid and anonymous feedback. See [Fig. 1] for the survey questions and [Fig. 2] for a summary of the survey results.

Fig. 2 (A–D) Curriculum survey responses.

The internal curriculum's implementation has been met with positive feedback from fellows, highlighting the benefits of a more focused and applicable educational experience. Three fellows filled out 112 evaluations of the curriculum lectures. Their survey responses rated the lectures as “Excellent” (86/112) or “Very Good” (23/112). The three fellows filled out 82 evaluations for the interactive sessions. The fellows responded in 65/82 surveys that they “Strongly agreed” that the sessions were well prepared and presented. The fellows rated 53/82 of the interactive sessions as “Excellent” and 26/82 as “Very Good.” Most of the free response feedback was positive and recommended keeping the lectures in the curriculum for future years.

In addition to the surveys, we sought live and unstructured feedback to ensure the fellows provided maximal input. During these sessions, the most impactful feedback we received was regarding the interactive sessions. At the beginning of the curriculum, it seemed that the interactive sessions for certain blocks were not as organized or structured. The original goal was to have four interactive sessions per block, but the fellows preferred fewer, more high-yield, and focused sessions as opposed to a session every week.

Overall, the feedback from the first year of the curriculum was extremely positive. The fellows felt the curriculum was directed toward appropriate learning goals and was more efficient than the prior external curriculum. The block structure worked well, and they were able to connect more with each block lead. They appreciated the asynchronous lectures and the synchronous didactics for applied learning. The fellows felt that the time spent reviewing the lectures and preparing for interactive sessions was more impactful for their learning than prior assignments in the external curriculum.

Despite the overall positive feedback, there were some blocks that were less helpful than others. The first block was an introductory block to the entire year, which was initially thought to be a nice overview of the curriculum. Feedback from the fellows stated that it was a bit overwhelming because many of the core concepts were mentioned without significant detail or explanation. After this feedback, block one was redesigned to be an overview of CI at our specific institution, without providing as much of a general overview of the entire curriculum.

Program leadership provided summarized fellow feedback individually to each of the block leads. Feedback was shared via email with a digital attachment and shared via a link to an online folder. Identifying marks were removed to protect fellows' identities in alignment with the Accreditation Council for Graduate Medical Education (ACGME) policy on teaching evaluations. In addition to the summarized feedback, in individual meetings, we provided edits to their overall content, specific recommendations for certain lectures, and more directed suggestions for their interactive sessions. We encouraged the block leads to evaluate their interactive sessions to ensure they were high-yield and to consider condensing them into fewer sessions for future years. During year 1 of the curriculum, we monitored the lecture evaluation forms being submitted and held our mid-year evaluation review meeting midway through the year (for blocks 1–7). This was conducted similar to the ACGME-mandated Program Evaluation Committee meetings.[13] We changed the length of two blocks and discussed with each block lead the type of sessions the block lead felt most impactful in retrospect. We approved tentative changes to interactive sessions and then adjusted the schedule accordingly. For example, instead of four 1-hour sessions in a block (as initially designed), some blocks held one 1.5-hour midpoint session and a short 30-minute session at the end of the block. The curriculum's adaptability has allowed for ongoing refinement based on feedback and changes in the field.

Conclusion

The development and implementation of an internal CI curriculum at UCI represents a significant step toward a more tailored, effective, and responsive educational model in the field. This case study highlights the challenges and rewards of curriculum innovation, offering insights for other programs considering similar initiatives.

Clinical Relevance Statement

This paper provides insight into the successes and challenges of implementing an internal curriculum. This article can provide a framework and guidance for other CI programs as they consider transitioning to their own internal curriculum.

Multiple-Choice Questions

1. Which of the following was not a challenge that was faced during the curriculum development process?

   • Lack of interest from faculty in contributing to the curriculum

   • Academic faculty time constraints

   • Faculty attrition

   • Difficulty with completing tasks by designated deadlines

   Correct Answer: The correct answer is option a. The challenges encountered included academic faculty time constraints which led to difficulty with completing tasks by the outlined deadline. In addition, faculty attrition led to challenges with the curriculum content timeline as well. Option a. is the correct answer because there was significant interest and enthusiasm from faculty to contribute to the curriculum, and this was not one of the challenges faced.

2. Which of the following outlines the main format of the curriculum?

   • Twenty-six block curricula completed once. The entire curriculum is asynchronous.

   • Twenty-six block curricula completed once. The curriculum is a mix of synchronous and asynchronous content.

   • Thirteen block curricula completed twice. The entire curriculum is asynchronous.

   • Thirteen block curricula completed twice. The curriculum is a mix of synchronous and asynchronous content.

   Correct Answer: The correct answer is option d. The design of our curriculum is 13 blocks that the fellows completed twice by the end of their 2-year fellowship. Each block has pre-recorded asynchronous lectures and synchronous interactive sessions. The goal of this format is to allow flexibility for fellows to learn content at their own speed when time permits and to schedule designated interactive sessions for solidifying knowledge and opportunities to ask questions.

Table 1 Depiction of curriculum block structure with categorical numeric sequence in column “Number” and the corresponding block topics are listed in column “Block”

Number	Block
1	Fundamental of Clinical Informatics
2	Health Systems
3	Decision Science
4	Computers and Networks
5	IS Lifecycle and Knowledge Repositories
6	Data, Standards, and Interoperability
7	Healthcare as a Business
8	Leadership
9	Project Management
10	QI Workflow Governance
11	Security, Reliability, Integrity
12	AI, Medical Intelligence, and Research Informatics
13	Legal, Ethical, Regulatory

Abbreviations: AI, artificial intelligence; IS, Information System; QI, Quality Improvement.

Table 2 It notes two columns. The first column displays percentages, while the second column specifies board content core areas. Each percentages indicates the portion of each core area that must be covered for the clinical informatics board exam

20%	I. Fundamental knowledge and skills: Fundamental knowledge and skills that provide clinical informaticians with a common vocabulary, a basic knowledge across all Clinical Informatics domains, and an understanding of the environment in which they function.
30%	II. Improving care delivery and outcomes: Knowledge and skills that enable a clinical informatician to develop, implement, evaluate, monitor, and maintain clinical decision support; analyze existing health processes and identify ways that health data and Health Information Systems (HIS) can enable improved outcomes; support innovation in the health system through informatics tools and processes.
15%	III. Enterprise Information Systems: Knowledge and skills that enable a clinical informatician to develop and deploy HIS that are integrated with existing information technology systems across the continuum of care, including clinical, consumer, and public health domains. Develop, curate, and maintain institutional knowledge repositories while addressing security, privacy, and safety considerations.
15%	IV. Data governance and data analytics: Knowledge and skills that enable a clinical informatician to establish and maintain data governance structures, policies, and processes. Incorporate information from emerging data sources; acquire, manage, and analyze health-related data; ensure data quality and meaning across settings; and derive insights to optimize clinical and business decision-making.
20%	V. Leadership and professionalism: Knowledge and skills that enable a clinical informatician to build support and create alignment for informatics best practices; lead health informatics initiatives and innovation through collaboration and stakeholder engagement across organizations and systems.

Table 3 The ACGME milestones that clinical informatics programs use in a semi-annual review of resident/fellow performance, which is reported to the ACGME. Milestones have six core competencies with multiple subcompetencies, which provide developmental framework and narrative descriptions that are targets for resident/fellows performance throughout their educational training

Abbreviation	Milestone competency name
PC1	Consumer Informatics Applications, Portals, and Telehealth
PC2	Existing and Emerging Data Sources
MK1	Project Management
MK2	Health Information Technology (HIT) Knowledge of Current and New Testing, Implementation, and Monitoring
SBP1	HIT Knowledge of Current and New Testing, Implementation, Monitoring
SBP2	Standards and Interoperability
SBP3	Data Integrity/Security
PBLI1	Optimization, Downtime, Functional Requirements
PBLI2	Clinical Decision Support
PBLI3	Analytics
PBLI4	Human–Computer Interaction and User Interfaces
PBLI5	Reflective Practice and Commitment to Personal Growth
PROF1	Governance
PROF2	Mentorship
PROF3	Professional Behavior and Ethical Principles
PROF4	Accountability/Conscientiousness
PROF5	Self-awareness and Help-Seeking
ICS1	Communicate Effectively with Multiple Constituencies
ICS2	Building Consensus
ICS3	Interprofessional and Team Communication
ICS4	Communication within Health care Systems

Table 4 It outlines the details of the curriculum in terms of block name, topic, objective and dates. Additionally, it lists the leading faculty per each block, and the corresponding board content core area and ACGME milestones competencies that each block covers

Number	Block	Block Lead (Faculty)
1	Fundamental of Clinical Informatics
July 1–July 30	Board Content Core Areas: 1, 3	Milestones: MK2, ICS3–4
	Objectives
Week 1	Define biomedical informatics Present an overview of clinical informatics Review a brief history of clinical informatics systems Understand informatics key concepts, models, and theories Review vocabularies, terminology, and nomenclature   Interactive Session: First-year fellows should be prepared to summarize the required reading (15 min). Second-year fellows should prepare and review five Board Review-style questions on this topic (15 min).
Week 2	Review a brief history and structure of U.S. Healthcare System Review the relationship of purchaser, provider, and payer Understand the organization of health care delivery models Review the transition from service-based to performance-based payment models   Interactive Session: First-year fellows should be able to summarize various U.S. health care payer models (15 min). Second-year fellows should be able to discuss their own involvement in current or past IS project(s) at UCI involving performance-based payment interventions to improve outcomes, data analysis, or reporting performance (15 min).
Week 3	Review ethics and professionalism in clinical informatics Understand HIPAA Privacy and Security Rules Understand ARRA and HITECH Acts   Interactive Session: First-year fellows should be able to summarize the required reading (15 min). Senior fellows should prepare and review five Board Review-style questions on this topic (15 min).
2	Health Systems
July 31–August 20	Board Content Core Areas: 1, 3	Milestones: MK2
Syllabus	Objectives
Week 1	Bioinformatics Public health informatics Health care Organizations (AMIA, Joint Commission, ISO, ANSI, SDO) Policy and regulatory frameworks related to the health care system (HIPAA, Patient Access rule, CLIA88, Hitech Act, ARRA, TJC)
Week 2	The flow of data, information, and knowledge within the health system Tools for health care coordination across systems Models (for the provision of health care—ACOs, single payer, etc.)
Week 3	Determinants of individual and population health Forces shaping the health care delivery and considerations regarding health care access Roles in informatics (nursing informatics, CMIO, etc.) Main clinical system functional requirements
Week 4	Ancillary services information systems Pathology Information Systems (workflows and information flows) Radiology Information Systems (workflows and information flows) Anesthesiology Information Systems
3	Decision Science
August 21–September 17	Board Content Core Areas: 1–3	Milestones: PBLI 2–4
Syllabus	Objectives
Week 1	Review binary representation: bits, bytes, ASCII/ISO versus Unicode (HEX) Understand simple Boolean operators (Gates) Understand Adders Understand Flip Flops Review practical applications of Boolean logic in Informatics: information retrieval, basic or pseudocode programming, CDS logic   Interactive session: Faculty will demonstrate in Epic PJX introduction to CER rule build with Boolean operators and order panel (OSQ) build using simple Boolean design for “and/or” logic; linked “and,” “or,” “followed by” and cascading panels (60 min).
Week 2	Understand how to calculate expected value, expected utility understand 2 × 2 Contingency Tables: sensitivity, specificity, TPR, FNR, TNR, FPR, PPV, NPV, LR, HR, OR, prevalence Understand ROC curves: accuracy Understand Bayes Theorem Understand Decision Analysis Tree and Markov Model Matrix   Interactive session: Senior fellow should prepare and review two Board Review-style 2 × 2 contingency table questions, 1 Bayes theorem question, 1 Decision Analysis Tree question, and 1 Markov question. First-year fellows should be able to solve sample questions (60 min).
Week 3	Define CDS Understand the five rights of CDS Review the ten commandments of CDS Review CDS design principles Understand the risks of CDS and barriers to implementation Legal, regulatory, and ethical issues regarding CDS   Interactive session  • Faculty will demonstrate in Epic PJX how to build LGL criteria and base records and introduce CER rule build (60 min).  • Longitudinal hands-on learning will be demonstrated during order-set work build and clinical–technical review meetings when CDS is required and by translating algorithms, pathways, and guidelines into order-set design considerations to incorporate CDS principles.
Week 4	Review knowledge representation systems Understand Arden Syntax (MLM) Recognize the “Curly Brackets Problem” Understand the types of Sematic Ambiguities and Vagueness Understand how semantics influences CDS   Interactive session: CPGs will be reviewed and fellows should be able to identify potential semantics ambiguity and/or vagueness (15 min). Fellows should be able to cite examples of common types of clinical heuristic errors (15 min).
4	Computers and Networks
September 18–October 15	Board Content Core Areas: 1, 3	Milestones: MK2, SBP3, PBLI1
Syllabus	Objectives
Week 1	What is a computer and how do they connect? Computation, including hardware, software (basic computational logic), and network connectivity. Hardware key concepts: CPU, display, human interface devices, memory, I/O Software key concepts: Machine code (bits/bytes), hexadecimal, memory, register, assembly language, Fortran, compilers and interpreters, control structures, pseudocode Network connectivity key concepts: router, switch, hub, bridge, ISO-OSI Communication Architecture Model, basic/common network topologies (point to point, network bus, star, ring, mesh, hybrid), client/server, peer-to-peer, WAN, LAN
Week 2	Databases Data from a device or computer input is managed and stored in a database What is a database (vs. a really big spreadsheet)? Key concepts in database architecture: DBMS, navigational database, hierarchical database, MUMPS (Cache), relational databases, and their relative advantages and disadvantages. Fundamental database rules and A.C.I.D. Brief intro to SQL concepts: primary key, foreign key, joins Modern NoSQL, flat file, and distributed file system usage in health care   Interactive session  • Senior fellows: Each develops two board review questions per lecture (a total of eight). Four are due at the first interactive session and four at the second. You can use outside resources as long as it is related to the general content outlined in the syllabus. References should be provided if appropriate. These should be emailed to me 2 days before the session.  • Junior fellows: For didactic session 1, each prepare a program in pseudocode pertinent to your Clinical specialty, the EHR, or informatics. It should be at least eight lines, contain at least one IF/ELSE clause, and at least one loop (while/until/etc.). Be prepared to share the code and explain the functionality line by line. These should be emailed to me 2 days before the session. For didactic session 2, each identify a downtime protocol in your clinical area and be prepared to discuss it (i.e., what system downtime does it address, why is it important, what are the potential failure points, what are the challenges when transitioning from downtime back to live, etc.).
Week 3	Enterprise data architecture and moving data Safe storage of disparate data elements Key concepts in enterprise data architecture and storage  1. Data warehouse versus data mart versus data lake  2. Master Patient Index  3. Hosting and redundancy  4. Downtimes and data repositories Key concepts in moving data: extract, transform, load, and combine/store different data types Transactional databases and reporting or research databases
Week 4	Data mining, knowledge discovery, and research informatics Basic approach, methodologies, and resources for knowledge discovery, and an understanding of research informatics as a core component of the clinical informaticist's skillset Anonymization, deidentification, and the Safe Harbor method Query tools and techniques, pivot tables, OLAP Clinical Research Informatics DIKW Electronic data capture and CTMS/CRMS CTSA Network i2b2 and SHRINE PCORNet, ACT, OHDSI, TriNetx OMOP Common Data Model COVID N3C
5	IS Lifecycle and Knowledge Repositories
October 16–November 12	Board Content Core Areas: 1, 3, 4	Milestones: PBLI4, SBP1, MK2
Syllabus	Objectives
Week 1	Information Systems—needs analysis, system selection, implementation, integration testing Defining requirements RFP/RFI/RFQ Contract negotiations Costs   Interactive session: RFP/RFI development
Week 2	Information Systems—implementation, maintenance Implementation procedure Transition and decommissioning Change Control Systems Downtime (scheduled vs. unscheduled) Disaster recover Clinician feedback
Week 3	Human–computer interaction: eval, usability engineering/testing HCI/Usability related errors Predictive models of HCI Hick–Hyman Law Fitt's Law Descriptive models of HCI Usability evaluation methods (testing, inspection, inquiry) Neilson's 10 Usability Heuristics, NIST Recommendations
Week 4	Models and theories of HCI, MESH, Search Prototyping Cognitive informatics Information display design models Mental models and errors Information repositories Search, archive, and retrieve   Interactive session: Librarian to review MESH/Search
6	Data, Standards, and Interoperability
November 13–December 10	Board Content Core Areas: 1, 2, 3	Milestones: PC2, SBP1–3
Syllabus	Objectives
Week 1	Standards in health information transmission Development and usage of interoperability/exchange standards like DICOM and FHIR, transaction standards like ANSI X12, and messaging standards like HL7. Lear levels of interoperability and how they apply to these transmission standards
Week 2	Standards, terminologies, taxonomies, and ontologies in health information formatting Development and usage of data standards, terminologies, and ontologies such as those in imaging (DICOM, RadLex), lab (LOINC), pharmacy (RxNorm), and diagnosis (SNOMED) among others (UMLS, MeSH, ICD, HPO, OMIM). Barriers to interoperability as they pertain to these standards, terminologies, and ontologies.   Interactive session: This will primarily focus on Q&A and review of the week's lecture and assigned reading. Senior fellows are expected to very briefly either:  • Present two relevant board-style questions and be prepared to discuss the answers and distractors.  • Present a short new topic to supplement the lecture or reading (no more than 10 min),  • Present a new or different way to conceptualize a topic from the week's material  • Discuss how elements of the week's topic are relevant to an ongoing project
Week 3	Data model standards, data management, data validation, and data lifecycle Common data model and interoperability standards such as FHIR, HL7 CDA, PCORnet, i2b2 (EAV), OMOP, Sentinel, and BRIDG. AHIMA data quality characteristics: accuracy, accessibility, comprehensiveness, consistency, concurrency, currency, definition, granularity, precision, and relevancy particularly as they pertain to the reuse of health care data. Types of data error and their associated mitigation strategies
Week 4	Specialized data sources Specialized and emerging standards and technologies, primarily to include SMART on FHIR. Specialized and emerging data sources and their storage, retrieval, and uses. This may include sources like OMIM, HPO, and PheKB. Since this section is about specialized and emerging data sources, it is likely to be variable in content and change regularly.   Interactive session To prepare for interactive sessions, fellows should come prepared to discuss.  • issues related to integrating emerging data sources into business and CDS  • data associated with workflow processes and clinical context
7	Healthcare as a Business
December 11–January 4	Board Content Core Areas: 1, 5	Milestones: PROF 2–5, ICS 1–4
Syllabus	Objectives
Week 1	Accounting Balance sheet/income statements Operating and capital budgeting Managerial versus financial accounting Financial statement analysis   Interactive session: Review accounting HW basics problems
Week 2	Finance Time value of money Capital budgeting Operations basics Marketing basics Strategic planning in health care White papers   Interactive session  • Review time value of money problems  • Plan to discuss how these concepts are relevant in informatics. Think about current projects you are working on at UCI and how you might have approached them differently or thought about them differently after learning these concepts.
Week 3	Curriculum break
Week 4	Care delivery and payment models Medicare/Medical/PPO/Fee for Service/Bundled Payments/ACO Value-Based Care   Interactive Session Discuss the RVU vs FTE White paper: discussion guided by the Analysis Questions  • Second-year fellows come prepared to discuss their white paper  • Give a 5-minute summary of the issue and the arguments used  • Discuss the process of writing a white paper:  • What was easier than expected? Harder than expected? Plan to discuss thoughts on Creating Value in Health care  • Prepare to present and discuss the articles that you found  • Be able to give a 5-minute summary of your article  • Discuss how it impacts us as health care professionals
8	Leadership
January 5–February 5	Board Content Core Areas: 5	Milestones: ICS 1–4, PROF 1–4, PBLI5
Syllabus	Objectives
Week 1	Leadership Leadership Principles, Models, and Methods Leadership styles Dimensions of Effective Leadership Five Dimensions of Centered Leadership Grid Theory Situational Leadership Model FROHLM, Emotional Intelligence   Interactive session: Manager versus Leader Table
Week 2	Negotiation, Conflict, and Motivation Negotiation Process Negotiation versus Conflict Management Conflict Management Collaboration Motivational Strategies Motivation Theories   Interactive session: Negotiation strategies for doctors and hospitals—group discussion of cases
Week 3	Teamwork Building and Managing Effective Teams Coaching/Mentoring/Cheerleading Team Productivity and Effectiveness Diversity, Equity, and Inclusion Giving Feedback Team Goal-setting Team Charter Components RACI Matrix Team Effectiveness, Tuckman Ladder, Group Management Processes, Consensus Mapping, Managing Meetings   Interactive session: Immersion session: Developing a Team Charter
Week 4	Communication and Learning Communication Strategies and Effective Communication Communication Programs Writing Effectively Intergenerational Communication Techniques Adult Learning Theories Teaching Modalities Methods to Assess the Effectiveness of Training Competency Development   Interactive session: Immersion Session: Developing a socialization plan using seven best practices in change management communication
9	Project Management
February 5–March 3	Board Content Core Areas: 1, 5	Milestones: MK1, ICS 1–4, PROF 1–4, ICS 1–4
Syllabus	Objectives
Week 1	Project Management—project planning and definitions Project versus ongoing operations Basic definitions in project management Project proposal Value of analyzing stakeholders Stakeholder register/analysis Project charter Statement of work   Interactive session: Fellows bring in a prewritten Statement of Work/Charter. Everyone will present their SOW/Charter for discussion and feedback
Week 2	Project Management—project risks Risk analysis How a project manager manages risk Risk planning and analysis process—static versus continuous   Interactive session  • First-year fellow presents their prewritten risk analysis  • Second-year fellow provides feedback  • If time, the second-year fellow presents their risk analysis and the first-year fellow provides feedback  • Board review questions if time
Week 3	Project Management—project scheduling Work breakdown structure Appreciate the importance realistic scheduling Task network and calculating the critical path Gantt Chart Performing and closing a successful project Project life cycles   Interactive session  • First-year fellows: present a WBS and Gantt Chart  • Second-year fellows provide feedback  • If time, second-year fellows present and first-year fellows give feedback  • Review critical path board review questions if time
Week 4	Change management Change management principles, models, and methods Assessment of organizational culture and behavior change theories Motivational strategies, methods, and models   Interactive session: Think about a recent organizational change you were a part of. If you were not a part of one recently then think about one you have observed. Come prepared to talk through the answers to the questions posted on teams
10	QI Workflow Governance
March 4–March 31	Board Content Core Areas: 1, 2, 4, 5	Milestones: PC1, PROF 1, PROF 3–4, ICS 1–4
Syllabus	Objectives
Week 1	Quality improvement Quality improvement principles and practices Distinguish between quality assurance and quality improvement Review QI Methodologies (Six Sigma, Lean, Root cause analysis, plan do study act cycle Quality standards and measures promulgated by quality organizations Review QI tools (Flowchart, Pareto, Ishikawa, Key Driver Diagram)
Week 2	Clinical workflow and process redesign Methods of workflow analysis Definitions of workflow Study of existing workflow Workflow analysis tools Principles of workflow re-engineering Models of workflow re-engineering User-centered design principles Usability testing Key performance indicators (KPIs) Adoption metrics Usability testing Data associated with workflow processes and clinical context   Interactive session: Clinical Workflow redesign scenario 1 and 2
Week 3	Governance Principles of governance Review definitions and responsibilities Governance theories Hospital governance structure
Week 4	Telemedicine Telehealth workflow and resources (e.g., software and hardware) Consumer-facing health informatics applications Regulated medical devices (pumps, telemetry monitors) that may be integrated into information systems Non-regulated medical devices (consumer devices)
11	Security, Reliability, Integrity
April 1–April 28	Board Content Core Areas: 1, 3	Milestones: SBP3, PC1, MK2, SBP1
Syllabus	Objectives
Week 1	Medical devices Classes of regulated medical devices Device communication Medical device standards organization Medical device integration Non-regulated medical devices
Week 2	Reporting requirements and Clinical Communication Channels   Interactive session: Come prepared to discuss the following. Pick a new device that needs integration within your health care institution (maybe an inpatient or ambulatory device). Discuss implementation of the new device with respect to security, integrity, and reliability. Consider best practices communication strategy and be prepared to discuss the following:  • Notification to medical staff of updated policies and procedures with some significant changes embedded within the document  • Updates to EMR  • Initiation of hand washing campaign within a health institution  • Verbal orders in the ICU
Week 3	Security Security threat assessment methods and mitigation strategies Security standards and safeguards     Interactive session: Prepare a case of a large-scale security breach (health system preferred but not required) and be prepared to discuss risk factors that led to the incident, downstream risk to the institution, costs associated with the breach, strategies that could have prevented the breach, and how the entity involved employed mitigation strategies during the event.
Week 4	Knowledge repositories, Data storage, HIE, Registries Information system failure modes and downtime mitigation strategies Approaches to knowledge repositories and their implementation and maintenance Data storage options and their implications Health information exchanges Clinical registries Patient matching strategies Master patient index Data reconciliation   Interactive session: Discussion on specific security topics related to UCIMC and discussion/presentation of individual cases on large scale security breach as described above under week 3 preparation.
12	AI, Medical Intelligence, and Research Informatics
April 29–May 26	Board Content Core Areas: 1, 2	Milestones: PC 1–2, MK2, SBP3, PBLI 3–4
Syllabus	Objectives
Week 1	Future of computer applications of biomedicine Definitions and appropriate use of descriptive, diagnostic, predictive, and prescriptive analytics Predictive analytic techniques, indications, and limitations
Week 2	Analytics tools and techniques AI ML
Week 3	NLP, Computer Perception (vision), Neural Networks Assignment—Read the NYT article “A.I. Is Mastering Language. Should We Trust What It Says” and come prepared to discuss if you agree with Professor Gary Marcus' comments in the article or not and why. Advanced modeling techniques
Week 4	Data visualization Precision medicine Knowledge management and archiving science Metadata and data dictionaries   Interactive session First-year fellow computer vision assignment—Create a classifier for retinal images using Microsoft Lobe AI Second-year fellow assignment: Because you completed the computer vision assignment in year 1, you will need to find some visual data that you can use to create an algorithm to detect something (doesn't what, anything) chest X-ray, MRI, derm photos, etc.
13	Legal, Ethical, Regulatory
May 27–June 23	Board Content Core Areas: 1, 2, 2	Milestones: PC 1–2, MK2, SBP3, PBLI 4
Syllabus	Objectives
Week 1	Ethics and Health Informatics   Interactive session: Senior to discuss how ethical considerations played into a current or past CI-related project
Week 2	Protecting Patient Health Information Stewardship of data Regulations, organizations, and best practices related to data access and sharing agreements, data use, privacy, security, and portability Reporting requirements   Interactive session Senior fellow to present a legal case where an institution was found to be in violation of regulations surrounding patient-generated data and discuss what measures the institution could have taken to comply with said regulation. We will review relevant board-style questions and be prepared to discuss the answers and distractors
Week 3	Quality Methods to measure and report organizational performance Clinical quality standards Uses of patient-generated data   Interactive session: Review the UCI Institutional Scorecard from March 2023. Critically analyze the measures in the context of the framework discussed in the lecture.
Week 4	Health Data Uses and Quality of Data Quality standards and measures promulgated by quality organizations. Facility accreditation quality and safety standards   Interactive session: Senior fellow will use the NQF website ( http://www.qualityforum.org/About_NQF/ ) to identify four specific performance measures that are endorsed by NQF for physician practices.  • How is each measure calculated, including the source of the data, the numerator, and the denominator?  • Do you think these measures are a good reflection of quality practice? Why or why not?  • Use Slicer Dicer to try to report on these measures.

Abbreviation: UCI, UC Irvine.

Conflict of Interest

None declared.

Protection of Human and Animal Subjects

This project was designated as exempt from needing IRB approval.

• References

• 1 Kim E, Van Cain M, Hron J. Survey of clinical informatics fellows graduating 2016-2024: experiences before and during fellowship. J Am Med Inform Assoc 2023; 30 (10) 1608-1613 Erratum in: J Am Med Inform Assoc. 2023 Nov 17;30(12):2100. doi: 10.1093/jamia/ocad192.
• 2 McClintock DS, Levy BP, Lane WJ. et al. A core curriculum for clinical fellowship training in pathology informatics. J Pathol Inform 2012; 3: 31
• 3 Lingham V, Chandwarkar A, Miller M. et al. A systematic approach to the design and implementation of Clinical Informatics Fellowship programs. Appl Clin Inform 2023; 14 (05) 951-960
• 4 Whitfill JT, Kalpas E, Garcia-Filion P. Reuniting long lost cousins: a novel curriculum in imaging informatics for clinical informatics fellows. J Digit Imaging 2022; 35 (04) 876-880
• 5 Patel TN, Chaise AJ, Hanna JJ. et al. Structure and funding of Clinical Informatics Fellowships: a national survey of program directors. Appl Clin Inform 2024; 15 (01) 155-163
• 6 Frenk J, Chen LC, Chandran L. et al. Challenges and opportunities for educating health professionals after the COVID-19 pandemic. Lancet 2022; 400 (10362): 1539-1556
• 7 Clinical Informatics Content Outline. American Board of Preventive Medicine. Updated 2024. Accessed February 28, 2024 at: https://www.theabpm.org/become-certified/exam-content/clinical-informatics-content-outline/
• 8 Silverman H, Lehmann CU, Munger B. Milestones: critical elements in Clinical Informatics Fellowship programs. Appl Clin Inform 2016; 7 (01) 177-190
• 9 Silverman HD, Steen EB, Carpenito JN, Ondrula CJ, Williamson JJ, Fridsma DB. Domains, tasks, and knowledge for clinical informatics subspecialty practice: results of a practice analysis. J Am Med Inform Assoc 2019; 26 (07) 586-593
• 10 Mankowitz SJ. Clinical Informatics Board Review and Self Assessment. 1st ed.. Switzerland: Springer; 2018
• 11 Finnell JT, Dixon BE. Clinical Informatics Study Guide: Text and Review. Springer; 2022
• 12 Shortliffe EH, Cimino JJ, Chiang MF. Biomedical Informatics: Computer Applications in Health Care and Biomedicine. Switzerland: Springer; 2021
• 13 ACGME Common Program Requirements for CI Fellowship, page 33. Accessed August 22, 2024 at: https://www.acgme.org/globalassets/pfassets/programrequirements/cprfellowship_2023.pdf

Publication History

Received: 02 April 2024

Accepted: 19 August 2024

Accepted Manuscript online:
20 August 2024

Article published online:
20 November 2024