Keywords
Curriculum - professional education - informatics - information science - systems
theory
1. Introduction: World and Role
1. Introduction: World and Role
In the world of informatics education, given that informatics is about structured
knowledge, an informatics curriculum should exhibit informatics concepts in a structured
form, and not just in content. For a competency-based program in an educational organization
[[1]], an informatics training program should not only convey knowledge, but should impart
practical skills. We developed the Informatics Stack to serve as the basis of a core
skill that students in an introductory informatics course could apply already during
the course. Because this skill is executed at the informatics bedside, rather than
as a result of intense analysis, and because we do not claim it as a rigorous discipline
of software engineering [[2]], we call it a “heuristic,” or rule of thumb [[3]]. The purpose of the Informatics Stack is to provide a heuristic tool to aids in the role of informatics practice that embodies a systems
perspective at the health-IT “bedside” and that helps in the informatician’s task
of translating between stakeholders and technical communities [[4]]. In the IMIA framework, we are targeting novice biomedical health informatics professionals
in bachelor, master, or doctoral training [[5]].
As illustration of the tasks and workflow in using the Stack, we offer these example
Use Cases. The first 2 are applicable prior to the start of a project, the third,
in the course of a project and the fourth, after a project’s completion.
-
Readiness assessment. A domain stakeholder asks you to solve a need through health IT services. Is the
request well formulated?
-
IT enthusiast. An IT enthusiast asks you to build or purchase new technology. Is there a need that
matches that technology?
-
Formative evaluation. You are brought into an ongoing project and asked to comment on the effort. Is the
project at risk?
-
Report review. You are reading a report about an information system and want to provide a critical
review. Is the report complete? Do the elements of the project cohere?
In the current article, we lay out the contents and use of the Stack, illustrating
specifics with details from an online course, Introduction to Biomedical and Public
Health Informatics (ME 600.903) taught at Johns Hopkins for the past 6 years, which
experience also provides a “verification” of the Stack (to use Friedman’s and Wyatt’s
term for a subjectivist evaluation [[6]]).
2. Method: The Informatics Stack
2. Method: The Informatics Stack
The notion of a “Stack” is the metaphor of the OSI stack – itself, standard content
in informatics education – where one level communicates to other levels only through
intermediary levels [[7]]. The Informatics Stack is more like a Russian doll: higher levels encompass lower levels. So one reads ► [Figure 1] as follows: Technology supports the representation and use of Data, Information, and Knowledge (acted on by Algorithms);
these activities are encapsulated in Modules which work together, in a system-of-systems as an Information System that supports Workflow, and with which human agents interact and who exhibit a range of Behaviors as a result of their level of technology Adoption. That workflow
supports the Functions and Goals of a variety of participants playing their Roles, who participate in an Organization that functions in a particular World. The headings of the current article take their language from
the Stack.
Figure 1 The Informatics Stack. The levels are defined in the text: lower levels support or
are encompassed by higher levels. The bidirectional arrows indicate the specific focus
of interchange or interoperability at each level. The domains of privacy, confidentiality,
and security are indicated. Evaluation, Ethics, Law, Research have different identities
at each level (not shown). The horizontal Line separates the what (Above) from the how (Below). Some Stack names are used as headers in the current article.
We here provide definitions. Example educational objectives, used in the Introductory
course, are presented in ► Online Appendix 1.
World
The context (political, economic, policy) within which the organization operates.
Organization
The local context. May be a formal organization (e.g., health system) or informal
(a family). For an organization that is part of another organization, the larger organization
may be the “World” for the first organization.
Perspective/Role
A role may be filled by many kinds of people (e.g., physicians and nurse practitioners
take on the role of care providers) while a single person may have multiple roles
(e.g., a doctor is a care provider, a teacher, a learner, an administrator, and a
researcher [[8]].
Goals/Functions
The goal is what needs to be accomplished (by role, to maintain an individual’s position within or simply to support the organization).
They may decompose into objectives, as in the Logical Framework [[9]]. The functions are how the goal will be achieved. There may be many functions for a single goal.
Workflow/Behavior/Adoption
This level encompasses the work and behavior performed by a role (almost always in
conjunction with other roles), and includes concerns like human-computer interaction,
and adoption of technologies and new workflows.
Information System
The information ecology that supports the Workflow: how does the system appear to
the user? May be explicitly a single system or may be, in reality, an interconnected
system of systems. Electronic Health Record (EHR) Systems constitute the classic information
system in clinical care, but so might a personal, wearable network, for consumer informatics,
whose boundary may end at the smart phone or may extend into the patient portal, on
the health system’s side.
Modules
Software (or other) entities that have a single information function. Modules may
be information systems of their own. For instance, to the pediatrician, an Immunization
Information System (IIS) is a module, supporting the doctor’s need for information
about a child’s vaccine status; the IIS itself has multiple components. To the public
health agency tasked with maximizing immunization rates, the IIS may be the Information System. Thus, an Information System (higher level) is and acts as
a system of systems [[10]].
Data, Information, Knowledge, Wisdom (DIKW), Algorithms
Computer scientists separate data structures from algorithms, so we follow their lead
here, and place DIKW on one side and Algorithms on the other, at this same level [[11], [12]]. The ways that data are organized, presented, or aggregated result in information.
A representation of a patient’s data within the context of the patient’s presentation,
e.g., as a JSON object (machine-oriented) or as a trend-line graph (user-oriented),
is information. The rules by which that information is used for action constitutes
knowledge. The understanding of when (and when not) to follow the machine’s rules
is wisdom. The algorithms the machine uses to collect and store data, to create or
display information from data, or to learn or apply knowledge to the information comprise
the Algorithms.
Technology
Comprises the hardware, software, networking, and the like, through which the higher
levels of the Stack are implemented. (E.g., the laptops, smartphones, used by students,
and the servers used by the LMS.)
2.1 Above the Line and Below the Line
The Stack separates into upper and lower parts: Information System and up are the
upper; Information System and down are the lower. We call the boundary between them
(within the Information System level), the Line: Above the Line, the concerns are
what will the information system accomplish; Below the Line, how it will accomplish them. The Line represents the task of translating for others,
across the Line [[4]]. Classifying verbalized concerns by stakeholders as Above or Below the Line is
an important competency in using the Stack in practice.
2.2 Use Cases, Revisited
Let us apply the Informatics Stack to our Use cases.
-
Readiness assessment. The informatician begins by understanding the context (World and Organization) of
the stakeholder, then, ideally, works down the Stack to determine the Above the Line
needs, in the language of the stakeholder; to determine the degree of health IT/informatics
sophistication held by the stakeholder, to gauge the level of “translation” required
below the Line; to monitor proposed designs for inconsistencies between the Below
the Line IT service proposed and the Above the Line needs; and to figure out the real
reason behind the request, in the case of inconsistencies.
-
IT enthusiast. The informatician’s goal is to figure out what is problem that the enthusiast thinks
she is solving with the new technology, essentially working up the Stack.
-
Formative evaluation. The informatician is given both an Above-the-Line and a Below-the-Line reality,
and the task is to assess their across-the-Line consistency, taking the stage of the
project into account.
-
Report review. This use case in practice is similar to formative evaluation, except that there
are no stakeholders to interview and functions more like a summative evaluation. This
use case serves as the basis of the students’ Final Project. The Stack can also be
used to structure a report comparing As–Is to To–Be solutions.
2.3 Vertical Issues of the Stack
There are “vertical” issues common across levels, although there may be specifics
at any one level. Six such issues are standards, privacy/confidentiality/security,
ethics, law, evaluation, and research. Current paradigms for informatics evaluation
and research focus either on stage of development (e.g., conceptual, lab, field, practice
[[13]]) or research method, divided into quantitative and qualitative methods [[6]]. The Table in ► Online Appendix 2 shows how those research and evaluation methods
should be targeted to their position in the Stack. The table links scientific disciplines
to each level; our list is informed by Shortliffe and Cimino [[14]] and by the IMIA educational framework [[5]] but we go the extra step of showing how and where the links are made.
2.4 Use in the Course
In our graduate course, ME 600.903 Introduction to Biomedical and Public Health Informatics,
required of informatics masters students and elective for masters of public health,
of information security, and of business administration, we structure the 8 weeks
of topics according to the Stack, starting at the bottom of the Stack, after an Introduction.
The summative exercise is a Final Project, where each group of 4–6 students reviews
a system, generally a Davies award winner from the previous 5 years [[15]]. Davies awards are awarded by the Health Information Management Systems Society,
based on extensive reports submitted by the candidate institutions, and so should
reflect the best on those candidates. The students complete a structured review of
that report in a group-shared wiki. The items for review comprise the Stack (see ►
Online Appendix 2). Our review of their answers comprises qualitative evaluation of
the Stack’s completeness and educational use.
The projects are scored (by the course director, HL) based on completeness, correctness,
quality, and, most importantly, consistency across the levels. Finally, students are
asked to reflect on what they saw in the report that they would not have seen without
the Stack framework, and are asked what aspects they saw in the report that were not
addressed by the Stack. Specifics of the Stack and the Project were changed over the
years in response to these reflections. In particular, detailed instructions were
provided, to overcome ambiguous earlier instructions and to reinforce that content
of their report may change as they explore the target report in more detail; the assignment
was pared down to 2 perspectives (preferably, one clinically related, the other, public
health); an abstract was required, to help the students to summarize their analysis;
clarifications were added on the differences between outputs and outcomes of an information
system; the required method of describing workflow was limited to using swimlane diagrams;
a section was added, highlighting information systems as systems of systems, with
modules being systems in their own right; sections were added on testing and evaluation
(and the differences between the two), interoperability, security, and ethics.
3. Results, Evaluation: Students’ Experience with the Stack
3. Results, Evaluation: Students’ Experience with the Stack
► [Table 1] shows the numbers of students and groups participating in the course. From 2013
and on, enrollment was a mix of graduate informatics, public health, clinical, computer
science and business students.
Table 1
Enrollment and number of Final Project Group.
|
Year
|
N Students
|
N Groups
|
|
2010[*]
|
94
|
26
|
|
2011[*]
|
103
|
22
|
|
2012[*]
|
126
|
23
|
|
2013
|
59
|
11
|
|
2014
|
60
|
13
|
|
2015
|
62
|
13
|
|
2016
|
34
|
8
|
|
Total
|
538
|
116
|
* Training subsidized by workforce-development grant from the US Office of the National
Coordinator for Health I T.
The reflections after 2013 were reviewed in detail, when the Stack details and the
Final Project requirements were stable. (The Johns Hopkins Institutional Review Board
judged this review as exempt from IRB review.) Of the surprises, every level of the Stack was identified by some student each year as something they
might have missed. At times, two different students in the same group identified complementary
levels (e.g., Below and Above the Line). Many described insight at how they previously
conflated issues at different levels. (E.g., “I would mix up Goals/Functions with
Workflow” [2014] or, “Without the frameworks I found it easy to be overwhelmed by
the organizational ‘C-suite speak’.” [2014] Many students appreciated the systematic
or structured examination. It “provided coherence as I went through the details of
every layer of the stack.” [2014] Several reported difficulty in separating their
analysis into the levels – and appreciated the benefit of having done so in the Project.
Only one students reported, “I am not truly certain that the framework added to my
comprehension.”
Of the issues not addressed, these fell into 3 components: First were issues that
are not included in the Stack. Primarily, that the Stack does not account for temporal issues: the “nuts and bolts” of development or implementation (over time), “continuous
process improvement” (and feedback), “sustainability,” “change management.” Other
issues included “likelihood of success,” “dynamic interactions between the consumers
of the system,” “community of users,” “scaling up,” and “cost.”
The second set of criticisms were issues the students said were not addressed, but
actually were. Their raising these concerns pointed to possible weaknesses in the
course (e.g., needing to stress certain points more), and not to the Stack itself.
Thus, some students pointed to “adoption” and “human factors” as not addressed. Another,
that interoperability was not addressed, despite an entire module devoted to standards;
the layering of standards into levels shown in ► [Figure 1] was the response to that criticism. Yet another, that the “whole is often greater
than the sum of its parts,” which is exactly the point of systems thinking embodied
by the course.
Third were issues addressed in the Stack and in the course, but that the particular
student missed or was missed in the particular report being critiqued. Fourth were
observations, such as that, unlike the OSI stack, what happens at one levels of the
Informatics Stack does affect choices made at all other levels. [2015] For instance,
the nursing role affects which data are needed in a documentation module differently
from a hospitalist–physician role; in the OSI stack, a packet of data at the datalink
level is handled the same, whether the packet represents an email message or a Web
page request.
Some students already applied the Stack outside the course: “I am using it to analyze
another project, and it just threw light on me on what should be done right now and
next.” [Computer science student]
Informally, alumni in both clinical and public health contexts have reported using
the Stack in their work in exactly the “bedside” manner it was intended. Others have
used it as a basis of consulting work, e.g., in creating a readiness-for-HIT checklist
for a hospital.
4. Discussion
We submit the Stack as a heuristic for students to gain an initial purchase on how
to tackle the complexity of an informatics environment in many informatics subdisciplines
(e.g., clinical, public health). The Stack incorporates many concerns from other informatics
curricula and informatics desiderata. We have demonstrated that an 8-week online course
is able to transmit the core knowledge of the Stack and to have them demonstrate their
ability to use it in the heuristic way it is intended.
We are not the first to suggest that informatics includes the levels described here.
For instance [and we provide approximate Stack equivalents], the US Government’s Federal
Enterprise Architecture includes the levels Performance Reference Model (PRM), Business
Reference Model [Organization], Data Reference Model [DIKW], Application Reference
Model [Modules], and Infrastructure Reference Model [Information Systems/Technology]
[[2]]. The PRM addresses the temporal components not addressed by the Stack.
The more recent mHealth Assessment and Planning for Scale (MAPS) Toolkit [[16]] involves “Axes” of Groundwork [World], Partnerships [Organization], Financial Health
[Organization], Technology & Architecture [Technology, DIKWA], Operations [Information
System], and Monitoring & Evaluation [Evaluation]. While the Axes and the Stack levels
appear comparable, the Toolkit is geared for the processes of deployment and system
evolution.
The notion of separating what is to be accomplished from how it is to be accomplished is an essential component of project management [[17]], often itself a component of informatics curricula, and other areas, such as statistical
analysis. But the distinction is not stressed in informatics education.
The Stack integrates information from many informatics textbooks currently available.
Reference has already been made to Shortliffe and Cimino [[14]], with its set of broad foundational chapters. A core diagram shows the many sciences
that contribute to informatics, but does not articulate how they do so, as in ► [Table 1]. The Hoyt and Yoshihari text is more practically oriented, with most chapters focused
on a variety of clinical systems, resources, or functions (such as disease management),
but without an organizing framework [[18]]. Enrico Coeria, who himself has written about complex adaptive systems and the
“wicked problem” of healthcare [[19], [20]], published a textbook that is focused on the clinical perspective [[21]]. The recent Health Informatics: A Systems Perspective, means, by the term “system,” the healthcare organization, not the system of systems
that is the focus of the Stack [[22]]. Another text takes the socio-technical perspective, one organizing principle of
the Stack [[23]]. The most recent text on public health informatics includes notions of “context”
and “science” of the field: government/legislation, information architecture, data
sources, data tools, standards, privacy, EHRs, ethics, project management [[24]].
The article that defines the Core Contents for the US Clinical Informatics Sub-specialty
includes the high-level outline, Fundamentals (which includes Ethics and Legal/regulatory
issues), the Functions of Clinical Decision Making and Care Process Improvement, then
Health Information Systems (which includes our Technology and DIKW levels), and Leading
and Managing Change (the temporal components) [[25]]. The new competencies for Masters Degree CAHIIM accreditation in the United States
employ the “foundations” of Health; Social and Behavioral Science; Information Science
and Technology; and their intersections [[26]]. The 3 foundations correspond to the top, middle, and bottom levels of the Stack,
respectively.
The limitations to the Stack were laid out by the students; most of them have to do
with time and with the actual realities of building a system. We judged these concerns
as having more to do with the dynamic process of software development, rather than
the more static (single-time point) notion of global or formative evaluation or assessment
intended here. Our curriculum addresses these concerns in two courses that focus on
software development and deployment, ME 600.900 From Design to Deployment and ME 600.902
Leading Change Through Health IT. Further criticism is that it is broad with little
depth. But such a trade off is appropriate for an introductory course. Finally, while
the Stack is taught as focused on clinical and public health contexts, it applies
to bioinformatics as well, if biologists (rather than subcellular components!) are
the true focus. Similarly for “consumer informatics” or “precision medicine informatics,”
and the patient.
5. Conclusion
The Informatics Stack is a compact method of conveying informatics principles and
skills, and embodies informatics thinking by its very nature. It is a broadly-encom
-passing heuristic whose application can be learned and applied by students from a
wide variety of backgrounds in an 8-week course.
