Optimizing the User Experience: Identifying Opportunities to Improve Use of an Inpatient Portal

Abstract

Background Patient portals specifically designed for the inpatient setting have significant potential to improve patient care. However, little is known about how the users of this technology, the patients, may interact with the inpatient portals. As a result, hospitals have limited ability to design approaches that support patient use of the portal.

Objectives This study aims to evaluate the user experience associated with an inpatient portal.

Methods We used a Think-Aloud protocol to study user interactions with a commercially available inpatient portal—MyChart Bedside (MCB). Study participants included 19 English-speaking adults over the age of 18 years. In one-on-one sessions, participants narrated their experience using the MCB application and completing eight specific tasks. Recordings were transcribed and coded into three dimensions of the user experience: physical, cognitive, and sociobehavioral.

Results Our analysis of the physical experience highlighted the navigational errors and technical challenges associated with the use of MCB. We also found that issues associated with the cognitive experience included comprehension problems that spurred anxiety and uncertainty. Analysis of the sociobehavioral experience suggested that users have different learning styles and preferences for learning including self-guided, handouts, and in-person training.

Conclusion Inpatient portals may be an effective tool to improve the patient experience in the hospital. Moreover, making this technology available to inpatients may help to foster ongoing use of technology across the care continuum. However, deriving the benefits from the technology requires appropriate support. We identified multiple opportunities for hospital management to intervene. In particular, teaching patients to use the application by making a variety of instructional materials available could help to reduce several identified barriers to use. Additionally, hospitals should be prepared to manage patient anxiety and increased questioning arising from the availability of information in the inpatient portal application.

Keywords

user experience - patient portal - patient education - usability

Background and Significance

Patient portals designed for the inpatient environment are becoming part of the ecosystem of health information technology (HIT) that supports patient care.[1] [2] This technological tool is a type of personal health record (PHR) designed to give patients access to elements of their provider's electronic health record (EHR). Portals typically allow for such functionality as checking laboratory results, scheduling appointments, paying bills, sending secure messages to providers, and providing access to patient education materials.[3] Portals can thus provide patients with important information regarding their care plan, and can facilitate access to patient education and communication with the care team regarding health care questions. As a result, patient portals are increasingly recognized as a tool that can potentially increase patient engagement by empowering patients to be partners in their health care.[4] [5] [6] [7]

Patient portal use in the ambulatory environment has been positively received,[3] [8] leading organizations and researchers to begin to focus on the use of patient portals in the inpatient environment.[9] [10] [11] Research on inpatient portals is in its infancy, but the benefits of these tools appear similar to those reported in the outpatient setting: improved safety and quality, increased satisfaction and patient engagement, decreased anxiety, and better communication with the care team.[10] [11] [12] [13] [14] However, research in this area has largely focused on outcome measures, and has given less consideration to aspects of human–computer interaction that are essential to achieving such outcomes.

In practice, a user's experience with an inpatient portal largely dictates the technology's ultimate utility. User experience pertains to the perceptions and reactions of use or anticipated use of a given technology.[15] Understanding user experience can contribute to refinement of the design and implementation of patient-facing technologies.[16] [17] [18] [19] [20] For instance, O'Leary et al investigated patient and provider perspectives regarding use of an inpatient portal.[19] While that study noted some of the technical challenges and opportunities to improve the design of the technology, it did not delve specifically into the user experience for patients.

Objective

This study aims to articulate perspectives of users regarding their interaction with an inpatient portal. As patients are the end user of this technology, focusing on the user experience can shed light for hospital management and the care team with respect to how they can best support patients in their use of inpatient portals. This issue is critical for hospitals attempting to employ patient-focused technology to enhance patient experience. Moreover, as patients utilize technology more and more to engage with their health care across the care continuum, understanding how patients perceive and utilize portals is essential to optimize portal impact. Findings from this study can therefore support efforts in both inpatient and outpatient settings that seek to improve user experience of the technology.

Methods

Study Setting and Recruitment

This study was conducted at a large Midwestern academic medical center (AMC) that offers comprehensive care across the continuum at six hospitals and 53 ambulatory site locations, including 30 community-based clinics. Volunteers were recruited to participate in the study through the AMC's Patient and Family Experience Program (PFEP). The PFEP includes groups of current and previous patients who volunteer to provide their perspectives about ways to improve the care experience at the AMC and are offered the opportunity to participate in research. Study inclusion criteria consisted of English speaking, over 18 years of age, no cognitive impairments, and no previous experience with the AMC's inpatient portal. We purposely selected individuals who were not currently inpatients but who could critically provide feedback about core user experience issues prior to studying a hospitalized patient population with high variability in health status that could confound understanding of technology-related issues. Appointments to assess users' perspectives about portal usability and their experience using the technology were scheduled with interested participants. All sessions took place between May and September 2016. Institutional review board (IRB) approval was received for all study activities.

Data Collection

User experience sessions offered patients access to a commercially available inpatient portal—Epic's MyChart Bedside (MCB) application. MCB is a password-protected application offered on an Android tablet. For the study sessions, we provided patients with a tablet equipped with MCB in a format identical to that being implemented in the inpatient setting throughout the AMC. MCB allows patients access to the following 11 pages: Home; Bedside Tutorial; Happening Soon; To Learn; Taking Care of Me; Messages; My Health, Dining on Demand; Notes; I Would Like; MyChart. A description of the functions offered on each page is provided in [Table 1].

Table 1 Pages and their functionality available in MyChart Bedside application

MyChart Bedside page	Function
Home	View reason for admission, current medical conditions, and current medications
Bedside tutorial	Watch 11-minute introduction video explaining the specific features available on MyChart Bedside
Happening soon	See schedule of previous and upcoming medication administration, procedures, and visitations
To learn	Access patient education materials and videos
Taking care of me	See descriptions and photos of members of the care team
Messages	Send secure messages to the care team
My health	View laboratory results and track vitals
Dining on demand	Order food directly from the hospital cafeteria (disabled in test environment)
Notes	Create personal written, audio, or video notes
I would like	Send request for patient services, including patient education, physical therapy, pastoral care, social work, discharge planning, pharmacy, or the gift shop
MyChart	Log-in or Create MyChart account available in the ambulatory environment

User experience sessions lasted approximately 60 minutes and were conducted in a quiet and private office. One volunteer was scheduled per session, and each session was moderated by an experienced investigator with the support of a research assistant. All sessions were audio and visually recorded using a digital camera so that we were able to capture participants' feedback and comments on specific features or screens. At the end of the session, participants received a $30 gift card in appreciation for their time.

Upon arrival at the session, study participants were provided with a brief survey asking questions about demographics (age, gender, and education level) and about their comfort with tablet technology and PHRs on a scale of 1 to 10 (1 = Very Comfortable; 10 = Not at all Comfortable). Scores were categorized into high (1–3)/medium (4–6)/low (7–10) levels.

Next, to evaluate user experience, we employed a Think-Aloud protocol. The Think-Aloud protocol is a commonly used approach to evaluate technology user experience, a component of the overall usability of a technology.[10] [21] [22] This approach was derived from research by cognitive psychologists and is typically deployed to evaluate the usability of a technology application.[23] The protocol encourages users to express out loud what they are looking at, thinking, doing, and feeling, as they navigate an application and perform specified tasks. Through this process, investigators are able to distill out the experience of using a given technology. Thus, using a sample of the intended end users as participants in a Think-Aloud protocol can provide insight into how users interact with the system. Using the Think-Aloud protocol, a sample size of 15 participants can be sufficient to reach saturation in identifying usability issues.[24]

We operationalized the Think-Aloud protocol by providing study participants access to a test patient that had been created on the MCB application. Participants were given time to freely explore the portal, and were instructed to describe their experience as they navigated the tablet and the technology. The investigators often elicited information to further probe participant perspectives on their experience using the tool. If the participant became stuck or lost in the MCB environment at any point, the investigators provided hints to help them navigate a task as necessary. If a participant continued to struggle, specific instructions were provided and investigators made note of the participant's difficulties.

After an initial exploration period, study participants were asked to complete eight tasks to closely imitate actual user experience. Participants continued to talk through their experience using the technology as they worked through the tasks. The tasks included: (1) view your schedule; (2) check your most recent test results; (3) view your health care providers; (4) communicate with your care team; (5) write a note to yourself; (6) request a patient education session; (7) read the diabetes education documents; and (8) add an event to your schedule.

Throughout the sessions, we also gathered feedback regarding MCB functionality, features, processes, the user interface, user–system interactions, as well as information about difficulties encountered during the session and participants' perceptions about interacting with the MCB app. Finally, investigators noted when users made operational errors caused by misinterpreting functionality, or made navigational errors on the tablet or while navigating the app interface.

Data Analysis

In line with the intent of our study to evaluate the user experience associated with using MCB, we focused our analysis on the content of the audio recordings associated with the Think-Aloud sessions. Results of a formal usability analysis of the MCB application studying user errors and system design issues is available elsewhere.[25] Audio recordings of sessions were transcribed verbatim, then analyzed using both inductive and deductive methods.

We coded and analyzed session transcripts using a coding dictionary and held frequent discussions among investigators. The coding dictionary was developed through an iterative process involving two experienced investigators (D.M.W. and A.S.M.) jointly coding four transcripts, then discussing findings about codes and emergent themes. Remaining sessions were independently coded by two coders (D.M.W. and T.M.), with coding overseen by an experienced investigator (A.S.M.). Any coding discrepancies were resolved through team discussions held throughout the coding process.

Given our interest in user experience with the technology, we applied a coding schema that draws from the Systems Engineering Initiative for Patient Safety (SEIPS) model.[26] SEIPS describes the work system, work processes, and resultant outcomes, and has been widely used to evaluate clinical and workflow changes, including the implementation and use of HIT.[26] [27] [28] In the SEIPS model, work processes are categorized by their physical, cognitive, and sociobehavioral dimensions. Given our focus on the process of using the inpatient portal, we developed codes based on these three aspects of user perspectives. The physical aspects related to the use of the inpatient portal included mechanistic actions and ergonomic issues. Cognitive dimensions encompassed the comprehension of information contained in the tool, as well as the user's emotional experience of using the technology, as this serves to reinforce technology use.[29] The sociobehavioral dimension was defined to capture the context within which the patient portal was being used. We also applied a fourth code for an emergent theme about suggestions to improve the experience of using the inpatient portal.

Results

Study Participants

A total of 19 volunteers participated in the study. Participant demographic information is presented in [Table 2]. The largest percentage of our sample was over 60 years old (47%), and was female (57%). Most participants used a computer daily, owned a tablet, and reported high levels of comfort with both tablets and PHRs.

Table 2 Study participant demographics

Characteristics	N (%) N = 19
Age
< 30	5 (26)
30–60	5 (26)
> 60	9 (47)
Gender
Female	11 (57)
Male	8 (42)
Education
Some college but no degree	5 (26)
Technical degree or certification	1 (5)
Bachelor	4 (21)
Master	5 (26)
Doctoral	2 (10)
Professional degree	1 (5)
I prefer not to answer	1 (5)
Computer use
Daily	18 (95)
A few times/month	1 (5)
Own a tablet
Yes	16 (84)
No	3 (16)
Tablet comfort level[1]
Low	3 (17)
Medium	2 (12)
High	12 (70)
Personal health record comfort level[a]
Low	2 (12)
Medium	2 (12)
High	13 (76)

^a Only 17 study participants responded to the questions about tablet and personal health record comfort levels. The reported percentages are adjusted for the sample size change.

User Experience

Our results are organized according to the three different dimensions of user experience—physical, cognitive, and sociobehavioral, with suggestions for improvement of user experience following as a fourth section.

Physical Experience

The most prominent physical issue associated with using MCB related to the physical layout of the application. Several users experienced navigational challenges as a result of the application layout. For example, one user noted, “I got a page that I don't know how I got it. And I don't know how to get out of it.” Such issues prompted additional physical work efforts by users: “I had to go back a few because I didn't know what button to push. So I do like most people do, just start pushing buttons.” Notably, issues related to navigating the application prevented users from finding the information that they were seeking in the application. As one user described this experience when looking at the Home screen, “I can't see any place where I obviously see test results. So what I did is I Would Like and…I would have expected a link to say test results...I went to I Would Like [for] help with finding my test results.”

Beyond the navigational challenges associated with MCB, users noted issues related to the physical experience of using the tablet itself. In a detailed narration of their experience using the application, one user described: “How to check your blood sugar? Let's see if I can get one of those patient education...No, that's not what I wanted. Back to the health encyclopedia. I don't know how that happened! ...Those are too small for my finger, that's why I still got the wrong one. I hit the one below it instead of the one that I am trying to get.” Another user questioned, “Am I not pressing hard enough? This is confusing because there are arrows there indicating forward or back. I want to go back and nothing is happening.” Other users also noted potential ergonomic barriers to use of the tablet, such as not being able to “hold” the tablet. These physical barriers may discourage use of MCB, as they can elicit negative attitudes and emotions, including anxiety and frustration. For instance, after one study participant had accidentally pushed the wrong button and had been logged out, they stated, “That's frustrating. I would stop using it. I don't like starting over with stuff like that. It feels broken.”

Cognitive Experience

Study participants noted several issues that were categorized as related to cognitive aspects of using the technology. First, users noted comprehension issues related to the information displayed in MCB. Study participants attempted to draw on their prior experiences with technology and similar tools as reference points to understand the information, but still expressed confusion. As one volunteer explained when searching for laboratory results:

That looks like a laboratory report, which I know because I have a science background and I've seen enough of these things, but I'm wondering how meaningful this is to somebody who doesn't know. I know when I get my laboratory reports from my doctor, it has my results and then it has the range that's considered normal and then if mine is abnormal, it comes out red or a different color. And I think that would be helpful...I would like to know whether I'm in the normal range for laboratory tests.

The issue of understanding the laboratory results information was persistent across study participants. As another participant succinctly expressed, “[T]he part about My Health,...I can't make any kind of educated opinion of what's going on there. It's just a bunch of numbers and graphs and I'm worried about where I'm supposed to be in that.”

Study participants also expressed a strong preference for personalized information that could be made available in MCB. One study participant stated this preference explicitly in regards to the information offered about a specific medication: “I want this page to be about me and my condition.” Others stated this preference for personalized information in relation to specific pages in MCB. For example, while exploring the My Health page, one participant noted, “I think that the information...about the medication is not extremely helpful. It's not personalized for me in any way. It's just kind of like an Internet research. So that's not very helpful.” This preference for personalization was particularly relevant and frequently voiced regarding the patient education materials available on the To Learn page. As one study participant stated when conducting the task asking them to review patient education materials, “but that's really just talking about diabetes and what it's all about. You know, kind of a generic textbook. It's not about me and what my patient education should be.”

Sociobehavioral Experience

The sociobehavioral issues related to patient MCB use capture the contextual experience of using the application with the support of the care team. As a result, analyzing comments related to the sociobehavioral theme frequently provided insight into how users might react as a result of the navigation (physical work) and comprehension (cognitive work) challenges of using the application. Study participants suggested that a primary reaction of users might be to ask questions: “If they don't understand it, it can create more questions than it can answer.” Even when participants felt they understood what they were viewing on the application, they still felt that the information might spark a large volume of questions: “[T]hat's nice you can track and actually see numbers improving. I have a feeling that this is going to raise a lot of questions from a patient. [For instance,] my blood pressure went down to this or up to this, what does that mean? My pulse rate was here before and now it is here...is it bad that it was this low at night?” Study participants were additionally concerned that the questions raised might place a substantial burden on the care team: “You're gonna have a lot of people asking questions about [test results]. I know I sure would...if my mother was in the hospital...they better have people that are real prepared to answer a lot of questions, because that will happen.”

In addition to concern about the questions that users might ask as a result of access to information in MCB, several study participants noted that the design of the application itself might promote anxiety that ultimately discourages use of MCB. This concern exhibited itself in two contrasting ways. First, it was noted that the certainty that MCB seems to provide around the timing of medication administration and scheduled procedures may make patients anxious about deviations from those schedules. For example, one study participant postulated, “Okay, I'm supposed to get my [medication] at 6 p.m. and here comes 6 p.m. and no [medication]; you're gonna get people hitting that bell like crazy [asking] 'Where's my pill?” Alternatively, other features in the application, such as the secure messaging function, appeared to create uncertainty: “I don't know how the response to those requests [happens]. Do they come personally, in-person, or do they come in a message, which I think is okay in certain circumstances.” In this case, if the application does not clearly indicate how responses will occur, it was noted that patients may circumvent the secure messaging system and use the call button, even for nonurgent issues.

Suggestions to Improve User Experience with Inpatient Portals

All study participants noted opportunities to improve the experience around inpatient portal use. A summary of the key issues identified in each dimension of the user experience as well as user-proposed solutions are presented in [Table 3].

Table 3 Summary of key issues identified in each dimension of user experience and solutions suggested by participants

Key issue	User-proposed solutions
Physical experience
Difficulty navigating MCB	• Provide instruction (tutorials, handouts, in-person) • Highlight and enlarge navigational features in the application • Use contrasting colors for scroll bars and menu items
Problems with the sensitivity of the tablet	• Consider using different tablets • Make the application available on a patient's own device
Ergonomic issues related to the tablet	• Provide a stand for the tablet, upon request • Offer a stylus, upon request
Cognitive experience
Comprehension of information in MCB	• Provide instruction (tutorials, handouts, in-person) • Clearly display normal ranges for reported laboratories and test results
Lack of personalized information in MCB	• Populate the information in the application with resources more specific to the individual patient
Sociobehavioral experience
Questions about information in MCB may burden the care team	• Provide instruction (tutorials, handouts, in-person) • Support in-person communication with the care team
Anxiety about information in MCB	• Increase communication from the care team regarding changes to the schedule • Provide instruction on how to understand information in the application, such as including built-in links to concise education about MCB content • Clearly display normal ranges for reported laboratories and test results

Abbreviation: MCB, MyChart Bedside.

An overarching suggestion expressed by several participants related to the support they thought they would need to best use MCB. While overall participants felt that MCB could be useful, they felt that the application was “complex,” contained “a lot of information,” and would require some time and resources to understand. Study participants did have access to a tutorial video in the MCB test environment, yet with an 11-minute run time, many felt that the information in the video was overwhelming: “I'm always impatient with tutorials.” The participants noted preferring that the information be made available in more concise and focused sections, such as by splitting the video into chapters: “I think that if you listed more of those things...in the tutorial, such as how do you type in, how do you get to the keyboard to type in requests, or type in comments.” In contrast to the self-guided learning offered by the tutorial, many participants stated a preference for printed materials, such as a “brochure,” that could easily be referenced. For example, one participant requested: “[A] little handout that would just say, ‘If you want to add an event, go here. If you want to do this, go here.’ Just highlights...that people could look at on one little sheet of paper. That would help me, I know that. Rather than fiddle around...”

As an alternative to enhance use of the inpatient portal, some study participants felt that in-person support would be useful to facilitate learning about MCB. As stated by one participant, “they should be shown how to use it… It's just the human touch. It's not gotta be long, it's not gotta be an hour just to go through some of these things. This is gonna get you to here, this gonna get you to here. I think that would be helpful.” It was felt that this human interaction could reduce some of the anxiety associated with learning how to use the application:

I'd want this to be as friendly as possible and as unintimidating as this could be. Whether that means somebody sit down with them and walk them through it and who shows them how to use it, give them that little cheat sheet. I really think a cheat sheet, it would be helpful. Short and sweet, not every single thing that's on here but just some basic things would be helpful.

The in-person training was suggested as a way to help lower barriers to use that might deter patients from using the tool. However, as the above quote demonstrates, different approaches (i.e., in-person or cheat sheets) to teaching the patients how to use MCB were differently perceived as valuable.

Discussion

Patient portals available in the inpatient setting may be an effective tool to support patients throughout their hospital stay. By offering access to education materials, laboratory results, a daily schedule, and the ability to message the care team, inpatient portals may be able to increase satisfaction, improve perceptions about care quality, and strengthen the relationship between the care team and the patient. The extent to which inpatient portals can achieve these goals is, however, dependent on the user experience interacting with the technology. Our study aimed to elaborate this user experience, and explored the physical, cognitive, and sociobehavioral dimensions of using an inpatient portal. Interestingly, our emergent findings regarding how patients can learn to use the tablet could address the issues raised in each of the three dimensions of user experience. Better instruction on how to use and interpret the information available in the application, for instance, could reduce navigational issues and alleviate comprehension concerns, as well as preempt the questions that patients may ask the care team.

Our findings about the sociobehavioral experience suggest three general approaches that can be used to teach patients how to use the portal: video tutorial, handouts, and in-person training. These different approaches accommodate individuals' learning habits and needs, though literature suggests that in-person training interventions are a particularly successful approach to increase patient access to and use of PHRs.[30] [31] Importantly, these three approaches are not mutually exclusively, but may in fact be complementary. These approaches will likely also be helpful to caregivers of family members that may similarly use the tool.[32] Thus, employing a variety of approaches to learning may be a constructive strategy to improve user experience with portal technology and address barriers to use related to patient education needs.[33]

Teaching patients how to use the application could also serve to address the cognitive work issue of patient comprehension of the material in MCB. Collins et al examined perspectives on inpatient portals from a variety of stakeholders, including patients, and found that patients view inpatient portals as supplementary to in-person communication.[34] In accordance with that work, teaching patients how to use the technology to support their care experience could not only focus on the mechanics of using the application, such as how to get around the application, but also on understanding the information in the application. For example, in-person training could show users how to interpret their laboratory results displayed in the application. This training could also outline to users' situations in which sending messages to clarify interpretations of laboratory results was most appropriate. This approach to educating patients would potentially serve the dual purpose of better informing patients about their health status and alleviating some of the anxiety that ready access to increased information available through the application may produce. However, additional research is needed to better understand how training can improve user experience associated with the use of technology during an inpatient stay.

From a patient-centered perspective, the finding that users may have a significant number of questions regarding the information available in the portal could be viewed as positive: these questions may facilitate efforts to help patients become both more informed and more engaged. However, similar to evidence from a study of patient access to physician notes in the ambulatory environment,[35] we also found evidence that the information available in the portal might elicit anxiety. Hospitals may be able to develop approaches to preemptively address this anxiety. For example, hospitals could warn patients that their medication administration schedule is subject to change, or that response times for messages may be variable. Ideally, hospitals would be able to utilize in-person instruction to communicate how to use the inpatient portal application, what the patient can expect from the hospital regarding the information displayed in the application, and how to use this information. In practice, the questions provoked by the information available in the application may in fact encourage dialogue between the care team and the patient, and help to create more informed patients. However, the extent to which the technology promotes communication is dependent on how the hospital implements and manages the technology, such as whether the care team is prepared with adequate training and clear goals are established regarding the use of MCB.[36] Alternatively, built-in training modules, improvements in user interface, and accommodating the appropriate level of health literacy in both the training and technology, can also be used to improve patient experience of the portal.[34]

Study Limitations

Our findings should be interpreted with respect to three key limitations. First, our study has limited generalizability. The use of a single application (MCB) was examined in this study. However, MCB is provided by Epic, whose EHR has the second largest market share in the United States and is used by over a fifth of U.S. acute care hospitals that have attested to Meaningful Use.[37] As a result of Epic's market share and position, our findings are likely widely relevant. Additionally, other vendors may look to imitate MCB, as it is one of the first commercially available inpatient portals.

Second, by focusing on the user perspective, our study did not incorporate the perspectives of other care team members or contextual factors associated with the implementation of the MCB. Care team members also require significant training with regard to how to use the portal and incorporate patient use of it into their practice.[36] The interaction between care team and patient use of MCB will likely influence user experience of the technology, and exploring this interaction is a ripe area for future research.

Third, frequent computer users were overrepresented in our study sample; as a result, challenges with using the technology may have been under reported, especially for those with comparatively less technology literacy. Future work might explore user experiences among individuals with lower technology capabilities. Similarly, future research should address how the user experience may differ by type of patient.

Conclusion

Patient portals show promise as a tool that can facilitate patient engagement and improve patients' experiences. With a detailed understanding of user experience associated with using these tools, hospitals can be better positioned to support and encourage patient use of the tool. In particular, hospitals may be able to offer multimodal approaches to teach patients how to use patient portals, including in-person training. This hands-on approach may be better able to create lasting engagement with the technology that continues as the patient transitions out of the hospital. Thus, the in-depth analysis of user experience can help to create management strategies that improve patient experience.

Clinical Relevance Statement

Based on our findings, hospitals seeking to implement an inpatient portal should consider, at a minimum, the ergonomic issues noted in our study that suggest that hospitals should opt for a tablet that is familiar to patients, lightweight, and easy to manipulate. Additionally, hospitals should act as a conduit between the end-user (e.g., the patient) and the vendor of the application in an effort to suggest upgrades and redesign improvements. Hospital managers should also consider their preparedness to respond to user concerns arising from the information provided in the application.

Multiple Choice Question

What approach(es) to learning do users prefer to familiarize themselves with the inpatient portal?

• Free exploration

• Video tutorial

• In-person training

• All of the above

Correct Answer: The correct answer is d, all of the above. Based on our study, different users expressed a preference for different approaches that would help them learn to use the inpatient portal. Some users preferred to freely explore the application, while others watched the video tutorial. While not offered in our study, some participants voiced a preference for in-person training. To accommodate these different preferences, hospitals implementing this technology may offer a blend of these approaches. This blended approach would most effectively address different user needs.

Conflict of Interest

None.

Acknowledgments

The authors are extremely grateful to the informants who participated in this study. We also thank our research team members who assisted at various stages of this project: Lindsey N. Sova, Jaclyn Volney, Barbara Barash, Jessica Garvey Smith, Michelle P. Zhou, Megan Chamberlain, Sharon Cross, and Pamela Thompson.

Protection of Human and Animal Subjects

Institutional Review Board approval was obtained from the Office of Responsible Research Practices at The Ohio State University to conduct this research.

• References

• 1 Ford EW, Hesse BW, Huerta TR. Personal health record use in the United States: forecasting future adoption levels. J Med Internet Res 2016; 18 (03) e73
• 2 Wald JS, Middleton B, Bloom A. , et al. A patient-controlled journal for an electronic medical record: issues and challenges. Stud Health Technol Inform 2004; 107 (Pt 2): 1166-1170
• 3 Ammenwerth E, Schnell-Inderst P, Hoerbst A. The impact of electronic patient portals on patient care: a systematic review of controlled trials. J Med Internet Res 2012; 14 (06) e162
• 4 Greysen SR, Magan Mendoza Y, Rosenthal J. , et al. Using tablet computers to increase patient engagement with electronic personal health records: protocol for a prospective, randomized interventional study. JMIR Res Protoc 2016; 5 (03) e176
• 5 O'Leary KJ, Lohman ME, Culver E, Killarney A, Randy Smith Jr G, Liebovitz DM. The effect of tablet computers with a mobile patient portal application on hospitalized patients' knowledge and activation. J Am Med Inform Assoc 2016; 23 (01) 159-165
• 6 Prey JE, Woollen J, Wilcox L. , et al. Patient engagement in the inpatient setting: a systematic review. J Am Med Inform Assoc 2014; 21 (04) 742-750
• 7 Rief JJ, Hamm ME, Zickmund SL. , et al. Using health information technology to foster engagement: patients' experiences with an active patient health record. Health Commun 2017; 32 (03) 310-319
• 8 Delbanco T, Walker J, Bell SK. , et al. Inviting patients to read their doctors' notes: a quasi-experimental study and a look ahead. Ann Intern Med 2012; 157 (07) 461-470
• 9 McAlearney AS, Sieck CJ, Hefner JL. , et al. High Touch and High Tech (HT2) proposal: transforming patient engagement throughout the continuum of care by engaging patients with portal technology at the bedside. JMIR Res Protoc 2016; 5 (04) e221
• 10 Peute LWP, Jaspers MWM. The significance of a usability evaluation of an emerging laboratory order entry system. Int J Med Inform 2007; 76 (2-3): 157-168
• 11 Sheehan B, Lucero RJ. Initial usability and feasibility evaluation of a personal health record-based self-management system for older adults. EGEMS (Wash DC) 2015; 3 (02) 1152
• 12 Ryu B, Kim S, Lee KH, Hwang H, Yoo S. Inpatient satisfaction and usage patterns of personalized smart bedside station system for patient-centered service at a tertiary university hospital. Int J Med Inform 2016; 95: 35-42
• 13 Walker DM, Sieck CJ, Menser T, Huerta TR, Scheck McAlearney A. Information technology to support patient engagement: where do we stand and where can we go?. J Am Med Inform Assoc 2017; 24 (06) 1088-1094
• 14 Huerta TR, McAlearney AS, Rizer MK. Introducing a patient portal and electronic tablets to inpatient care. Ann Intern Med 2017; 167 (11) 816-817
• 15 Law EC, Roto V, Hassenzahl M, Vermeeren A, Kort J. Understanding, scoping and defining user experience: a survey approach. Proceedings of the SIGCHI conference on human factors in computing systems. ACM 2009; 719-728
• 16 Bolle S, Romijn G, Smets EM, Loos EF, Kunneman M, van Weert JC. Older cancer patients' user experiences with web-based health information tools: a Think-Aloud study. J Med Internet Res 2016; 18 (07) e208
• 17 Haun JN, Lind JD, Shimada SL. , et al. Evaluating user experiences of the secure messaging tool on the Veterans Affairs' patient portal system. J Med Internet Res 2014; 16 (03) e75
• 18 Kim J. A qualitative analysis of user experiences with a self-tracker for activity, sleep, and diet. Interact J Med Res 2014; 3 (01) e8
• 19 O'Leary KJ, Sharma RK, Killarney A. , et al. Patients' and healthcare providers' perceptions of a mobile portal application for hospitalized patients. BMC Med Inform Decis Mak 2016; 16 (01) 123
• 20 Wade-Vuturo AE, Mayberry LS, Osborn CY. Secure messaging and diabetes management: experiences and perspectives of patient portal users. J Am Med Inform Assoc 2013; 20 (03) 519-525
• 21 Cohen T, Kaufman D, White T. , et al. Cognitive evaluation of an innovative psychiatric clinical knowledge enhancement system. Stud Health Technol Inform 2004; 107 (Pt 2): 1295-1299
• 22 Yu H, Lee M, Kaufman D. , et al. Development, implementation, and a cognitive evaluation of a definitional question answering system for physicians. J Biomed Inform 2007; 40 (03) 236-251
• 23 Ericsson KA, Simon HA. Verbal reports as data. Psychol Rev 1980; 87 (03) 215-251
• 24 Faulkner L. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003; 35 (03) 379-383
• 25 Yen PY, Walker DM, Smith JG, Zhou MP, Menser T, McAlearney AS. Usability evaluation of a commercial inpatient portal. Int J Med Inf 2017; 110: 10-18
• 26 Holden RJ, Carayon P, Gurses AP. , et al. SEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics 2013; 56 (11) 1669-1686
• 27 Carayon P, Wetterneck TB, Rivera-Rodriguez AJ. , et al. Human factors systems approach to healthcare quality and patient safety. Appl Ergon 2014; 45 (01) 14-25
• 28 Valdez RS, Holden RJ, Novak LL, Veinot TC. Transforming consumer health informatics through a patient work framework: connecting patients to context. J Am Med Inform Assoc 2015; 22 (01) 2-10
• 29 Weir CR, Nebeker JJ, Hicken BL, Campo R, Drews F, Lebar B. A cognitive task analysis of information management strategies in a computerized provider order entry environment. J Am Med Inform Assoc 2007; 14 (01) 65-75
• 30 Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res 2015; 17 (02) e44
• 31 Laccetti AL, Chen B, Cai J. , et al. Increase in cancer center staff effort related to electronic patient portal use. J Oncol Pract 2016; 12 (12) e981-e990
• 32 Grossman LV, Choi SW, Collins S. , et al. Implementation of acute care patient portals: recommendations on utility and use from six early adopters. J Am Med Inform Assoc 2017; DOI: 10.1093/jamia/ocx074.
• 33 Dalal AK, Dykes PC, Collins S. , et al. A web-based, patient-centered toolkit to engage patients and caregivers in the acute care setting: a preliminary evaluation. J Am Med Inform Assoc 2016; 23 (01) 80-87
• 34 Collins SA, Rozenblum R, Leung WY. , et al. Acute care patient portals: a qualitative study of stakeholder perspectives on current practices. J Am Med Inform Assoc 2017; 24 (e1): e9-e17
• 35 Delbanco T, Walker J, Darer JD. , et al. Open notes: doctors and patients signing on. Ann Intern Med 2010; 153 (02) 121-125
• 36 Hefner JL, Sieck CJ, Walker DM, Huerta TR, McAlearney AS. System-wide inpatient portal implementation: survey of health care team perceptions. JMIR Med Inform 2017; 5 (03) e31
• 37 Office of the National Coordinator for Health Information Technology. Certified Health IT Vendors and Editions Reported by Hospitals Participating in the Medicare EHR Incentive Program. Secondary Certified Health IT Vendors and Editions Reported by Hospitals Participating in the Medicare EHR Incentive Program; 2016. Available at: https://dashboard.healthit.gov/quickstats/pages/FIG-Vendors-of-EHRs-to-Participating-Hospitals.php . Accessed September 14, 2017