The My Diabetes Care Patient Portal Intervention: Usability and Pre-Post Assessment

Abstract

Background My Diabetes Care (MDC) is a novel, multifaceted patient portal intervention designed to help patients better understand their diabetes health data and support self-management. MDC uses infographics to visualize and summarize patients' diabetes health data, incorporates motivational strategies, and provides literacy level–appropriate educational resources.

Objectives We aimed to assess the usability, acceptability, perceptions, and potential impact of MDC.

Methods We recruited 69 participants from four clinics affiliated with Vanderbilt University Medical Center. Participants were given 1 month of access to MDC and completed pre- and post-questionnaires including validated measures of usability and patient activation, and questions about user experience.

Results Sixty participants completed the study. Participants' mean age was 58, 55% were females, 68% were Caucasians, and 48% had limited health literacy (HL). Most participants (80%) visited MDC three or more times and 50% spent a total of ≥15 minutes on MDC. Participants' median System Usability Scale (SUS) score was 78.8 [Q1, Q3: 72.5, 87.5] and significantly greater than the threshold value of 68 indicative of “above average” usability (p < 0.001). The median SUS score of patients with limited HL was similar to those with adequate HL (77.5 [72.5, 85.0] vs. 82.5 [72.5, 92.5]; p = 0.41). Participants most commonly reported the literacy level–appropriate educational links and health data infographics as features that helped them better understand their diabetes health data (65%). All participants (100%) intended to continue to use MDC. Median Patient Activation Measure® scores increased postintervention (64.3 [55.6, 72.5] vs. 67.8 [60.6, 75.0]; p = 0.01).

Conclusion Participants, including those with limited HL, rated the usability of MDC above average, anticipated continued use, and identified key features that improved their understanding of diabetes health data. Patient activation improved over the study period. Our findings suggest MDC may be a beneficial addition to existing patient portals.

Keywords

patient portal - interfaces and usability - patient engagement - diabetes mellitus - patient self-care

Background and Significance

Diabetes self-management can prevent or delay highly morbid diabetes-related complications including kidney failure, vision loss, and cardiovascular disease; yet patients struggle to consistently engage in recommended self-care behaviors.[1] [2] Patient activation (i.e., knowledge, skills, and confidence to manage their own health care) is essential to optimal diabetes self-management.[3] [4] Studies have found that patients with higher activation scores are more likely to engage in diabetes self-care behaviors.[4] [5]

Patient portals offer a promising platform to increase patient activation, enhance care, and promote self-management while overcoming the limitations of costly, difficult-to-scale, face-to-face interventions.[6] [7] Patient portals may provide an engaging and convenient means for patients to track and visualize their health data, obtain education and guidance, and connect with their health care team.[8] However, despite increasing patients' access to their health information, user interface design and health literacy issues have led to varied impact of patient portals on patient engagement and clinical outcomes.[7] [9] Studies examining barriers to portal use highlight the need for improved usability.[9] [10] Disjointed and complex user interfaces can frustrate and discourage users who struggle to identify relevant information.[7] Studies suggest that incorporating user-friendly data displays and effective self-management strategies may improve uptake and satisfaction.[11] However, little is known about how to optimize patient portals to improve patients' understanding of their data and support self-management of chronic diseases like diabetes.

To leverage the potential of patient portals and overcome existing limitations, we recently applied user-centered design sprint methodology and key strategies for patient engagement to develop a patient portal intervention called My Diabetes Care (MDC).[12] MDC is a multifaceted intervention embedded within an established patient portal, My Health at Vanderbilt (MHAV), and designed to help patients better understand their diabetes health data and support self-management. MDC uses infographics to visualize and summarize patients' diabetes health data, incorporates motivational strategies (e.g., social comparisons), and provides literacy level–appropriate educational resources.[12] [13] MDC is grounded in the well-established Chronic Care Model (CCM) adapted for eHealth (i.e., health care practices supported by electronic processes and communication; [Fig. 1]).[14] By leveraging elements within the model's five domains (self-management support, delivery system design, decision support, clinical information systems, and eHealth education), MDC has the potential to create more informed and activated patients leading to improved outcomes.

Fig. 1 The eHealth Enhanced Chronic Care Model (eCCM) with key aspects of the intervention superimposed in italics.

Objectives

We aimed to assess the usability and acceptability of MDC among patients with type 2 diabetes mellitus (T2DM) and users' perceptions of specific features and potential improvements. In addition, we sought to assess the potential impact of MDC on secondary cognitive and behavioral outcomes including patient activation.

Methods

Study Design, Setting, and Participant Recruitment

Between December 2018 and May 2019, we conducted a multimethod, pre-post study of MDC's usability, acceptability, and potential impact (Clinicaltrials.gov Identifier: NCT03786718). Potential participants were identified automatically by querying the electronic health record (EHR) for patients at four Vanderbilt University Medical Center (VUMC) affiliated clinics (three primary care and one endocrinology) located in Nashville, TN, that met the discrete inclusion and exclusion criteria. An EHR (Epic Systems Corp.) stores all clinical data, and patients receive access to their clinical data via an integrated patient portal, MHAV, run on Epic's MyChart platform. At VUMC, over 80% of established primary care patients with diabetes have an MHAV account. Patients were eligible if they were age ≥21 years, had T2DM, were taking at least one antihyperglycemic medication, reported reliable access to a computer with internet access, and had an existing MHAV account. We excluded patients with known cognitive deficits, severe visual or hearing impairment, unintelligible speech (e.g., dysarthria), residing in long-term care facilities, or currently participating in another diabetes-related research study.

We sent identified patients (n = 1,316) a letter describing the study. Study flyers were also placed at the four clinic sites. Interested patients (n = 141) completed an online study eligibility screener. Eligible patients (n = 78) could complete an electronic consent form and enroll online via REDCap (Research Electronic Data Capture).[15] A research assistant reviewed the study procedures with participants by phone and confirmed eligibility. To reflect a range of patient experience with diabetes as well as groups with distinct usability challenges, we limited enrollment based on patient demographics to enroll a sample with at least 20% representation of each of the following characteristics: (1) limited health literacy and (2) age older than 65 years.[9] [16] Participants could have one or more of these characteristics. Health literacy was assessed using a validated one-item screener that asked respondents to rate their confidence independently filling out medical forms.[17] Consistent with prior studies, we categorized participants noting any lack of confidence filling out medical forms as having limited health literacy.[9] [18] Participants were given access to MDC for 1 month. Of the 78 eligible patients, 69 enrolled and 60 patients ultimately completed the study (i.e., used MDC and submitted both pre- and poststudy questionnaires).

Intervention

MDC was built using Substitutable Medical Applications, Reusable Technology on Fast Healthcare Interoperability Resource (SMART on FHIR) standards and launched from under the “Resources” menu within MHAV using the SMART on FHIR launch method.[19] [Fig. 2] shows screenshots of MDC and highlights several features designed to help patients better understand their diabetes health data as well as support self-management:

Fig. 2 My Diabetes Care (MDC) screenshot and features.

• Star status indicator: The corresponding star filled in when the patient's most recent HbA1C, blood pressure, cholesterol, or flu vaccine status was at goal (e.g., HbA1C < 7). This corresponds to a value in the green zone on the individual infographics for each measure.

• Info icons: Hover-over info icons adjacent to the title of each measure of diabetes health provided a brief literacy level–appropriate (at or below an eighth grade reading level) description of the measure and its relevance to the users' health.[20]

• Health data infographic: For the three selected quantitative measures of diabetes health (i.e., A1c, blood pressure, and cholesterol), an infographic—a ruler with green, yellow, and red zones depicting goal, caution, and warning ranges, respectively[21]—displayed the patient's value relative to these ranges. Hovering over the “Me” indicator icon revealed the patient's three most recent values for the measure and their associated dates.

• Patients Like Me: A “Patients Like Me” indicator icon showed the average value of the diabetes health measure among similar patients (i.e., VUMC patients with diabetes of the same gender, age group, and insulin-use status). For example, if the user was a 53-year-old woman with diabetes on insulin, then the “Patients Like Me” indicator icon indicated the average value among other VUMC patients who are women in their 50s on insulin. Hovering over the “Patients Like Me” indicator icon revealed this information to the user.

• Literacy level–appropriate educational links: For each selected measure of diabetes health, MDC provided links to patient education resources appropriate for patients with limited health literacy that were developed at our institution[22] and by Learning About Diabetes, Inc.[23] These resources were specifically designed to be easily understood and provided general information about the importance of each measure and self-care education for improving diabetes health.

• Secure messaging: A “Message Your Doctor” button allowed patients to send a secure message to members of their health care team. The button called attention to a standard feature of MHAV that was also available to MHAV users outside of MDC via the “Messaging” icon at the top of the screen.

• Diabetes online community: An embedded link provided users the ability to navigate directly to the American Diabetes Association (ADA) Support Community designed to empower and encourage patients with diabetes through learning and interacting with others.[24] To access the community, patients had to create a separate account with the ADA and log in with their ADA username and password.

Data Collection

Study participants completed questionnaires electronically via e-mail using REDCap at two time points: baseline/preintervention (T ₀) and 1-month follow-up/postintervention (T ₁). Questionnaires included several validated scales to assess study outcomes as well as intervention-specific questions to assess attitudes and experience with MDC. The baseline questionnaire (T ₀) included basic demographic questions, items about computer usage, and validated measures of health literacy[25] and eHealth literacy.[26]

To elicit deeper understanding of participants' perceptions and experience with specific features of MDC, an experienced interviewer (WM) conducted phone interviews with a subsample of patients using a semistructured interview guide ([Supplementary Appendix A] [available in the online version]). Subjects were invited to participate in interviews on a rolling basis until saturation was reached (i.e., no new usability concerns or suggested improvements in the preceding two interviews).[27]

Participants were compensated $40 and $35 for completing the pre- and postintervention questionnaires, respectively, and an additional $40 if they completed a semistructured interview. To maximize our ability to uncover opportunities for improvement, we incentivized use of MDC by providing participants an additional $5 if they used MDC for a total of ≥10 minutes during the study period.

Outcome Measures

The primary outcomes were usability and user experience at T ₁. Usability of MDC was assessed by the 10-item System Usability Scale (SUS).[28] To prevent confusion, the scale was adapted for the study by replacing the word “system” in each item with the name of the study intervention. The items were rated on a 5-point Likert scale; item responses were summed and then converted to a score ranging from 0 (worst) to 100 (best). Based on prior research, a score above 68 was considered above average.[29] Study-specific questionnaire items assessed participants' perceptions of the content, layout, and acceptability of MDC. Participant statements from semistructured interviews provided a deeper understanding of user experience.

Secondary outcomes included self-reported system usage including total number of MDC visits, total duration of MDC use, and use of embedded educational resources, secure messaging, and the ADA Online Support Community. In addition, we assessed the potential impact of the MDC by examining the pre-post change (T ₀ to T ₁) in the following secondary cognitive and behavioral outcomes assessed using validated scales: patient activation,[30] diabetes self-efficacy,[31] diabetes care understanding,[32] diabetes knowledge,[33] diabetes self-care,[34] and diabetes distress.[35] Descriptions of each scale are provided in [Supplementary Appendix B] (available in the online version).

Data Analysis

Quantitative

We used descriptive statistics to characterize the study participants and survey responses. Due to non-normality, we used nonparametric tests to analyze the data. We used a one sample median test to compare the median SUS score at T ₁ to the threshold score of 68 indicative of “above average” usability[29] and Wilcoxon signed-rank sum test to assess whether there was a significant improvement in the continuous secondary cognitive/behavioral outcomes from baseline to end of study (T ₀ to T ₁). We performed the Wilcoxon–Mann–Whitney test to compare the distributions of two independent groups, McNemar's test for paired proportions, and Fischer's exact test for independent proportions. All analyses were completed using SAS Enterprise 7.15 (SAS Institute Inc., Cary, NC). In the context of the primary outcome, SUS score, and assuming a standard deviation of 12 based on prior studies,[36] with a sample size of at least 50, a one-sample t-test would detect an absolute difference in mean SUS scores of at least 5 points above the threshold score of 68 with 82% power.

Qualitative

We transcribed participants' semistructured interviews for coding. Similar to other health app usability studies[12] [37] and consistent with grounded theory,[38] we used selective coding to identify participants' statements addressing six established elements of user experience[39] (i.e., the core category): (1) useful—fulfilling a need; (2) usable—ease of use; (3) accessible—easy to access and comprehend; (4) desirable—design elements that evoke emotion or appreciation; (5) findable—easy to navigate; and (6) credible—trustworthy and believable. Two trained coders, a medical student author (TK) and an undergraduate research assistant independently coded all interviews and then resolved any differences by consensus. We reviewed participant statements in each category to inform potential revisions of MDC that could improve user experience.

Results

Participants

Of the 69 patients that enrolled, 60 (87%) completed the study (i.e., used MDC and submitted both pre- and poststudy questionnaires). [Table 1] shows the baseline characteristics of the 60 participants who completed the study. Participants' mean age was 58 years, 55% were females, 68% were Caucasians, 40% were taking insulin, and the average hemoglobin A1c level was 7.5% (range: 4.9–12.5). Regarding characteristics associated with unique usability challenges, 33% of participants were aged ≥65 years and 48% had limited health literacy. Compared to participants who completed the study, noncompleters were significantly more likely to have a most recent A1c ≥9 (8/58 [14%] vs. 5/9 [56%]; p = 0.01; see comparison of completers vs. noncompleters contained in [Supplementary Table S1] [available in the online version]).

Table 1 Characteristics of the study participants at baseline (n = 60)

Characteristic
Gender, n (%)
Female	33 (55)
Male	27 (45)
Age
Age, mean ± SD (range)	57.5 ± 13.0
Under 65	40 (67%)
65 and over	20 (33%)
Race/ethnicity, n (%)
Black	13 (22%)
White	41 (68%)
Other	6 (10%)
Education, n (%)
High school/general educational development (GED)	5 (8)
Some college/technical school	24 (40)
College degree	19 (32)
Graduate or professional degree	12 (20)
Health literacy, n (%)
Adequate	31 (52)
Limited	29 (48)
eHealth Literacy
eHEALS, mean ± SD (range)	31.5 ± 5.2 (11–40)
Patient portal usage at baseline, n (%)
Weekly	26 (43)
Monthly	29 (48)
Other	5 (8)
Device usage, n (%)[a]
Desktop	50 (83)
Tablet	40 (67)
Smartphone	52 (87)
Medical history
A1c, mean ± SD (range)[b]	7.5 ± 1.6 (4.9–12.5)
A1c ≥ 9, n (%)[b]	8 (14)
Use insulin, n (%)	24 (40)
Years with diabetes, mean ± SD (range)	13.1 ± 10.0 (1–54)
Hyperlipidemia, n (%)	47 (78)
Hypertension, n (%)	51 (85)

^a Participants could select more than one so totals may exceed 100%.

^b Two participants had no available A1C value.

Self-reported System Usage

Most participants (80%) reported visiting MDC three or more times and half (50%) of them reported spending a total of ≥15 minutes on MDC during the 1-month study period. Literacy level–appropriate educational links and info icons with explanations of the health measures were the two most reported features used (85 and 82%, respectively). Additional details on system usage are available in [Supplementary Table S2] (available in the online version).

Usability

Participants' median SUS score was 78.8 [ 72.5, 87.5], significantly greater than the threshold value of 68 indicative of “above average” usability (p < 0.001). The median SUS score of patients with limited health literacy was not significantly different from those with adequate health literacy (77.5 [72.5, 85.0] vs. 82.5 [72.5, 92.5]; p = 0.41). Participants aged ≥65 years had lower median SUS scores than younger participants (73.8 [53.8, 85.0] vs. 82.5 [75.0, 91.3]; p = 0.01). When compared with younger participants, those aged ≥65 years were less likely to report MDC was “easy to use” (85%, 34/40 vs. 55%, 11/20; p = 0.004), had “well-integrated functionality” (80%, 32/40 vs. 45%, 9/20; p = 0.009), and “wanted to use the [MDC] frequently” (85%, 34/40 vs. 55%, 11/20; p = 0.02). eHealth literacy was not significantly lower among participants aged ≥65 years compared with those younger than 65 years (median eHEALS score, 31.0 [30.0, 33.0] vs. 31.5 [29.0, 34.0]; p = 0.80).

User Experience and Acceptability

Questionnaire

[Fig. 3] shows the MDC features that participants reported were useful to managing their diabetes as well as those they felt should be removed. More than half of the participants reported the health data infographic (i.e., ruler), patients like me, and literacy level–appropriate educational links were useful for managing their diabetes. Whereas few patients desired any of the seven features to be removed, the two features participants most reported should be removed were patients like me and star status (7%).

Fig. 3 Participants' attitudes regarding My Diabetes Care (MDC) features. *Participants' responses to separate survey items querying the MDC features they found useful to managing their diabetes as well as those they felt should be removed. Secure message was not presented as a response option for removal because it is a standard feature of the underlying MyChart platform.

In addition, participants most frequently reported the literacy level–appropriate educational links and health data infographics were features that helped them better understand their diabetes health data (65%). Participants most frequently indicated the ability to upload blood sugar readings as a functionality that would make MDC more useful if added (73%). All participants (100%) reported they would continue to use MDC going forward.

Semistructured Interviews

[Table 2] shows illustrative quotes from the semistructured interviews reflecting core categories of user experience. Saturation was reached after 14 participants were interviewed. The quotes highlight some suggested areas for improvement including the following: integrating MDC into the patient portal mobile app, creating a walkthrough for MDC to orient new users, refining the ruler infographic, updating information as new therapies emerge, adding more diet information including recipes, and depicting improvement or worsening of values (e.g., A1C) over time.

Table 2 Illustrative quotes of suggested improvements from semi-structured interviews

User experience category	Illustrative quote
Accessible
Mobile app integration	It would be best for me or for a lot of people to be smartphone or iPhone or iPad accessible or whatever
Findable
Locating features	[Other patients] are not going to know much about even how to navigate around it without instructions
Usable
Orientation of values on ruler infographic	[Values on the ruler] they're horizontal. For me, to understand, […] so I turned my head sideways to kind of see what I'm reading, you know for the [ruler]. And I'm kind of sideways reading, and I'm like, “Nah, I don't really like how long I had my head cocked to the side to read that,” you know?
Useful
Educational links	There's nothing here about, let's say drugs for diabetes. What's new, what you should worry about, because some drugs do start having some very strange side effects […] There are new drugs [...] here's some interesting information, you may want to discuss it with your doctor kind of thing. I don't mean it should go beyond and suggest you switch More […] nutrition information [and] recipes would be good
Infographic	Where was it the last time[…] If it was right on that [infographic], it might be easier […] you don't have to say the eight is better or worse [than your last A1C], you could just see where it was on the [infographic], in one picture

Secondary Cognitive and Behavioral Outcomes

[Table 3] shows pre-post changes in cognitive and behavioral outcomes. Patient activation, diabetes self-efficacy, and self-reported adherence to a healthy diet improved, whereas no change was observed in other elements of self-reported diabetes self-care, diabetes care understanding, or diabetes knowledge. Similarly, there was no change in diabetes distress. A subgroup analysis of patients with limited health literacy showed postintervention improvement in both diabetes self-efficacy and diabetes care understanding (see [Supplementary Table S3] [available in the online version]).

Table 3 Pre-post intervention change in cognitive and behavioral outcomes

		Preintervention		Postintervention
Outcome	n	mean ± SD	median [Q1, Q3]	mean ± SD	median [Q1, Q3]	p -value[a]
Patient activation[30]	60	64.3 ± 11.9	64.3 [55.6, 72.5]	68.2 ± 12.6	67.8 [60.6, 75.0]	0.01
Diabetes self-efficacy[31]	59	26.6 ± 5.3	26.0 [23.0, 30.0]	28.1 ± 5.2	29.0 [25.0, 32.0]	0.001
Diabetes care understanding[32]	59	3.6 ± 0.7	3.4 [3.0, 4.0]	3.6 ± 0.6	3.7 [3.0, 4.0]	0.15
Diabetes knowledge[33]	59	10.7 ± 1.2	11.0 [10, 12]	10.7 ± 1.4	11.0 [10, 12]	0.77
Diabetes distress[35]	59	6.1 ± 5.1	5.0 [2.0, 9.0]	6.1 ± 5.3	5.0 [2.0, 9.0]	0.86
Diabetes self-care[34]
General diet	60	3.7 ± 1.9	4.0 [2.8, 5.0]	4.1 ± 1.6	4.0 [3.5, 5.0]	0.02
Specific diet	60	3.5 ± 1.6	3.5 [2.0, 5.0]	3.8 ± 1.5	4.0 [2.5, 5.0]	0.13
Exercise	60	2.5 ± 2.1	2.5 [0.5, 3.8]	2.6 ± 2.0	2.3 [1.3, 4.0]	0.35
Foot care	60	3.2 ± 2.4	3.5 [1.0, 4.8]	3.1 ± 2.3	3.0 [1.0, 5.0]	0.65
Glucose testing	50[b]	4.9 ± 2.2	5.3 [3.0, 7.0]	5.0 ± 2.1	6.0 [3.0, 7.0]	0.67

^a Wilcoxon signed-rank sum test.

^b Ten participants reported they do not test their blood sugar.

[Table 4] shows the pre-post changes in knowledge of diabetes health measures. Participants' likelihood of correctly identifying the goal range for systolic blood pressure (SBP) and low-density lipoprotein (LDL) cholesterol improved over the study period (28 vs. 47%, p = 0.02 and 26 vs. 53%, p < 0.001, respectively). Participants' likelihood of identifying the correct definition of A1c, SBP, LDL cholesterol, flu vaccine, and the goal range for A1c and the recommended frequency of flu vaccination did not change over the study period.

Table 4 Pre- and postintervention change in knowledge of diabetes health measures

Identify the correct	n	Preintervention, N (%)	Postintervention, N (%)	p-value[a]
Definition of A1c	58	40 (69)	45 (78)	0.30
Goal range for A1c	59	50 (85)	50 (85)	1.00
Definition of systolic blood pressure	59	43 (73)	46 (78)	0.63
Goal range for systolic blood pressure	59	17 (28)	28 (47)	0.02
Definition of LDL cholesterol	59	22 (37)	25 (42)	0.55
Goal range for LDL cholesterol	58	15 (26)	31 (53)	<0.001
Definition of flu vaccine	60	58 (97)	59 (98)	1.00
Recommended frequency of flu vaccination	60	58 (97)	59 (98)	1.00

Abbreviation: LDL, low-density lipoprotein.

^a McNemar's test.

Discussion

Our study showed that participants, including those with limited health literacy, highly rated the usability of MDC and all participants anticipated continuing to use it. Our data on user experience showed the majority of participants found that a ruler infographic depicting goal, caution, and warning ranges for diabetes health data and links to literacy level–appropriate diabetes health information helped them to better understand their diabetes health data. This study also identified areas for improvement including adding information about diabetes medications and periodic news about new discoveries, medicines, lifestyle recommendations, and recipes. It is hoped that these enhancements may help promote and sustain engagement with MDC over time. Consistent with the eHealth CCM,[14] we found a 4-point increase in patient activation scores among study participants after 1 month of access to MDC. Changes of 3 to 4 points in the Patient Activation Measure® are associated with a change from not engaging to engaging in particular diabetes self-care behaviors.[5] Although a larger randomized controlled trial is needed to more definitively assess the impact MDC on patient activation, the fact that diabetes self-efficacy scores also improved suggests that using MDC may positively affect diabetes patients' cognitions.

Beyond MDC, our study has important implications for patient portals more broadly. First, whereas patient portals allow patients to access their health data, patients may have difficulty understanding the data and their implications for their health. Complex data displays with multiple test results in a small format and without clearly defined normal values or information on the clinical significance of abnormal results can make it challenging to locate and determine the meaning of a particular test result.[7] Patients in our study reported that a simplified visual representation of normal, moderately abnormal, and more severely abnormal ranges and associated literacy level–appropriate materials improved their understanding of their diabetes health data. In addition, participants' ability to identify the goal range for SBP and LDL cholesterol improved over the 1-month study period. To get the most out of viewing their test results on patient portals, patients need to understand (1) what the test is for, (2) how to interpret the test, and (3) what to do in response.[40] [41] Further research is needed to directly assess the impact of different data displays and user interface designs on patients' ability to achieve this level of understanding.[41]

Second, among patients with diabetes, limited health literacy is typically associated with worse health outcomes and can be a barrier to patient portal use.[9] [42] Tieu et al have shown that despite expressing enthusiasm for patient portal use, patient portal users with limited health literacy struggle with medical terminology and a lack of literary level–appropriate health information.[9] [18] By addressing these issues in the user-centered design of MDC, we found that patients with both adequate and limited health literacy rated the usability of MDC high. Whereas patient portals have the potential to worsen health disparities by further advantaging well-educated patients with greater resources, if designed and implemented appropriately, patient portals also have the potential to lower health literacy demands by ensuring that patients are presented with the health information and resources in a format that is convenient and easy to navigate and understand.[43] Further attention to the use of audio, video, and graphics within patient portals, offering portals in patients' preferred language, and improving computer and smartphone access and training may lead to additional improvements in the ability of vulnerable groups to realize the full benefits of patient portals.[10]

Third, prior research indicates that older adults are less likely to be able to use patient portals for health-related tasks.[16] Price-Haywood et al observed that older adults with hypertension or diabetes found accessing patient portals cumbersome, expressed concerns about the level computer literacy required, and desired technical support.[44] Similarly, we found that patients aged ≥65 years rated the usability of the intervention lower than those younger than 65 years. However, older patients in our study did not report lower eHealth literacy. Wildenbos et al noted that older patient portal users not only experienced more user interactions problems but also had higher expectations of portal content.[45] To ensure that the benefits of patient portals reach the greatest number of patients, health care organizations should provide and promote the availability of technical support and training and enable patients to delegate proxy users.[46]

Several other studies have examined the usability of technology-delivered diabetes self-management interventions using the SUS. Georgsson et al examined the usability of Care4Life, an interactive short messaging service (SMS) with an accompanied diabetes patient web portal.[47] Unlike MDC, Care4Life was not tethered to an EHR; rather, patients input their data (e.g., blood glucose, weight, etc.) and then visualize the data via a web service. Like MDC, Care4Life uses infographics with a red–yellow–green color scheme to help patients interpret their data. Among the 10 patients studied, the average SUS score (80.5) closely matched the SUS score for MDC. Notably both SUS scores were considerably higher than the published SUS scores of some commercially available diabetes apps.[48] [49]

Limitations

Our study has important limitations. Whereas our sample size was modest and represented a convenience sample that may limit its generalizability, the sample was larger than other studies of this type[37] [50] and provided sufficient power to determine if the usability of MDC was above a threshold value for “above average” usability on the SUS. We purposively sampled patients aged ≥65 years and those with limited health literacy to help ensure MDC is satisfying to patients who may have unique usability challenges.[9] [16] However, patients with very low literacy levels are less likely to have e-mail and internet access and to be patient portal users.[18] Therefore, the limited health literacy patients in our study are unlikely to include patients with the very lowest literacy levels. We limited our study to adult patients given that T2DM is rare (prevalence <0.01%) in adolescents and few adolescents have autonomous patient portal accounts.[2] As patient portal use grows, more research is needed to determine if patient portal interventions would work for adolescents including those with joint (child and guardian) access accounts.[51] Approximately 11% of individuals sent a recruitment letter responded. Whereas this response rate is similar to other studies,[52] it is relatively low and respondents—patients interested in participating in a study of a patient portal intervention for diabetes—may differ from the overall population in their attitudes, baseline levels of activation, and enthusiasm for technology-delivered interventions; therefore, our results may be subject to self-selection bias. Participants in our study used the intervention for a relatively short duration and therefore user experience may not represent user experience over longer periods. Notably, participants with an A1C above 9% were less likely to complete the study, highlighting the challenge of sustaining engagement among some patients at high risk.

The compensation paid to study participants may have also inflated system usage; however, promoting usage was important to gather feedback from users at this formative stage. Due to a programming issue identified after the start of the study, we were not able to reliably capture user analytics. Therefore, our system usage data are limited to self-report and subject to recall bias. Identifying this issue in the context of this study, allowed us to correct the issue prior to a planned randomized controlled trial and underscores the importance of usability studies in advance of larger outcome trials. Whereas we encouraged honest feedback to improving the design MDC, social desirability bias may have influenced participants' evaluations. Our study lacked a control group and was not designed nor powered to definitely assess the impact of MDC on cognitive or behavioral outcomes; therefore, the observed pre-post changes in patient activation and other secondary outcomes should only be viewed as exploratory and need to be confirmed in future studies. For this reason, we chose not to adjust for multiple comparisons.[53] Finally, whereas these cognitive and behavioral outcomes are important, we plan to assess more objective clinical outcomes such as change in glycemic control (HbA1c) in future studies.

Conclusion

Our findings suggest MDC may be a beneficial addition to existing patient portals for patients with diabetes. Participants in our study also provided feedback on the intervention that may create additional value for patients. The results support continued efforts to further refine the intervention and more definitively assess its impact of cognitive, behavioral, and clinical outcomes. By making use of existing health information technology (HIT) infrastructure and resources, patient portal interventions like MDC are more sustainable than costly and difficult to scale face-to-face interventions. By using SMART on FHIR standards and integrating MDC into Epic's MyChart platform with more than 127 million patient health records,[54] the intervention is also highly scalable. Finally, we hope that by attending to the unique needs of patients with limited health literacy, our intervention will be useful to the greatest number of patients.

Clinical Relevance Statement

This study advances knowledge in HIT and chronic disease self-management and education in important ways: (1) it furthers our understanding of effective patient-facing health data displays and health educational content for patients including those with limited health literacy; (2) it provides a model to assist in the creation of patient portal interventions that are effective and satisfying for patients to use; and (3) it introduces a specific, highly scalable patient portal intervention for patients with diabetes that demonstrates high usability and the potential to increase patient activation and knowledge while being integrated into a common EHR platform used by millions of patients.

Multiple Choice Questions

1. The My Diabetes Care patient portal intervention was designed to:

   • Provide patients with summary of their diabetes treatment plan.

   • Help patients better understand their diabetes health data as well as support self-management.

   • Prevent hypoglycemic episodes.

   Correct Answer: The correct answer is option b.

2. The objective of this study was to:

   • Assess the usability, acceptability, perceptions, and potential impact of My Diabetes Care.

   • Measure the impact of My Diabetes Care on medication adherence.

   • Evaluate the effect of My Diabetes Care on glycemic control.

   Correct Answer: The correct answer is option a.

Conflict of Interest

None declared.

Acknowledgments

The authors would like to thank Vanderbilt HealthIT for their help in programming and providing technical support for My Diabetes Care. The authors would also like to thank Audriana Nigg for her help with recruitment, data collection and preparation, and administrative support. We would like to express our deep gratitude to all the patients who participated in the study. Finally, we would like to acknowledge our cherished colleague and coauthor, Dr. Kenneth Wallston, who died after the submission of this manuscript. Ken was an inspiring colleague who generously shared his wisdom, experience, and passion for research to improve the lives of countless patients.

Author Contributions

W.M., A.J.H., S.T.R., K.A.W., and T.A.E. developed the concept and designed the study. Acquisition of data was done by W.M.. All the authors participated in the analysis and interpretation of data. W.M. drafted the manuscript. All the authors participated in critical revision of the manuscript for important intellectual content. W.M., A.J.H., G.B.H., T.K., S.T.R., and T.A.E. gave the final approval of the version to be published. W.M. was responsible for obtaining funding and supervised the study.

Protection of Human and Animal Subjects

The Vanderbilt University Institutional Review Board approved this study.

^† Kenneth A. Wallston passed away on October 27, 2020.

• References

• 1 Powers MA, Bardsley J, Cypress M. et al. Diabetes self-management education and support in type 2 diabetes: a joint position statement of the American Diabetes Association, the American Association of Diabetes Educators, and the Academy of Nutrition and Dietetics. Diabetes Educ 2017; 43 (01) 40-53
• 2 Centers for Disease Control and Prevention. National Diabetes Statistics Report, 2020. Atlanta, GA: Centers for Disease Control and Prevention, U.S. Department of Health and Human Services; 2020
• 3 Zimbudzi E, Lo C, Ranasinha S. et al. The association between patient activation and self-care practices: a cross-sectional study of an Australian population with comorbid diabetes and chronic kidney disease. Health Expect 2017; 20 (06) 1375-1384
• 4 Almutairi N, Hosseinzadeh H, Gopaldasani V. The effectiveness of patient activation intervention on type 2 diabetes mellitus glycemic control and self-management behaviors: a systematic review of RCTs. Prim Care Diabetes 2020; 14 (01) 12-20
• 5 Rask KJ, Ziemer DC, Kohler SA, Hawley JN, Arinde FJ, Barnes CS. Patient activation is associated with healthy behaviors and ease in managing diabetes in an indigent population. Diabetes Educ 2009; 35 (04) 622-630
• 6 Dalal AK, Dykes P, Samal L. et al. Potential of an electronic health record-integrated patient portal for improving care plan concordance during acute care. Appl Clin Inform 2019; 10 (03) 358-366
• 7 Baldwin JL, Singh H, Sittig DF, Giardina TD. Patient portals and health apps: Pitfalls, promises, and what one might learn from the other. Healthc (Amst) 2017; 5 (03) 81-85
• 8 Tulu B, Trudel J, Strong DM, Johnson SA, Sundaresan D, Garber L. Patient portals: an underused resource for improving patient engagement. Chest 2016; 149 (01) 272-277
• 9 Tieu L, Sarkar U, Schillinger D. et al. Barriers and facilitators to online portal use among patients and caregivers in a safety net health care system: a qualitative study. J Med Internet Res 2015; 17 (12) e275
• 10 Lyles CR, Nelson EC, Frampton S, Dykes PC, Cemballi AG, Sarkar U. Using electronic health record portals to improve patient engagement: research priorities and best practices. Ann Intern Med 2020; 172 (11, Suppl): S123-S129
• 11 Nystrom DT, Singh H, Baldwin J, Sittig DF, Giardina TD. Methods for patient-centered interface design of test result display in online portals. EGEMS (Wash DC) 2018; 6 (01) 15
• 12 Martinez W, Threatt AL, Rosenbloom ST, Wallston KA, Hickson GB, Elasy TA. A patient-facing diabetes dashboard embedded in a patient web portal: design sprint and usability testing. JMIR Human Factors 2018; 5 (03) e26
• 13 Martinez W, Wallston KA, Schlundt DG. et al. Patients' perspectives on social and goal-based comparisons regarding their diabetes health status. BMJ Open Diabetes Res Care 2018; 6 (01) e000488
• 14 Gee PM, Greenwood DA, Paterniti DA, Ward D, Miller LMS. The eHealth Enhanced Chronic Care Model: a theory derivation approach. J Med Internet Res 2015; 17 (04) e86
• 15 Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap): a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42 (02) 377-381
• 16 Taha J, Sharit J, Czaja SJ. The impact of numeracy ability and technology skills on older adults' performance of health management tasks using a patient portal. J Appl Gerontol 2014; 33 (04) 416-436
• 17 Sarkar U, Piette JD, Gonzales R. et al. Preferences for self-management support: findings from a survey of diabetes patients in safety-net health systems. Patient Educ Couns 2008; 70 (01) 102-110
• 18 Tieu L, Schillinger D, Sarkar U. et al. Online patient websites for electronic health record access among vulnerable populations: portals to nowhere?. J Am Med Inform Assoc 2017; 24 (e1): e47-e54
• 19 Boston Children's Hospital and Health Level Sevel International. HL7 FHIR SMART Application Launch Framework Implementation Guide Release 1.0.0. 2018. Accessed January 23, 2021 at: http://www.hl7.org/fhir/smart-app-launch/
• 20 Kincaid JP, Fishburne Jr RP, Rogers RL, Chissom BS. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Millington, TN: Institute for Simulation and Training, University of Central Florida; 1975
• 21 American Diabetes Association. Introduction: standards of medical care in diabetes-2018. Diabetes Care 2018; 41 (Suppl. 01) S1-S2
• 22 Wolff K, Chambers L, Bumol S. et al. The PRIDE (Partnership to Improve Diabetes Education) Toolkit: development and evaluation of novel literacy and culturally sensitive diabetes education materials. Diabetes Educ 2016; 42 (01) 23-33
• 23 Learning About Diabetes Inc. Learning About Diabetes. Accessed February 10, 2018 at: http://www.webcitation.org/6x8VV2atx
• 24 American Diabetes Association. ADA Support Community. 2020 . Accessed April 27, 2020 at: https://community.diabetes.org/home
• 25 Sarkar U, Schillinger D, López A, Sudore R. Validation of self-reported health literacy questions among diverse English and Spanish-speaking populations. J Gen Intern Med 2011; 26 (03) 265-271
• 26 Norman CD, Skinner HA. eHEALS: the eHealth Literacy Scale. J Med Internet Res 2006; 8 (04) e27
• 27 Ando H, Cousins R, Young C. Achieving saturation in thematic analysis: Development and refinement of a codebook. Compr Psychol 2014; 3: 3
• 28 Brooke J. SUS: quick and dirty usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland IL. eds. Usability Evaluation Industry. London: Taylor & Francis; 1996: 189-194
• 29 Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact 2008; 24 (06) 574-594
• 30 Hibbard JH, Mahoney ER, Stockard J, Tusler M. Development and testing of a short form of the patient activation measure. Health Serv Res 2005; 40 (6, Pt 1): 1918-1930
• 31 Wallston KA, Rothman RL, Cherrington A. Psychometric properties of the perceived diabetes self-management scale (PDSMS). J Behav Med 2007; 30 (05) 395-401
• 32 Fitzgerald JT, Davis WK, Connell CM, Hess GE, Funnell MM, Hiss RG. Development and validation of the Diabetes Care Profile. Eval Health Prof 1996; 19 (02) 208-230
• 33 Quandt SA, Ip EH, Kirk JK. et al. Assessment of a short diabetes knowledge instrument for older and minority adults. Diabetes Educ 2014; 40 (01) 68-76
• 34 Toobert DJ, Hampson SE, Glasgow RE. The summary of diabetes self-care activities measure: results from 7 studies and a revised scale. Diabetes Care 2000; 23 (07) 943-950
• 35 McGuire BE, Morrison TG, Hermanns N. et al. Short-form measures of diabetes-related emotional distress: the Problem Areas in Diabetes Scale (PAID)-5 and PAID-1. Diabetologia 2010; 53 (01) 66-69
• 36 Solomon M, Wagner SL, Goes J. Effects of a web-based intervention for adults with chronic conditions on patient activation: online randomized controlled trial. J Med Internet Res 2012; 14 (01) e32
• 37 Nelson LA, Bethune MC, Lagotte AE, Osborn CY. The usability of diabetes MAP: a web-delivered intervention for improving medication adherence. JMIR Human Factors 2016; 3 (01) e13
• 38 Strauss A, Corbin J. Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Newbury Park, CA: Sage; 1990
• 39 Usability.gov. User Experience Basics. Accessed September 2, 2019 at: https://www.usability.gov/what-and-why/user-experience.html
• 40 Giardina TD, Baldwin J, Nystrom DT, Sittig DF, Singh H. Patient perceptions of receiving test results via online portals: a mixed-methods study. J Am Med Inform Assoc 2018; 25 (04) 440-446
• 41 Turchioe MR, Myers A, Isaac S. et al. A systematic review of patient-facing visualizations of personal health data. Appl Clin Inform 2019; 10 (04) 751-770
• 42 Marciano L, Camerini A-L, Schulz PJ. The role of health literacy in diabetes knowledge, self-care, and glycemic control: a meta-analysis. J Gen Intern Med 2019; 34 (06) 1007-1017
• 43 Apter AJ. Can patient portals reduce health disparities? A perspective from asthma. Ann Am Thorac Soc 2014; 11 (04) 608-612
• 44 Price-Haywood EG, Harden-Barrios J, Ulep R, Luo Q. eHealth literacy: patient engagement in identifying strategies to encourage use of patient portals among older adults. Popul Health Manag 2017; 20 (06) 486-494
• 45 Wildenbos GA, Maasri K, Jaspers M, Peute L. Older adults using a patient portal: registration and experiences, one year after implementation. Digit Health 2018; 4: 2055207618797883
• 46 Son H, Nahm E-S. Older adults' experience using patient portals in communities: challenges and opportunities. Comput Inform Nurs 2019; 37 (01) 4-10
• 47 Georgsson M, Staggers N. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. J Am Med Inform Assoc 2016; 23 (01) 5-11
• 48 Fu HNC, Rizvi RF, Wyman JF, Adam TJ. Usability evaluation of four top-rated commercially available diabetes apps for adults with type 2 diabetes. Comput Inform Nurs 2020; 38 (06) 274-280
• 49 Fu HN, Adam TJ, Konstan JA, Wolfson JA, Clancy TR, Wyman JF. Influence of patient characteristics and psychological needs on diabetes mobile app usability in adults with type 1 or type 2 diabetes: crossover randomized trial. JMIR Diabetes 2019; 4 (02) e11462
• 50 Nelson LA, Mayberry LS, Wallston K, Kripalani S, Bergner EM, Osborn CY. Development and usability of REACH: a tailored theory-based text messaging intervention for disadvantaged adults with type 2 diabetes. JMIR Human Factors 2016; 3 (02) e23
• 51 Steitz BD, Wong JIS, Cobb JG, Carlson B, Smith G, Rosenbloom ST. Policies and procedures governing patient portal use at an academic medical center. JAMIA Open 2019; 2 (04) 479-488
• 52 Nijland N, van Gemert-Pijnen JE, Kelders SM, Brandenburg BJ, Seydel ER. Factors influencing the use of a web-based application for supporting the self-care of patients with type 2 diabetes: a longitudinal study. J Med Internet Res 2011; 13 (03) e71
• 53 Althouse AD. Adjust for multiple comparisons? It's not that simple. Ann Thorac Surg 2016; 101 (05) 1644-1645
• 54 Moukheiber Z. Behind Epic Systems, A Low-Key Health IT Company Called InterSystems. Forbes; 2013 . Accessed April 18, 2019 at: https://www.forbes.com/sites/zinamoukheiber/2013/03/04/behind-epic-systems-a-low-key-health-it-company-called-intersystems/?sh=7c981f6bd88a

Publication History

Received: 08 November 2020

Accepted: 01 April 2021

Article published online:
30 June 2021