Overcoming Challenges to Inclusive User-based Testing of Health Information Technology with Vulnerable Older Adults: Recommendations from a Human Factors Engineering Expert Inquiry

Keywords

User-Centered design - social inclusion - medical informatics applications - aged - health services accessibility

1 Introduction

To support the aging population, development of Health Information Technologies (HIT) for older adults is on the rise [[1], [2]]. These technologies, including health and wellness apps for smartphones (mHealth), wearables (medical devices), patient portals, smart home technologies and other digital health devices (eHealth), are being developed and marketed -- often directly to the consumer -- to promote self-management of chronic illness, maximize functional status, and foster safe, independent living. For example, HIT can improve patient medication adherence through the use of automated reminder systems (e.g., “smart” medication dispensers), provide self-care advice for people with chronic illnesses like diabetes or heart failure, or identify and mitigate fall risk factors in the home [[3] [4] [5]]. Existing research on HIT design emphasizes the importance of engaging older adult end-users throughout the design lifecycle to align interventions with user requirements, patient values, and context of use and to optimize usability [[6], [7]]. When this is not the case, HIT has been perceived to be “more trouble than it is worth” [[8], [9]], too time-consuming [[10]], unreliable [[11]], or generally burdensome [[12], [13]].

A range of usability engineering methods (UEM's) can be applied to provide insight on the user interface and/or functional problems end-users encounter when interacting with a system. Developers and human factors engineers can involve end-users by conducting usability tests and cataloging usability issues, workflow pain points, and eliciting new requirements. Common usability testing methods with end-users include user-simulations with think-aloud protocols, retrospective think aloud, semi-structured interviews, and self-reported satisfaction surveys (e.g., the System Usability Scale or Single Ease Questionnaire) [[14], [15]]. Such UEMs can contribute to improve HIT designs by aligning interventions with users' needs and 16]. In a paper examining the ethical concerns inherent with testing older adults [[17]], the authors argued that autonomous older adults with good functional status and high health literacy are more likely to participate in usability testing of HIT. By contrast, adults with advanced chronic illness, cognitive impairment, and physical disabilities may be under-sampled, and therefore, under-represented in usability testing. The same can be said of patients struggling with health-related social needs such as poverty, housing instability, and food insecurity. Not enrolling older adults and especially vulnerable ones in HIT usability-testing may lead to low accessibility of HIT for this demographic. Barriers to HIT use specific to this group will not be discovered and removed through the design process [[18], [19]]. To guide consumer-informatics design, (GW, LP, MJ) reviewed the literature on barriers to technology use among older adults and created the MOLD-US conceptual framework.

The MOLD-US framework identifies four key aging processes that may affect HIT use: (1) cognition, (2) motivation, (3) physical ability, and (4) perception [[20]]. Besides age-related declining capabilities, vulnerable older adults may also experience impairments arising from comorbidities, such as declining eyesight as a complication of diabetes, decreasing neuromuscular function with peripheral neuropathy or increased cognitive decline due to dementia. When (vulnerable) older adults are participating in usability tests, testing methods may not adequately account for the cognitive and functional impairments seen in older adults. For example, the Think-Aloud method provides a unique source of information on cognition: it generates direct data on concurrent thought processes during task performance [[21]]. As such, this method carries a certain “cognitive load” and makes assumptions about the cognitive abilities of participants, including communication, attention, working memory, visuospatial processing, and speed of comprehension. These abilities often progressively decline as a function of aging and older adults may not be able to participate in UEMs such as the Think-Aloud method. Consequently, sampling bias favoring younger adults can skew usability test results and lead to the implementation of HIT interventions that disenfranchise (vulnerable) older adults. This is especially of concern when these interventions are aimed to help consumers self-manage chronic illness or meaningfully participate in shared medical decision-making. New developments from the International Standards Organization (ISO) provide quality requirements and a health app quality score calculation method specifically for mobile applications [[22], [23]]. The new ISO standard aims to support the development and assessment of “easy to use” health- and wellness apps. Although attention has been paid to the importance of testing with intended end-users, how to perform these studies and what challenges developers and manufacturers face when targeting older adults remain undefined.

This paper offers an overview of expert-based recommendations for developers of consumer-facing HIT on how to conduct best-practice HIT usability testing with (vulnerable) older adults. Our ultimate aim is to support inclusive HIT design and evaluation for this demographic, and in doing so, improve the alignment of evaluated HIT to (vulnerable) older user populations.

2 Methods: Expert Inquiry

We first conducted a workshop at the Medical Informatics Europe Conference and invited experts in the human and organizational factors of HIT implementation and research to share their insights on usability testing with (vulnerable) older adults. The International Medical Informatics Association Working Group on Human Factor Engineering in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group on Human and Organizational Factors of Medical Informatics (HOFMI) endorsed and promoted this workshop. The aim of the workshop was to share expert insights, including how to improve representation of older adults, best-practice testing methods, and ways to improve the quality and validity of results. In total, 10 experts joined the workshop, of which seven had a high level of expertise. Afterwards, we completed a second round of expert inquiry to validate and iterate on the results from the workshop and to propose a final set of expert-derived recommendations. Five experts from the IMIA WG participated in the second round of expert inquiry.

2.1 Study Design

First, in a plenary workshop, we gave participating experts a summary of the MOLD-US framework to help the group understand the heterogeneity of aging and related comorbidities. We then shared our two research questions: (1) what barriers affect user-based usability testing of eHealth solutions with older adults? ; and (2) what are the best methods for usability testing with older adults?

Second, workshop facilitators evenly divided experts into three groups addressing each of the main HIT domains: eHealth, mHealth, and medical devices. We divided the groups according to experience level (using academic rank as a proxy), research focus, and number of human factors engineering publications in healthcare. LP, GW, and RM each facilitated one group. Participants answered the main questions by writing their answers on index cards, providing a brief explanation of the answer, and pinning them to a board. The facilitators used a semi-structured collective interview grid to explore in detail the barriers and testing methods listed on index cards and help experts synthesize and structure their answers. The facilitators then guided each group through an affinity mapping activity wherein participants discussed and sorted each barrier card into one or more related testing methods. The workshop was concluded by a plenary discussion on each group's result board and index cards, aimed to gain general consensus on the recommendations.

Finally, through purposeful sampling [[24]] members of the IMIA HFE in Health informatics WG with experience with HIT usability testing with (vulnerable) older adults (BL, DL, HM, LVV, VL) were invited to participate in the second round of expert inquiry. After formal acceptance to participate, the results from the workshop were shared to validate and provide additional insights to the results. Participants could iterate on each other's comments and were asked to use the following process to provide additional insights and validate the topics: (1) read the results in-depth; (2) provide comments; and (3) add any missing or needed discussion. Two of the authors (LP and TE) then completed a thematic analysis by coding all comments on the basis of the UCD framework [[25]] to identify if additional themes needed to be added and to validate the final organization of themes, recommendations, and key elements.

3 Results: Recommendations

The experts provided nine recommendations supported by 37 key-elements for conducting HIT usability testing with older adults. Recommendations were organized into three overarching themes: (1) empathetic approach and trust-building, (2) new requirements to testing and study design, and (3) adjustments to UEMs for testing with older adults. The following paragraphs describe each theme in detail and provide a checklist of the recommendations and key-elements per theme.

3.1 Empathetic Approach and Trust-Building

All expert groups mentioned that it was especially important to have an empathetic approach towards older adults throughout the whole scope of the evaluation project. This approach is important to build trust; if the older adults trust the evaluators, they will be more likely to share their true experiences and feedback regarding the HIT intervention. Building empathy with older adults and proctoring successful usability sessions requires several complementary strategies. First, when working with intended users of HIT, it is crucial to apply the more general “universal communication precautions protocol” that is used in communicating with patients [[26]]. The universal communications precautions approach outlines standard steps to optimize communication and understanding whilst avoiding implicit bias. Evaluators should emphasize throughout the evaluation(s) that there are no “incorrect” actions; all observations are important, and any findings or concerns identified will be used to improve the product and help countless other users. Also, be certain to provide brief and clear instruction, and avoid technical jargon. These instructions should be provided both verbally and in writing whenever possible. Experts also recommend using standardized easy-to-process scripts such as the script provided in [Table 1].

Table 1 Example standardized script for trust building in HIT usability testing

It is important to understand that usability testing for older adults is a social experience, not just an opportunity for researchers to garner information on designs. In addition, researchers should understand and respond to each older adult's motivation to participate. Participants may want to help science or technical innovation, see the testing session as a means for social contact, or a means to feel like a worthy and contributing member of society. Experts agreed that such an approach would increase older adults' motivation to participate in usability testing and their genuineness in expressing experiences with the tested intervention. Experts also recommended to frequently emphasize during testing why the older adult's involvement in this type of research is important. In doing so, the social impact can be explained and the difference between (the scope of) the research project and usual care activities can be addressed in more detail. Older adults may not be aware of this difference and, particularly those with cognitive impairments, may have trouble understanding research activities.

Finally, HIT evaluators must be adequately trained in communication skills, with emphasis upon cultural competence and effective techniques to engage older adults and those with physical and cognitive disabilities. Usability testing sessions should be followed up with direct observations and feedback sessions to continually reflect on communication skills and refining testing approaches (see [Table 2]).

Table 2 Recommendation and checklist of key elements mentioned by experts on the theme Empathetic approach and building trust.

3.2 New Requirements to Testing and Study Design

Experts concluded that setting additional requirements for the testing phase would improve the quality of testing. Experts specified three add-ons to plan and perform a usability test with older adults: (1) ensure pre-usability testing; (2) stimulate the involvement of relatives, friends, and caregivers; and (3) analyze older participants' capacities and skills through intake meetings and context analysis.

A foremost prerequisite when testing HIT with older patients was to perform pre-usability testing. Though bugs are often detected in usability testing, when involving older adults, it is crucial that program bugs have been eliminated as much as possible. Experiencing bugs in a system increases stress among older adult participants and it deters them from actually reporting on their interaction with design aspects that are the focus of the intended evaluation. When wireframes, mock-ups or prototypes are being used in testing with older adults, evaluators should take time to explain that most of the functionalities will not work as expected. During an early design phase, tasks should be kept simple and sessions should focus on just a few key features at a time. If the product passes initial tests, later tests can include more complex tasks or higher levels of fidelity. In doing so, pre-testing would be a new phase in the planning and performance of the HIT evaluation study. Evaluators first perform a pre-test of the version of the HIT application to detect bugs and/or complications older adults may face during usability testing. The detected bugs and/or complications can then be either solved before actual testing or included in the usability test script to support expectation management of the older adult when interacting with the HIT.

Next, encouraging involvement of family, friends or caregivers> in (vulnerable) older adult usability testing of HIT is crucial. This recommendation requires delicacy; some adults might consider this a sensitive or embarrassing topic if they do not believe they have friends, loved ones, or caregivers. Nonetheless, having a family member or friend present during the HIT usability testing is likely to support the older adult and enhance comprehensibility of the evaluation tasks to be performed by participants. Experts suggested including family members when using certain usability testing methods such as interviews or surveys. That is, conducting an interview with both a family member and the older adult present as well as interviewing the older adult and the family member separately. Involving caregivers (e.g., nurses, care navigators) as part of the expert team, evaluation team, or advisory panel is also essential, considering they are a consistent presence and contact person for many older adults.

According to the experts, intake meetings and context analysis of participant characteristics that are specific to the actual use of the HIT innovation may help to gain insight into what developers or Human Factor experts / usability researchers need to consider during the usability test. Depending on what is assessed or tested, and the population of study, researchers may want to include in their study design assessments of physical / cognitive skills in case they need to control for them. Ideally, these assessments would be conducted by professionals, as they can require additional expertise and sensitivity to administer. Alternatively, with informed consent, this information about the participants could be extracted from electronic records. If relevant, it is recommended to assess participants' digital skills before conducting the usability tests, to make sure to cover the range of low to high digital skills.

Experts mentioned that theoretical frameworks may inform contextual analysis pre-testing, such as those that recognize age-related and disease-related barriers possibly influencing HIT use, such as the MOLD-US framework or the extensions to this framework for considerate mHealth design [[19], [20]] (see [Table 3]).

Table 3 Recommendations and checklist of key elements mentioned by experts throughout the study on the theme Pre-testing and study design.

3.3 Adjustments to UEMs for Testing with Older Adults

Experts recommended several adjustments of current UEMs to improve testing with older adults. First, they recommended adapting the instructions and locations for testing to older adult participants. These adjustments focus on recognizing cognitive barriers of older adults by adapting the length and the set-up of the test. For example, as mentioned above, instead of having one longer usability test session in which several tasks are evaluated, a set of multiple short sessions can be performed. Each session may consist of one brief task, followed by a brief interview to obtain relevant information from the participant. Usability evaluations likewise can be performed in a set of several evaluations for instance spread out in a week's time or by planning a specific pop-up in a healthcare facility such as a day-test location at an outpatient clinic where older adults are invited to test an application. This allows for an unrushed execution of usability tasks. Another recommended adjustment, when recording usability problems, is to explicitly and recurrently explain to older adults why the evaluation is recorded and that recordings will be processed anonymously. This aims to provide the opportunity for the older adult to provide continuous consent [[27]].

Other recommendations regarding UEMs relate to the location of the evaluation. It is always best to emulate the context of use; a gold standard is to conduct tests in the users' homes or to shadow the participant there for a short period of time. If this is not possible, an alternative is to hold tests in “living labs”. These are laboratories designed to replicate the home environment and equipped with cameras and microphones to observe behavior as unobtrusively as possible. For example, living labs can mimic older adults equipped apartments and contain context specific elements such as grab bars, walk in bathtubs and telephones with big numbers. Performing evaluations at settings that are unfamiliar to older adults or look like a clinical lab or office, should be avoided as the results are highly unlikely to be representative of the real-world experiences of older adults.

Flexibility of testing protocols (e.g., providing more time to participants, taking breaks between testing) and constructing usability testing approaches with specific goals in mind can be more efficient, maximizing the useful information obtained whilst minimizing the burden on participants. Other suggested adjustments aim to adapt the usability testing method to allow for collective testing. These additions were related to the social experience of older adults participating in usability evaluations and the representativeness of the test to participants' social use of the HIT. With the traditional Think Aloud protocol, participants verbalize their thought processes while completing a task. Typically, the individual does this without assistance. However, this may not reflect how the individual would complete this task in context. Therefore, as a variant to an individual usability test including the Think Aloud method, the “peer discovery” method was suggested. In this variant, the older adult can interact with the technology together with a family member or caregiver. The idea behind this approach is that they can help each other during the Think Aloud as they would in their personal context. When peers work together they express their impressions, frustrations, and thought processes more naturally; therefore, this variant yields a clearer picture of how a technology is used and where users struggle. The peer discovery approach also increases the sense of naturalism and mitigates the sense of artificiality often associated with usability testing. In relation to this, experts mentioned the concept of “peer community” usability testing as an addition to UEM methods. In this variant the older adult can use the technology together in a group setting with other older adults, again to stimulate a more natural expression of their impressions, frustrations, and thought processes in interacting with the eHealth intervention. The idea of peer discovery and community is valuable as long as it replicates actual use in context. For example, it is unwanted for other users to assist participants with tasks when that would not be the case in the real world.

Lastly, suggestions were given on methods for usability testing of HIT when implemented in practice (post-design): shadowing, observing an older end-user of the evaluated technology in their environment for a period of time in combination with log file data via analytics software. The latter allows for a completely unobtrusive way of gathering objective data which can supplement and help interpret data from interviews and usability-tests. Triangulating methods may give a more accurate insight into people's daily interactions with the HIT intervention, as well as causal factors contributing to non-use of the HIT (see [Table 4]).

Table 4 Recommendations and checklist of key elements mentioned by experts throughout the study on the theme Suggested adjustments to UEM's for testing with older patients.

4 Discussion

The recommendations provided in this paper aim to support evaluators, healthcare professionals, decision makers, software developers and other HIT stakeholders in performing user-based usability evaluation studies of HIT interventions for older adults. These recommendations can be immediately applied to improve the design, planning, and execution phases of usability evaluations. We performed an expert inquiry wherein human factors experts shared their strategies for conducting tests of consumer-facing technologies with older adults. By sharing these experiences and best-practices, we aimed to increase awareness of aging processes influencing the quality and inclusivity of usability evaluation studies with older adults. Though some of these recommendations might apply to usability testing in other populations, we believe they are most directly applicable to evaluation studies involving older adults and can empower older adults to engage in participatory design and develop products that best meet their needs. In doing so, we aim to contribute to the scientific evidence base for HIT interventions.

4.1 Benefits of Proposed Recommendations to Redesigns

The recommendations can be applied to any HIT evaluation involving older adults throughout the user-centered design lifecycle. Insights derived from testing with older adults may improve the perceived usability and usefulness of a product by older adults [[28]]. For example, informational displays and data visualizations should be accessible to individuals with visual impairment. This will assist people with low health literacy, low digital literacy, and cognitive decline [[29] [30]]. Further, it is important to configure features and functions to best meet the needs and requirements of target users. Similarly, workflows and information flows should be simple and intuitive. For example, developers should strive to minimize the number of steps required to complete a task and ensure consistency in input/output features [[28]]. Older adults indicate they prefer clear instructions and online support to help them understand how to complete their tasks using the technology [[18], [28]]. Applying the recommendations of this paper to usability testing with older adults is likely to improve insights on how to best design these aspects of information presentation, navigational structure, data interoperability, and clarity of instructional support for older adults.

To assess the validity and completeness of our recommendations, we encourage researchers and HIT developers to report how they integrated these recommendations into their application design lifecycle and their usability tests with older adults. We recommend engaging a multi-disciplinary team; not only of human factors researchers and HIT developers, but also healthcare professionals, cognitive scientists, neuropsychologists, and geriatricians. Geriatricians and geriatric care teams (including physical rehabilitation specialists, nurses, and social workers) may be especially valuable to the design process by leveraging their expertise working with elderly patients. These specialists can help clarify the physical and cognitive impairments that affect functional status, provide in-depth knowledge on the age-related and disease-related factors that can limit HIT adoption, and suggest evidence-based and people-centered strategies to respectfully engage older adults. This is relevant to possibly expanding the identified issues of this population to usability testing and further developing the overview of recommendations to solve such issues.

4.2 Proposed Recommendations Aiding Representative Project Management

Some of the recommendations emerging from this expert inquiry have implications for project management, research design, and proctoring of usability evaluations. Experts offered a wealth of recommendations, including: (1) providing communication training to evaluators; (2) performing “pre-usability” tests; (3) hosting intake meetings; (4) conducting assessments of older adults' capacities and skills; (5) observing or shadowing the participants; and (6) having more than one test-moment with a participant. Implementing these recommendations can be time and resource intensive. Nevertheless, these issues are frequently encountered during testing, leading to higher expenses and/or delays in deliverables. Therefore, including these steps, such as holding an intake meeting to assess older adults' capacities and skills (e.g., IT literacy, functional status, quality of life), might lend deep insight into their barriers to participate (e.g., mobility issues impacting participation in location-sensitive usability tests). We believe our recommendations can inform and streamline usability testing project management, even in the earliest phases of HIT development. For example, considering these recommendations when projecting staffing resources and a research budget (e.g., reimbursement of participant and caregiver travel expenses) can improve the precision of cost estimates. Regarding the length of a project involving end-users in user-centered design processes, previous research has shown that the contribution of end-users significantly altered the ultimately designed technological intervention from the initial prototype [[31]]. Yet as a consequence of end-users' involvement, it took longer than expected to develop the intervention. Usability testing with older adults requires more time than with younger populations due to the added elements during the preparation and execution phase of the test, such as the intake meeting or performing several test sessions instead of one. Therefore, it is important to budget time and resources accordingly in the initial project proposal and at pre-defined milestones based upon the end-users insights gained through the usability tests. Paradoxically, both financial resources and development time are scarce in HIT development, implementation and evaluation projects. We nevertheless stress that investing finances and time for the involvement of older adult end-users in a user-centered design process is crucial to provide accessible and inclusive HIT solutions. Beyond ethical questions, or missed opportunities, when usability testing is not representative of potential end-users of a HIT, with respect to characteristics of individuals, their goals or their social and environmental context, there is a significant risk that all of the invested development funding will be wasted because the intervention is not scalable or sustainable due to poor adoption.

4.3 Challenges of Proposed Recommendations Related to Standardization in Usability Testing of HIT

The recommendations proposed in this paper make an important contribution to the performance of UEMs for an aging patient-consumer demographic. Benefits notwithstanding, our recommendations may be challenging to operationalize and standardize across usability testing research. By not complying with scientific requirements of standardization in HIT usability testing, it can be difficult to compare findings or establish benchmarks. More research is needed to test and validate standardized instruments and methods with an older adult population. Future work can focus upon how to empirically compare results from standardized usability test with older adults with and without incorporating these recommendations. To conclude, in the near term we believe it is important to strike a balance between scientific constraints of usability testing and barriers in older adults' participation.

5 Conclusion

A comprehensive set of nine recommendations and a checklist of 37 elements to support the application of these recommendations in usability test evaluations of HIT with older adults have been developed by means of experts' insights. Further, the results of usability tests will become more robust when these recommendations are applied, contributing to an important step towards evidence-based, inclusive and accessible HIT for vulnerable older adults.

• References

• 1 Gilbert B, Goodman E, Chadda A, Hatfield D, Forman D, Panch T. The Role of Mobile Health in Elderly Populations. Curr Geriatr Rep 2015;4(4):347-52.
• 2 Kim BY, Lee J. Smart Devices for Older Adults Managing Chronic Disease: A Scoping Review. JMIR Mhealth Uhealth 2017 May 23;5(5):e69.
• 3 Hamm J, Money A, Atwal A. Fall Prevention Self-Assessments Via Mobile 3D Visualization Technologies: Community Dwelling Older Adults' Perceptions of Opportunities and Challenges. JMIR Hum Factors 2017 Jun 19;4(2):e15.
• 4 Joe J, Demiris G. Older adults and mobile phones for health: a review. J Biomed Inform 2013 Oct;46(5):947-54.
• 5 Morey SA, Barg-Walkow LH, Rogers WA. Managing Heart Failure On the Go: Usability Issues with mHealth Apps for Older Adults. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 2017;61(1):1-5.
• 6 Levac D, Colquhoun H, O'Brien KK. Scoping studies: advancing the methodology. Implement Sci 2010;5:69.
• 7 Matthew-Maich N, Harris L, Ploeg J, Markle-Reid M, Valaitis R, Ibrahim S, et al. Designing, Implementing, and Evaluating Mobile Health Technologies for Managing Chronic Conditions in Older Adults: A Scoping Review. JMIR Mhealth Uhealth 2016 Jun 9;4(2):e29.
• 8 Barakat A, Woolrych RD, Sixsmith A, Kearns WD, Kort HS. eHealth Technology Competencies for Health Professionals Working in Home Care to Support Older Adults to Age in Place: Outcomes of a Two-Day Collaborative Workshop. Med 2 0. 2013 Sep 5;2(2):e10.
• 9 Nielsen JA, Mengiste SA. Analyzing the diffusion and adoption of mobile IT across social worlds. Health Informatics J 2014;20: 87”103.
• 10 Chan KC, Wong L, Chan DB. Design of a large scale community-based self-management system for diabetes mellitus. Stud Health Technol Inform 2012;182:58-66.
• 11 Öberg U, Isaksson U, Jutterström L, Orre CJ, Hörnsten Å. Perceptions of Persons With Type 2 Diabetes Treated in Swedish Primary Health Care: Qualitative Study on Using eHealth Services for Self-Management Support. JMIR Diabetes 2018;3(1):e7.
• 12 Dale LP, Whittaker R, Eyles H, Mhurchu CN, Ball K, Smith N, et al. Cardiovascular Disease Self-Management: Pilot Testing of an mHealth Healthy Eating Program. J Pers Med 2014;4(1):88-101.
• 13 Huang Y, Hsu Y. Social networking-based personal home telehealth system: A pilot study. J Clin Gerontol Geriatr 2014 Dec;5(4):132”9.
• 14 Brooke J. SUS-A quick and dirty usability scale. Usability Evaluation Industry 1996;189(194):4”7.
• 15 Sauro J, Dumas JS. Comparison of three one-question, post-task usability questionnaires. In : Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2009:1599”608.
• 16 Hong Y, Goldberg D, Dahlke DV, Ory MG, Cargill JS, Coughlin R, et al. Testing Usability and Acceptability of a Web Application to Promote Physical Activity (iCanFit) Among Older Adults. JMIR Hum Factors 2014 Oct 13;1(1):e2.
• 17 Wildenbos GA, Jaspers M, Peute L. The equity paradox: older patients' participation in patient portal development. Int J Qual Health Care 2019 Dec 31;31(10):793-7.
• 18 Wildenbos GA, Jaspers MWM, Schijven MP, Dusseljee-Peute LW. Mobile health for older adult patients: Using an aging barriers framework to classify usability problems. Int J Med Inform 2019 Apr;124:68-77.
• 19 Engelsma T, Jaspers MWM, Peute LW. Considerate mHealth design for older adults with Alzheimer's disease and related dementias (ADRD): A scoping review on usability barriers and design suggestions. Int J Med Inform 2021 Aug;152:104494.
• 20 Wildenbos GA, Peute L, Jaspers M. Aging barriers influencing mobile health usability for older adults: A literature based framework (MOLD-US). Int J Med Inform 2018 Jun;114:66-75.
• 21 Jaspers MW, Steen T, van den Bos C, Geenen M. The think aloud method: a guide to user interface design. Int J Med Inform 2004 Nov;73(11-12):781-95.
• 22 International Standard Organization for Standardization. ISO 21801-1:2020. _Cognitive accessibility - Part 1: General guidelines.
• 23 International Standard Organization for Standardization. ISO/TS 82304-2:2021. _Health software - Part 2: Health and wellness apps ' Quality and reliability.
• 24 Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, Hoagwood K. Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research. Adm Policy Ment Health 2015;42(5):533-44.
• 25 International Standard Organization for Standardization. ISO 9241-210:2010. _Ergonomics of human-system interaction ' Part 210: Human-centred design for interactive systems.
• 26 Coleman C. Health Literacy and Clear Communication Best Practices for Telemedicine. Health Lit Res Pract 2020;4(4):e224-e229.
• 27 Murphy K, Jordan F, Hunter A, Cooney A, Casey D. Articulating the strategies for maximising the inclusion of people with dementia in qualitative research studies. Dementia (London) 2015 Nov;14(6):800-24.
• 28 Czaja S. Designing for Older Adults. 3rd ed. Milton, United Kingdom: CRC Press LLC; 2019.
• 29 Centers for disease control and prevention. Older Adults- Are you communicating effectively with older adults? [cited 2021 Nov 25] Available from: https://www.cdc.gov/healthliteracy/developmaterials/audiences/olderadults/index.html
• 30 Geboers B, Uiters E, Reijneveld SA, Jansen CJM, Almansa J, Nooyens ACJ, et al. Health literacy among older adults is associated with their 10-years' cognitive functioning and decline - the Doetinchem Cohort Study. BMC Geriatr 2018 Mar 20;18(1):77.
• 31 Colquhoun HL, Sattler D, Chan C, Walji T, Palumbo R, Chalmers I. Applying User-Centered Design to Develop an Audit and Feedback Intervention for the Home Care Sector. Home Health Care Manag Pract 2017;29(3):148”60.
• 32 Hornbaek K. Current practice in measuring usability: Challenges to usability studies and research. Int J Hum Comput Stud 2006;64(2):79”102.