Expanding Human Computer Interaction Methods to Understand User Needs in the Design Process of Personal Health Systems

Summary

Objectives: To review innovative human computer interaction methods researchers utilize to identify stakeholders' needs that inform the design of personal health systems outside of clinical environments.

Methods: A selective review of recent literature.

Results: Summaries of exemplar needs analysis papers showing how researchers utilize novel methods to surface the lived experiences of users.

Conclusions: The medical informatics community is encouraged to ensure that we are designing health technology for all individuals - including underrepresented and underserved populations — by investigating the complex needs of target users. This paper summarizes the novel ways researchers have explored target populations via social media and engaged populations as part of the design team. Medical informaticians should continue investigating the soundness of these methods by comparing the design outcomes with currently utilized user-centered methods and to report on unintended consequences

Keywords

User-computer interface - needs assessment - social media - participatory design - usability methods

1 Introduction

The aim of this paper is to highlight how researchers utilize novel methods to identify the sociotechnical needs of people outside of clinical environments as they navigate the tensions of managing their health and life. Medical informatics researchers have noted the importance of blending human computer interaction methods with medical informatics research to ensure that people can proactively participate to their personal health management that often happens outside of clinical environments[1] [2] [3]. By default, we typically design and build systems for ourselves, which leads to one-size-fits-some systems and a high abandonment rate[4] [5] [6]. A recent computing research visioning report on smart and pervasive health challenged researchers and funders to be committed to engaging with diverse stakeholders – especially those at risk of health disparities – to improve the generalizability of their research[7]. Thus, researchers and practitioners should develop a rich understanding of diverse stakeholders’ needs before designing, developing, and deploying a system to address a given health issue.

We summarize current literature that explores novel methods on how needs are identified. The papers selected go beyond traditional user-centered design methods - such as in-person observations, interviews, and inspection methods (e.g.,[8] [9]) - to show how researchers in different fields utilize new methods to give voice to underserved and sometimes stigmatized groups. The selected studies illustrate how researchers explore a target population's needs by analyzing: (1) how users utilize commodity technology and (2) how users engage as design team members.

2 Methods

The studies were selected by searching PubMed/Medline, Google Scholar, ACM, and IEEE search engines for research published in the last three years where researchers reported on novel user need assessment and design methods to explore lay population needs outside of clinical environments. We chose an exploratory, selective approach, instead of a comprehensive systematic review because researchers in the different fields classify their publications quite differently. For example in health informatics, researchers who work with people who manage their personal health utilize keywords such as “patient” or “consumer”[10]. However, researchers in human computer interaction rarely utilize these terms – instead they utilize keywords that note the action (e.g., self tracking, photo sharing, blogging), design method (e.g., participatory design, mixed methods), target population (e.g., family, HIV, eating disorder, children, chronic illness), or technology (e.g., social media, online health communities, Twitter). In addition, we respected each field's publication culture in terms of what articles to review. For example, in computing-oriented human computer interaction disciplines, conference publications are highly regarded and full papers are submitted for peer review[11]. To this end, we reviewed top-tier conferences (<25% acceptance rate) in the last three years that have established program committee subcommittees for health-related papers and reviewed each full paper title for health-oriented topics (e.g., target population), then reviewed the abstract of health-oriented papers based on those titles identified, and all the abstracts in health-related conference sessions to identify papers that dealt with identifying user needs outside of clinical environments. We then followed backward and forward citation trees to identify relevant literature. For each paper, when available, we detailed the design methods utilized, sample size, and duration of the study to provide the community with an understanding of how methods, context, and analysis vary.

3 Results

Sociotechnical researchers develop an understanding of people's needs through observations and contact with target populations. In some cases, co-located methods (where researchers and participants are physically present together) are not possible because of access, mobility, physical, social, political, and economic constraints. In addition, researchers may not be in a location with access to the target population – for example when a research team with limited resources needs to accommodate the perspective of people with rare diseases in a health system design, since only 10% of the world's population has a rare disease, the research team must utilize alternative methods to gain their perspective[12]. We found that sociotechnical researchers are overcoming these barriers to gain a deeper sense of people's needs by studying how people currently appropriate social media to discuss their health. Conversely, a research team may have access to the target population (e.g., underserved communities), but because of some challenges (e.g., power differential, especially in cases where researchers are perceived as the experts and participants defer to the researchers’ ideas when participants are experts in their own experience; as well as past negative experiences), researchers want to empower and democratize the design process. In these cases, we found a burgeoning group of researchers engaging target users in participatory or co-design methods.

3.1 Appropriating Mainstream Technology

When technology does not meet one's needs, people typically abandon it[4] - especially in assistive technology[5] [6] - and they re-appropriate technology that works for them to address their health needs[5]. More recently, we have seen people appropriate mainstream technology to amplify their voice[13] [14] [15] [16], share their experiences, and find a community[13] [14] [17] [18], but not always in the best interest of their health[15]. We acknowledge that there has been much research on social media related to health[19] [20] [21], however we focus here on qualitative analysis of social media that informs researchers about a target population's lived experience, rather than predicting future implications.

A positive example of utilizing mainstream technology to inform the design of personal health systems is highlighted by Lazar et al.[13] who qualitatively analyzed 189 older adult bloggers making 219,654 posts over a 12-year time period to show how ageism plays in their everyday lives and their interactions with the world. The authors draw on the discourse of an online collective action among older adults as a way of understanding what ageism means for design. The results remind researchers that underrepresented groups in sociotechnical system design, such as older adults, have valuable, creative insights that should be taken into account in the design process so future technologies are not constrained by the design team's possible misconceptions (e.g., medication management[22] [23]). Another positive example from Liu et al. investigated how 36 people with chronic illnesses, such as HIV, cancer, or diabetes, utilized YouTube videos as a way to share information on their illness (e.g., medication regimes) and to share their personal experiences (e.g., emotions surrounding diagnoses)[24]. They found that people used YouTube commenting features to receive and provide social support to others who may be going through similar experiences[25]. This research builds on the rich medical informatics literature[26] [27] [28], investigating how people build community through online health forums by surfacing community building in a new, unstructured, public medium. These methods relying on mainstream technologies appear to be promising as a means to explore complex needs of target users through their lived experience.

But results can be mixed as to the value of this type of technology to gather users’ needs. A neutral example is from Andalibi et al. who analyzed 788 images posted on Instagram over one month and 1,949 comments associated with 444 posts related to depression. They found that Instagram users, as opposed to users’ behavior on other social media, disclosed stigmatizing, negative experiences and found support in doing so[14]. They identified possibly negative consequences because they found that some Instagram users thought that their accounts were “secret” even though they were publicly viewable and people sometimes posted images of themselves; thus, they were only secret in that Instagram users may not have actively let others know they had this Instagram account. This research is a reminder that people may present themselves differently on various social media platforms. It is also a reminder to explore target users’ mental models of how the technology platform functions to mitigate any unintended consequences of their participation.

Finally, an example of appropriating technology for a negative health impact is from Pater et al.[15] who presented a qualitative analysis of 575 social media posts (e.g., Twitter, Instagram, and Tumblr) over 12 days to investigate how people use the different platforms to build a community and communicate about eating disorders. They identified how user communities avoided automated censorship to “hide in plain sight” by changing hashtags so that they could continue sharing their motivating or negative health impact messaging. We highlight this research to contrast it with the other studies highlighted here and with some of the more qualitative social media medical informatics literature that note the positive effects of community building from sharing information on social media[19]. We encourage sociotechnical researchers to consider how social media may facilitate building communities that promote negative health behaviors which could impact further intervention or application design.

In addition to qualitatively analyzing the data that users generate on social media, researchers have also investigated ways to engage target populations on social media via carefully scaffold interactions to better understand one's needs, similar to an online focus group. MacLeod et al. developed the Asynchronous Remote Communities (ARC) method that builds on remote needs assessment methods, which has included instant messaging[29], email[30], phone[30], and video[31] interviewing methods. They qualitatively investigated the needs of people with rare diseases by engaging 11 participants in a 22-week study on a private Facebook group[12] [16] [17]. The researchers found that online communities provided invaluable support, but participants needed more tangible support. Prabhakar et al. utilized ARC to understand the needs of 10 first-time pregnant women, 20 pregnant mothers with children, and 18 new mothers via a secret Facebook group[18] [32]. They found that pregnant women and new mothers considered their partner as the primary source of support, however their instrumental support needs and the people they turned to for assistance changed after they had their baby, thus other forms of social connectedness were needed[32]. Similar to qualitatively analyzing social media, ARC provided researchers with the ability to better understand underrepresented groups of people who are not geographically co-located (e.g., people with rare diseases) or have time and personal constraints (e.g., women with infants). However, unlike other social media research methods, in ARC-based studies, researchers can guide the conversation and participants are aware that they are part of a study (which could influence their behavior).

3.2 Engaging Stakeholders in the Design Process of Health Systems

We also explored how sociotechnical researchers in health have engaged target populations as partners in the design process of health systems and apps. Although participatory design has been around in various forms (e.g., co-design, action research) since the 1970s[33], the human-computer interaction community started utilizing it to design computing systems with people in the 1990s[34], and medical informatics was introduced to participatory design practices to design clinical systems in 2000[35]. In human computer interaction, researchers have used participatory design methods in health domains to design systems within[36] and outside the clinical environment, whereas medical informatics is beginning to utilize methods in clinical settings[37]. Here we highlight how sociotechnical researchers utilize participatory design methodologies to better understand lay populations outside of clinical settings.

The participatory design process to create TreatYoSelf, a mobile app that can help young people with HIV adhere to their medication regime, is notable because the design team included a patient, a clinician, and a designer who facilitated three design workshops with a total of 13 young people with HIV to ultimately design a minimal gamified system that engendered community through motivational, nonjudgmental, and “corny” messaging[38]. This research provides insights into how participatory design could make intervention messaging and the resulting application more relatable depending on the specific population a research group is working with.

Pina et al.[39] explored family-based tracking for relatively healthy families and families with a child suffering from a chronic condition by interviewing 24 families and conducting participatory design sessions with children. Overall, they found tracking needs to accommodate the ability of one to track for himself/herself and others, while also providing support to limit what others see to protect one's privacy. Similar to TreatYoSelf, privacy was linked to judgment and participants did not want to be judged for their occasional indulgences, which could be construed as non-adherence. In this case, participatory design provides insight into the complexity of people's lives, starting with the understanding that sociotechnical communities’ current practices of design for managing one person's health are not realistic in a family setting and that adherence must be balanced with an individual's goals at that moment.

One approach to participatory design is described by Barry et al.[40] who used phronesis to design a mobile application where new mothers can log their current psychological mental health status. The concept of phronesis comes from Aristotelian virtue ethics. It describes wisdom and judgment garnered from practical experience of specific situations in context. In applying phronesis, design team participants which included everyone from mothers to healthcare professionals, and to researchers, acknowledge their contextual experiences and thereby also note their subjective value systems. Once the design team shares their knowledge and value systems, they can collectively identify how values should be prioritized in the design and functionality of the application. We highlight this article because a common struggle in participatory design research is balancing the goal of full participant engagement in the design process while mitigating power differentials, especially in disempowered or underserved communities[33] [41]. Thus, this research provides a methodical way of sharing values, goals, and expertise to make a more cohesive design team.

4 Discussion

In this selective review, we summarized novel techniques that sociotechnical researchers utilize to understand the needs of people outside clinical environments, from passive, qualitative analysis of social media interactions, to directed methods that engage people via social media, to co-located, collaborative methods that expose the complexity and context of people's experiences. We focus on these novel methods to encourage researchers to broaden their knowledge of how data is collected from target populations. However, we also acknowledge that more research must be done to explore the limitations of these methods and understand how successful they may be at informing the design of sociotechnical systems. We see a space for comparative studies to explore results yielded from ethnographic observations versus user-centered techniques (e.g., interviews and design workshops), versus qualitative social media analysis. Likewise, we have not explored the differences that user-centered versus participatory design methods yield in the design and deployment of a sociotechnical system. When considering people's lack of long term use in sociotechnical systems to manage one's health[4], we believe this would be a fruitful area of research to see if claims of increased uptake result from one's participation in the system design[41] [42].

A critique of sociotechnical researchers’ work to identify the needs of underserved communities is that they investigate fringe groups. However, given the lack of diversity on technology development teams[40] and documented racial bias in the computing industry[43] [44], we set the stage to unintentionally or intentionally leave them out, or worse, design systems that are biased against them. Indeed, a 2017 Computing Community Consortium workshop report on smart and pervasive health noted the pressing needs to address underrepresented groups that are adversely impacted by health disparities[7]. Health-oriented sociotechnical researchers have the potential to work together, such as at the Workshop on Interactive Systems in Healthcare (WISH)[45], to design, build, and deploy systems that iteratively check on their biases, dismantle them, and evaluate the success of methodologies that lead to more inclusive system design. To do this, we need researchers to continue seeking opportunities to work together, share research findings, and improve each other's work.

Despite much of the research summarized here utilizing sharing mechanisms, either within a family[39] or broadly within a private social media group[17] [18], few studies discussed the implications of privacy and possible concerns with sharing data[14] [15]. Although we acknowledge that the sociotechnical research community has discussed the possible implications of sharing data, especially with employee wellness programs and possible insurance program penalties[46], a challenge to future health-oriented human computer interaction research would be to evaluate privacy concerns and their implications. Given that health data privacy is highly sensitive and can prompt emotional and stigmatizing feelings, we recommend that researchers consider unintended consequences[19] and report on these in their publications. In addition, we encourage the community to consider that concerns about privacy could bias answers related to privacy, and that researchers should be aware of work that reported on an indirect mechanism for evaluating privacy[47].

5 Conclusion

We need to consider the complex and messy context of people's lives to design appropriate sociotechnical interventions. Sociotechnical researchers have a long history of utilizing co-located ethnographic methods and co-located and remote user-centered methodologies to identify user needs. However, with new social media platforms and the need to build systems for more diverse populations, we need to consider novel methods to ensure that context is taken into account in the research design and analysis. In this paper, we summarize novel ways researchers have explored the context of target populations via social media and how they engaged populations as part of the design team. We encourage the community to investigate the strengths and limitations of these methods with comparative studies while considering the unintended consequences of sharing data with these methods.

• References

• 1 Lai AM, Siek KA. Designing and Deploying Mobile Health Interventions. In: Patel V, Kannampallil T, Kaufman D. , editors. Cognitive Informatics for Biomedicine: Human Computer Interaction in Healthcare. Springer; 2015. . p. 279–306
• 2 Jimison HB, Pavel M, Parker A, Mainello K. The Role of Human Computer Interaction in Consumer Health Applications: Current State, Challenges and the Future. In: Patel V, Kannampallil T, Kaufman D. , editors. Cognitive Informatics for Biomedicine: Human Computer Interaction in Healthcare. Springer; 2015. . p. 259–78
• 3 Wachter RM. Perspectives on Safety Published November 2017 In Conversation With. . . Wanda Pratt, PhD; 2017. Podcast: https://psnet.ahrq.gov/perspectives/perspective/236
• 4 Clawson J, Pater JA, Miller AD, Mynatt ED, Mamykina L. No Longer Wearing: Investigating the Abandonment of Personal Health-tracking Technologies on Craigslist. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. UbiComp ’15. New York, NY, USA: ACM; 2015. p. 647–58. Available from: http://doi.acm.rg/10.1145/2750858.2807554
• 5 Dawe M. Desperately seeking simplicity: how young adults with cognitive disabilities and their families adopt assistive technologies. In: Grinter R, Rodden T, Aoki P, Cutrell E, Jeffries R, Olson G. , editors. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, NY: ACM; 2006. . p. 1143–52
• 6 Lazar A, Koehler C, Tanenbaum J, Nguyen DH. Why We Use and Abandon Smart Devices. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing UbiComp ’15. New York, NY, USA: ACM; 2015. p. 635–46. Available from: http://doi.acm.rg/10.1145/2750858.2804288
• 7 Myantt E, Hager GD, Kumar S, Lin M, Patel S, Stankovic J. , et al. Research Opportunities and Visions for Smart and Pervasive Health. Computing Community Cosortium; 2017. https://cra.org/ccc/wp-content/uploads/sites/2/2017/06/Smartand-PervasiveHealth-White-Paper-June-2017.pdf
• 8 Nielsen J, Molich R. Heuristic Evaluation of User Interfaces. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’90. New York, NY, USA: ACM; 1990. . p. 249–56. Available from: http://doi.acm.org/10.1145/97243.97281
• 9 Savoy A, Patel H, Flanagan M, Weiner M, Russ A. Systematic Heuristic Evaluation of Computerized Consultation Order Templates: Clinicians’ and Human Factors Engineers’ Perspectives. 2017; 41 (08) 1-9 . Available from: http://dx.doi.org/10.1007/s10916-017-0775-7
• 10 Lai AM, Hsueh PYS, Choi YK, Austin RR. Present and Future Trends in Consumer Health Informatics and Patient-Generated Health Data. Yearb Med Inform 2017; Aug; 26 (01) 152-9 . Available from: http://view.ncbi.nlm.nih.gov/pubmed/29063559
• 11 Patterson D, Snyder L, Ullman J. Best Practices Memo: Evaluating Computer Scientists and Engineers for Promotion and Tenure; 1999 September
• 12 MacLeod H, Jelen B, Prabhakar A, Oehlberg L, Siek K, Connelly K. A Guide to Using Asynchronous Remote Communities (ARC) for Researching Distributed Populations. EAI Endorsed Transactions, Special Issue on Pervasive Health 2016; 2017
• 13 Lazar A, Diaz M, Brewer R, Kim C, Piper AM. Going Gray, Failure to Hire, and the Ick Factor: Analyzing How Older Bloggers Talk About Ageism. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. CSCW ’17. New York, NY, USA: ACM; 2017. . p. 655–68. Available from: http://doi.acm.rg/10.1145/2998181.2998275
• 14 Andalibi N, Ozturk P, Forte A. Sensitive Self-disclosures, Responses, and Social Support on Instagram: The Case of #Depression. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. CSCW ’17. New York, NY, USA: ACM; 2017. . p. 1485–500. Available from: http://doi.acm.rg/10.1145/2998181.2998243
• 15 Pater JA, Haimson OL, Andalibi N, Mynatt ED. “Hunger Hurts but Starving Works”: Characterizing the Presentation of Eating Disorders Online. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. CSCW ’16. New York, NY, USA: ACM; 2016. . p. 1185–200. Available from: http://doi.acm.rg/10.1145/2818048.2820030
• 16 MacLeod H, Bastin G, Liu LS, Siek K, Connelly K. “Be Grateful You Don’T Have a Real Disease”: Understanding Rare Disease Relationships. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. CHI ’17. New York, NY, USA: ACM; 2017. . p. 1660–73. Available from: http://doi.acm.rg/10.1145/3025453.3025796
• 17 MacLeod H, Jelen B, Prabhakar A, Oehlberg L, Siek K, Connelly K. Asynchronous Remote Communities (ARC) for Researching Distributed Populations. In: Proceedings of the 10th EAI International Conference on Pervasive Computing Technologies for Healthcare. PervasiveHealth ’16. ICST, Brussels, Belgium, Belgium: ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering); 2016. p. 1–8. Acceptance Rate: 35%. Available from: http://dl.acm.org/citation.cfm?id=3021319.3021320
• 18 Prabhakar AS, Guerra-Reyes L, Kleinschmidt VM, Jelen B, MacLeod H, Connelly K. , et al. Investigating the Suitability of the Asynchronous, Remote, Community-based Method for Pregnant and New Mothers. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. CHI ’17. New York, NY, USA: ACM; 2017. . p. 4924–34. Available from: http://doi.acm.rg/10.1145/3025453.3025546
• 19 Hors-Fraile S, Atique S, Mayer MA, Denecke K, Merolli M, Househ M. The Unintended Consequences of Social Media in Healthcare: New Problems and New Solutions. Yearb Med Inform 2016; Nov; 1: 47-52 . Available from: http://view.ncbi.nlm.nih.gov/pubmed/27830230
• 20 Choudhury MD, Kiciman E. The Language of Social Support in Social Media and its Effect on Suicidal Ideation Risk. In: Proceedings of the International Conference on Web and Social Media (ICWSM). AAAI; 2017. Available from: https://www.microsoft.com/en-us/research/publication/language-social-support-social-media-effect-suicidal-ideation-risk/
• 21 Park A, Hartzler AL, Huh J, Hsieh G, McDonald DW, Pratt W. “How Did We Get Here?”: Topic Drift in Online Health Discussions. J Medical Internet Res 2016; Nov; 18 (11) e284 . Available from: http://view.ncbi.nlm.nih.gov/pubmed/27806924
• 22 Palen L, Aaløkke S. Of Pill Boxes and Piano Benches: ”Home- made” Methods for Managing Medication. In: Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work. CSCW ’06. New York, NY, USA: ACM; 2006. . p. 79–88. Available from: http://doi.im./10.1145/1180875.1180888
• 23 Lee ML, Dey AK. Real-time Feedback for Improving Medication Taking. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’14. New York, NY, USA: ACM; 2014. . p. 2259–68. Available from: http://doi.acm.org/10.1145/2556288.2557210
• 24 Liu LS, Huh J, Neogi T, Inkpen K, Pratt W. Health Vlogger-viewer Interaction in Chronic Illness Management. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’13. New York, NY, USA: ACM; 2013. . p. 49–58. Available from: http://doi.acm.org/10.1145/2470654.2470663
• 25 Huh J, Liu LS, Neogi T, Inkpen K, Pratt W. Health Vlogs As Social Support for Chronic Illness Management. ACM Trans Comput-Hum Interact 2014; Aug; 21 (04) 23:1-23:31 . Available from: http://doi.acm.org/10.1145/2630067
• 26 Meier A, Lyons EJ, Frydman G, Forlenza M, Rimer BK. How cancer survivors provide support on cancer-related Internet mailing lists. . J Med Internet Res 2007 May;9(2). Available from: http://view.ncbi.nlm.nih.gov/pubmed/17513283
• 27 Setoyama Y, Yamazaki Y, Namayama K. Benefits of peer support in online Japanese breast cancer communities: differences between lurkers and posters. . J Med Internet Res 2011 Dec;13(4). Available from: http://view.ncbi.nlm.nih.gov/pubmed/22204869
• 28 Lasker JN, Sogolow ED, Sharim RR. The role of an online community for people with a rare disease: content analysis of messages posted on a primary biliary cirrhosis mailinglist. J Med Internet Res 2005; Mar; 7 (01) e10 . Available from: http://dx.doi.org/10.2196/jmir.7.1.e10
• 29 Voida A, Mynatt ED, Erickson T, Kellog WA. Interviewing Over Instant Messaging. In: CHI ’04 EA. ACM; 2004. p. 1344–7
• 30 Dimond JP, Fiesler C, DiSalvo B, Pelc J, Bruckman AS. Qualitative Data Collection Technologies: A Comparison of Instant Messaging, Email, and Phone. In: GROUP ’12. ACM; 2012. p. 277–80
• 31 Hillman S, Forghani A, Pang C, Neustaeder C, Judge TK. Conducting Interviews with Remote Participants. In: Studying and Designing Technology for Domestic Life. Morgan Kaufmann; 2015. p. 11–53
• 32 Prabhakar A, Guerra-Reyes L, Effron A, Kleinschmidt VM, Driscoll M. , et al. “Let Me Know If You Need Anything”: Support Realities of New Mothers. In: Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth ‘17). New York, NY, USA: ACM; 2017. . p.31-40. Available from: https://doi.org/10.1145/3154862.3154863
• 33 Simonsen J, Robertson T. Routledge International Handbook of Participatory Design. 1st ed. New York, NY: Routledge; 2013
• 34 Muller MJ. The Human-computer Interaction Handbook. Hillsdale, NJ, USA: L. Erlbaum Associates Inc.; 2003. . p. 1051–68. Available from: http://dl.acm.org/citation.cfm?id=772072.772138
• 35 Gennari JH, Reddy M. Participatory design and an eligibility screening tool. Proceedings AMIA Symposium; 2000. p. 290–4. Available from: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2243801/
• 36 Kusunoki D, Sarcevic A, Zhang Z, Yala M. Sketching Awareness: A Participatory Study to Elicit Designs for Supporting Ad Hoc Emergency Medical Teamwork. Comput Supported Coop Work 2015; Feb; 24 (01) 1-38 . Available from: http://dx.doi.org/10.1007/s10606-014-9210-5
• 37 Pollack AH, Miller A, Mishra SR, Pratt W. PD-atricians: Leveraging Physicians and Participatory Design to Develop Novel Clinical Information Tools. AMIA Annual Symposium proceedings AMIA Symposium 2016; 2016: 1030-9 . Available from: http://view.ncbi.nlm.nih.gov/pubmed/28269900
• 38 Marcu G, Dowshen N, Saha S, Sarreal RR, Andalibi N. TreatYoSelf: Empathy-driven Behavioral Intervention for Marginalized Youth Living with HIV. In: Proceedings of the 10th EAI International Conference on Pervasive Computing Technologies for Healthcare. PervasiveHealth ’16. ICST, Brussels, Belgium, Belgium: ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering); 2016. . p. 69–76. Available from: http://dl.acm.org/citation.cfm?id=3021319.3021330
• 39 Pina LR, Sien SW, Ward T, Yip JC, Munson SA, Fogarty J. , et al. From Personal Informatics to Family Informatics: Understanding Family Practices Around Health Monitoring. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. CSCW ’17. New York, NY, USA: ACM; 2017. . p. 2300–15. Available from: http://doi.acm.org/10.1145/2998181.2998362
• 40 Barry M, Doherty K, Marcano Belisario J, Car J, Morrison C, Doherty G. mHealth for Maternal Mental Health: Everyday Wisdom in Ethical Design. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. CHI ’17. New York, NY, USA: ACM; 2017. . p. 2708–56. Available from: http://doi.acm.org/10.1145/3025453.3025918
• 41 Unertl KM, Schaefbauer CL, Campbell TR, Senteio C, Siek KA, Bakken S. , et al. Integrating Community-Based Participatory Research and Informatics Approaches to Improve the Engagement and Health of Underserved Populations. J Am Med Inform Assoc 2016; 23 (01) 60-73 . Available from: http://dx.doi.org/10.1093/jamia/ocv094
• 42 Norton MI, Mochon D, Ariely D. The ‘IKEA Effect’: When Labor Leads to Love. Harvard Business School Marketing Unit; 2011;11(091)
• 43 Garvie C, Frankle J. Facial-Recognition Software Might Have a Racial Bias Problem. In: The Atlantic. https://www.theatlantic.com/technology/archive/2016/04/the-underlying-bias-of-facial-recognition-systems/476991/ ; 2016. Accessed: 28 December 2017
• 44 Sweeney L. Discrimination in online ad delivery. Commun ACM 2013; 56 (05) 44-54 . Available from: http://doi.acm.rg/10.1145/2447976.2447990
• 45 Kientz JA, Huh J, Marcu G, Pina L. Workshop on Interactive Systems in Healthcare (WISH). In: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. CHI EA ’17. New York, NY, USA: ACM; 2017. . p. 3059–64. Available from: http://doi.acm.org/10.1145/3027063.3108900
• 46 Christophersen M, Mørck P, Langhoff TO, Bjørn P. Unforeseen Challenges - Adopting Wearable Health Data Tracking Devices to Reduce Health Insurance Costs in Organizations. In: Universal Access in Human- Computer Interaction. Access to Learning, Health and Well-Being - 9th International Conference, UAHCI 2015, Held as Part of HCI International 2015, Los Angeles, CA, USA, August 2-7, 2015, Proceedings, Part III; 2015. p. 288–99. Available from: https://doi.org/10.1007/978-3-319-20684-428
• 47 Braunstein A, Granka L, Staddon J. Indirect Content Privacy Surveys: Measuring Privacy Without Asking About It. In: Proceedings of the Seventh Symposium on Usable Privacy and Security. SOUPS ’11. New York, NY, USA: ACM; 2011. . p. 15:1–15:14. Available from: http://doi.acm.org/10.1145/2078827.2078847