Implementation Fidelity of Chatbot Screening for Social Needs: Acceptability, Feasibility, Appropriateness

Raina Langevin; Andrew B. L. Berry; Jinyang Zhang; Callan E. Fockele; Layla Anderson; Dennis Hsieh; Andrea Hartzler; Herbert C. Duber; Gary Hsieh

doi:10.1055/a-2035-5342

Applied Clinical Informatics, Inhaltsverzeichnis

Appl Clin Inform 2023; 14(02): 374-391
DOI: 10.1055/a-2035-5342

Research Article

Implementation Fidelity of Chatbot Screening for Social Needs: Acceptability, Feasibility, Appropriateness

Raina Langevin

¹Department of Human Centered Design and Engineering, University of Washington, Seattle, Washington, United States

,

Andrew B. L. Berry

²Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern University, Chicago, Illinois, United States

,

Jinyang Zhang

³Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, Washington, United States

,

Callan E. Fockele

⁴Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, United States

,

Layla Anderson

⁴Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, United States

,

Dennis Hsieh

⁵Department of Emergency Medicine, Harbor-University of California Los Angeles Medical Center, Torrance, California, United States

,

Andrea Hartzler

⁶Biomedical Informatics and Medical Education, University of Washington, Seattle, Washington, United States

,

Herbert C. Duber

⁴Department of Emergency Medicine, University of Washington School of Medicine, Seattle, Washington, United States

⁷Office of Health and Science, Washington State Department of Health, Seattle, Washington, United States

,

Gary Hsieh

¹Department of Human Centered Design and Engineering, University of Washington, Seattle, Washington, United States

› Institutsangaben

Abstract

Objectives Patient and provider-facing screening tools for social determinants of health have been explored in a variety of contexts; however, effective screening and resource referral remain challenging, and less is known about how patients perceive chatbots as potential social needs screening tools. We investigated patient perceptions of a chatbot for social needs screening using three implementation outcome measures: acceptability, feasibility, and appropriateness.

Methods We implemented a chatbot for social needs screening at one large public hospital emergency department (ED) and used concurrent triangulation to assess perceptions of the chatbot use for screening. A total of 350 ED visitors completed the social needs screening and rated the chatbot on implementation outcome measures, and 22 participants engaged in follow-up phone interviews.

Results The screened participants ranged in age from 18 to 90 years old and were diverse in race/ethnicity, education, and insurance status. Participants (n = 350) rated the chatbot as an acceptable, feasible, and appropriate way of screening. Through interviews (n = 22), participants explained that the chatbot was a responsive, private, easy to use, efficient, and comfortable channel to report social needs in the ED, but wanted more information on data use and more support in accessing resources.

Conclusion In this study, we deployed a chatbot for social needs screening in a real-world context and found patients perceived the chatbot to be an acceptable, feasible, and appropriate modality for social needs screening. Findings suggest that chatbots are a promising modality for social needs screening and can successfully engage a large, diverse patient population in the ED. This is significant, as it suggests that chatbots could facilitate a screening process that ultimately connects patients to care for social needs, improving health and well-being for members of vulnerable patient populations.

Keywords

patients - chatbot interaction - implementation - digital - social determinants of health

Volltext

Referenzen

References
1 Marmot M. Social determinants of health inequalities. Lancet 2005; 365 (9464): 1099-1104
2 Gottlieb LM, Wing H, Adler NE. A systematic review of interventions on patients' social and economic needs. Am J Prev Med 2017; 53 (05) 719-729
3 Adler NE, Glymour MM, Fielding J. Addressing social determinants of health and health inequalities. JAMA 2016; 316 (16) 1641-1642
4 Marmot M, Friel S, Bell R, Houweling TA, Taylor S. ; Commission on Social Determinants of Health. Closing the gap in a generation: health equity through action on the social determinants of health. Lancet 2008; 372 (9650): 1661-1669
5 Sulo S, Feldstein J, Partridge J, Schwander B, Sriram K, Summerfelt WT. Budget impact of a comprehensive nutrition-focused quality improvement program for malnourished hospitalized patients. Am Health Drug Benefits 2017; 10 (05) 262-270
6 Castrucci B, Auerbach J. Meeting individual social needs falls short of addressing social determinants of health. Health Affairs Blog 2019 Accessed March 2, 2023 at: https://www.healthaffairs.org/do/10.1377/forefront.20190115.234942/
7 Malecha PW, Williams JH, Kunzler NM, Goldfrank LR, Alter HJ, Doran KM. Material needs of emergency department patients: a systematic review. Acad Emerg Med 2018; 25 (03) 330-359
8 Gordon JA. The hospital emergency department as a social welfare institution. Ann Emerg Med 1999; 33 (03) 321-325
9 Hsu C, Cruz S, Placzek H. et al. Patient perspectives on addressing social needs in primary care using a screening and resource referral intervention. J Gen Intern Med 2020; 35 (02) 481-489
10 Tong ST, Liaw WR, Kashiri PL. et al. Clinician experiences with screening for social needs in primary care. J Am Board Fam Med 2018; 31 (03) 351-363
11 Persaud S. Addressing social determinants of health through advocacy. Nurs Adm Q 2018; 42 (02) 123-128
12 Bleacher H, Lyon C, Mims L, Cebuhar K, Begum A. The feasibility of screening for social determinants of health: seven lessons learned. Fam Pract Manag 2019; 26 (05) 13-19
13 Zulman DM, Maciejewski ML, Grubber JM. et al. Patient-reported social and behavioral determinants of health and estimated risk of hospitalization in high-risk veterans affairs patients. JAMA Netw Open 2020; 3 (10) e2021457
14 Power-Hays A, Li S, Mensah A, Sobota A. Universal screening for social determinants of health in pediatric sickle cell disease: a quality-improvement initiative. Pediatr Blood Cancer 2020; 67 (01) e28006
15 Chang C, Ceci C, Uberoi M, Waselewski M, Chang T. Youth perspectives on their medical team's role in screening for and addressing social determinants of health. J Adolesc Health 2022; 70 (06) 928-933
16 Berger-Jenkins E, Monk C, DʼOnfro K. et al. Screening for both child behavior and social determinants of health in pediatric primary care. J Dev Behav Pediatr 2019; 40 (06) 415-424
17 Katz-Wise SL, Reisner SL, White Hughto JM, Budge SL. Self-reported changes in attractions and social determinants of mental health in transgender adults. Arch Sex Behav 2017; 46 (05) 1425-1439
18 Gottlieb L, Hessler D, Long D, Amaya A, Adler N. A randomized trial on screening for social determinants of health: the iScreen study. Pediatrics 2014; 134 (06) e1611-e1618
19 Tai-Seale M, Downing NL, Jones VG. et al. Technology-enabled consumer engagement: promising practices at four health care delivery organizations. Health Aff (Millwood) 2019; 38 (03) 383-390
20 Gold R, Bunce A, Cowburn S. et al. Adoption of social determinants of health EHR tools by community health centers. Ann Fam Med 2018; 16 (05) 399-407
21 Ancker JS, Barrón Y, Rockoff ML. et al. Use of an electronic patient portal among disadvantaged populations. J Gen Intern Med 2011; 26 (10) 1117-1123
22 Singh P, Jonnalagadda P, Morgan E, Fareed N. Outpatient portal use in prenatal care: differential use by race, risk, and area social determinants of health. J Am Med Inform Assoc 2022; 29 (02) 364-371
23 Anthony DL, Campos-Castillo C, Lim PS. Who isn't using patient portals and why? Evidence and implications from a national sample of US adults. Health Aff (Millwood) 2018; 37 (12) 1948-1954
24 Wu AW, Weston CM, Ibe CA. et al. The Baltimore community-based organizations neighborhood network: enhancing capacity together (CONNECT) cluster RCT. Am J Prev Med 2019; 57 (02) e31-e41
25 Capp R, Misky GJ, Lindrooth RC. et al. Coordination program reduced acute care use and increased primary care visits among frequent emergency care users. Health Aff (Millwood) 2017; 36 (10) 1705-1711
26 Kaufman S, Ali N, DeFiglio V, Craig K, Brenner J. Early efforts to target and enroll high-risk diabetic patients into urban community-based programs. Health Promot Pract 2014; 15 (2, Suppl): 62S-70S
27 Lin MP, Blanchfield BB, Kakoza RM. et al. ED-based care coordination reduces costs for frequent ED users. Am J Manag Care 2017; 23 (12) 762-766
28 Wilcox D, McCauley PS, Delaney C, Molony SL. Evaluation of a hospital: community partnership to reduce 30-day readmissions. Prof Case Manag 2018; 23 (06) 327-341
29 Laranjo L, Dunn AG, Tong HL. et al. Conversational agents in healthcare: a systematic review. J Am Med Inform Assoc 2018; 25 (09) 1248-1258
30 Montenegro JL, da Costa CA, da Rosa Righi R. Survey of conversational agents in health. Expert Syst Appl 2019; 129: 56-67
31 Xiao Z, Zhou MX, Liao QV. et al. Tell me about yourself: using an AI-powered chatbot to conduct conversational surveys with open-ended questions. ACM Trans Comput Hum Interact (TOCHI) 2020; 27 (03) 1-37
32 Kocielnik R, Agapie E, Argyle A. et al. HarborBot: a chatbot for social needs screening. AMIA Annu Symp Proc 2019; 2019: 552-561
33 Kocielnik R, Langevin R, George JS. , et al. Can I talk to you about your social needs? Understanding preference for conversational user interface in health. Paper presented at: CUI 2021: 3rd Conference on Conversational User Interfaces, Bilbao, Spain; July 27–29, 2021:1–10
34 Lucas GM, Gratch J, King A, Morency LP. It's only a computer: virtual humans increase willingness to disclose. Comput Human Behav 2014; 37: 94-100
35 Auriacombe M, Moriceau S, Serre F. et al. Development and validation of a virtual agent to screen tobacco and alcohol use disorders. Drug Alcohol Depend 2018; 193: 1-6
36 Philip P, Micoulaud-Franchi JA, Sagaspe P. et al. Virtual human as a new diagnostic tool, a proof of concept study in the field of major depressive disorders. Sci Rep 2017; 7 (01) 42656
37 Lucas GM, Rizzo A, Gratch J. et al. Reporting mental health symptoms: breaking down barriers to care with virtual human interviewers. Front Robot AI 2017; 4: 51
38 Stowell E, Lyson MC, Saksono H. , et al. Designing and evaluating mHealth interventions for vulnerable populations: a systematic review. Paper presented at: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, Canada; April 21–26, 2018:1–17
39 Cornelius JB, St Lawrence JS, Howard JC. et al. Adolescents' perceptions of a mobile cell phone text messaging-enhanced intervention and development of a mobile cell phone-based HIV prevention intervention. J Spec Pediatr Nurs 2012; 17 (01) 61-69
40 Kocaballi AB, Sezgin E, Clark L. et al. Design and evaluation challenges of conversational agents in health care and well-being: selective review study. J Med Internet Res 2022; 24 (11) e38525
41 Tudor Car L, Dhinagaran DA, Kyaw BM. et al. Conversational agents in health care: scoping review and conceptual analysis. J Med Internet Res 2020; 22 (08) e17158
42 Milne-Ives M, de Cock C, Lim E. et al. The effectiveness of artificial intelligence conversational agents in health care: systematic review. J Med Internet Res 2020; 22 (10) e20346
43 Kamita T, Ito T, Matsumoto A, Munakata T, Inoue T. A chatbot system for mental healthcare based on SAT counseling method. Mob Inf Syst 2019; 2019: 1-11
44 Ly KH, Ly AM, Andersson G. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet Interv 2017; 10: 39-46
45 Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health 2017; 4 (02) e19
46 Inkster B, Sarda S, Subramanian V. An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study. JMIR Mhealth Uhealth 2018; 6 (11) e12106
47 Griol D, Callejas Z. Mobile conversational agents for context-aware care applications. Cognit Comput 2016; 8 (02) 336-356
48 Galescu L, Allen J, Ferguson G, Quinn J, Swift M. . Speech recognition in a dialog system for patient health monitoring. Paper presented at: 2009 IEEE International Conference on Bioinformatics and Biomedicine Workshop, Washington, DC, United States; November 1–4, 2009:302–307 IEEE.
49 Rhee H, Allen J, Mammen J, Swift M. Mobile phone-based asthma self-management aid for adolescents (mASMAA): a feasibility study. Patient Prefer Adherence 2014; 8: 63-72
50 Vita S, Marocco R, Pozzetto I. et al. The'doctor apollo'chatbot: a digital health tool to improve engagement of people living with HIV. J Int AIDS Soc 2018; 21 (Suppl 8): e25187
51 Denecke K, Hochreutener SL, Pöpel A, May R. Self-anamnesis with a conversational user interface: concept and usability study. Methods Inf Med 2018; 57 (5-06): 243-252
52 Ni L, Lu C, Liu N, Liu J. . Mandy: towards a smart primary care chatbot application. In: International Symposium on Knowledge and Systems Sciences, Bangkok, Thailand; Singapore: Springer; November 17–19, 2017:38–52
53 Razzaki S, Baker A, Perov Y. et al. A comparative study of artificial intelligence and human doctors for the purpose of triage and diagnosis arXiv preprint arXiv 2018;1806:10698. 2018 June 27
54 Ghosh S, Bhatia S, Bhatia A. Quro: facilitating user symptom check using a personalised chatbot-oriented dialogue system. Stud Health Technol Inform 2018; 252: 51-56
55 Wilson N, MacDonald EJ, Mansoor OD, Morgan J. In bed with Siri and Google Assistant: a comparison of sexual health advice. BMJ 2017; 359: j5635
56 Wang H, Zhang Q, Ip M, Lau JT. Social media–based conversational agents for health management and interventions. Computer 2018; 51 (08) 26-33
57 Elmasri D, Maeder A. . A conversational agent for an online mental health intervention. In: International Conference on Brain Informatics, Cham,: Springer; October 13, 2016: 243–251
58 Chaix B, Bibault JE, Pienkowski A. et al. When chatbots meet patients: one-year prospective study of conversations between patients with breast cancer and a chatbot. JMIR Cancer 2019; 5 (01) e12856
59 Fan X, Chao D, Zhang Z, Wang D, Li X, Tian F. Utilization of self-diagnosis health chatbots in real-world settings: case study. J Med Internet Res 2021; 23 (01) e19928
60 Weiner BJ, Lewis CC, Stanick C. et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci 2017; 12 (01) 108
61 Creswell JW, Creswell JD. . Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, CA: Sage Publications; 2017
62 Tanabe P, Gimbel R, Yarnold PR, Kyriacou DN, Adams JG. Reliability and validity of scores on The Emergency Severity Index version 3. Acad Emerg Med 2004; 11 (01) 59-65
63 Centers for Medicare and Medicaid Services. . The accountable health communities health-related social needs screening tool. AHC Screening Tool. 2019. Accessed March 15, 2022 at: https://innovation.cms.gov/files/worksheets/ahcm-screeningtool.pdf
64 Benefits Eligibility Screening Tool (BEST). . ( n.d. ). Accessed December 9, 2021 at: https://www.disabilitybenefitscenter.org/glossary/benefits-eligibility-screening-tool
65 Johnson S, Liu P, Campa D. , et al. Los Angeles County Health Agency: Social & Behavioral Determinants of Health Screening Guide [White Paper]. 2019
66 Harris PA. Research Electronic Data Capture (REDCap) - planning, collecting and managing data for clinical and translational research. BMC Bioinformatics 2012; 13 (Suppl 12): A15
67 Emerald City Resource Guide [Internet].. Real Change; 2021. Accessed December 2022 at: https://www.realchangenews.org/emerald-city-resource-guide
68 Washington 2-1-1 [Internet]. Washington 211; 2022. Accessed December 12, 2022 at: https://wa211.org/
69 Morris NS, MacLean CD, Chew LD, Littenberg B. The Single Item Literacy Screener: evaluation of a brief instrument to identify limited reading ability. BMC Fam Pract 2006; 7 (01) 21
70 Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005; 15 (09) 1277-1288
71 Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009; 4 (01) 50
72 Hennink MM, Kaiser BN, Marconi VC. Code saturation versus meaning saturation: how many interviews are enough?. Qual Health Res 2017; 27 (04) 591-608
73 Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc 2004; 11 (02) 104-112
74 Lybarger K, Ostendorf M, Yetisgen M. Annotating social determinants of health using active learning, and characterizing determinants using neural event extraction. J Biomed Inform 2021; 113: 103631
75 LaForge K, Gold R, Cottrell E. et al. How 6 organizations developed tools and processes for social determinants of health screening in primary care: an overview. J Ambul Care Manage 2018; 41 (01) 2-14
76 Buitron de la Vega P, Losi S, Sprague Martinez L. et al. Implementing an EHR-based screening and referral system to address social determinants of health in primary care. Med Care 2019; 57 (Suppl 6, Suppl 2): S133-S139
77 Drake C, Batchelder H, Lian T. et al. Implementation of social needs screening in primary care: a qualitative study using the health equity implementation framework. BMC Health Serv Res 2021; 21 (01) 975
78 Wallace AS, Luther B, Guo JW, Wang CY, Sisler S, Wong B. Implementing a social determinants screening and referral infrastructure during routine emergency department visits, Utah, 2017–2018. Prev Chronic Dis 2020; 17: E45
79 Hassan A, Blood EA, Pikcilingis A. et al. Youths' health-related social problems: concerns often overlooked during the medical visit. J Adolesc Health 2013; 53 (02) 265-271
80 Sundar KR. Universal screening for social needs in a primary care clinic: a quality improvement approach using the your current life situation survey. Perm J 2018; 22: 18-089
81 Butler ED, Morgan AU, Kangovi S. Screening for unmet social needs: Patient engagement or alienation?. NEJM Catal 2020; 1 (04) 10

Zusatzmaterial

Supplementary Material