Contributors to Electronic Health Record-Integrated Secure Messaging Use: A Study of Over 33,000 Health Care Professionals

Keywords

communication - electronic health records - adoption of new technology - secure messaging - interprofessional communication

Background and Significance

Communication between clinicians is an integral part of health care delivery; it accounts for more than half of all information exchange and is vital for the effective and safe care of patients.[1] [2] Although face-to-face communication is preferred, it is often not possible due to a lack of physical proximity or other constraints.[3] As such, alternate modes of clinician-to-clinician communication have been increasingly used, including synchronous (e.g., telephone) and asynchronous (e.g., email and pager) modalities. With the widespread use of mobile phones, there has been a dramatic increase in the use of text-based asynchronous messaging platforms like secure messaging.[4] [5] [6] [7]

Secure messaging allows for Health Insurance Portability and Accountability Act (HIPAA)-compliant text messaging between clinicians. Both standalone mobile text messaging applications (e.g., TigerText and Voalte) and electronic health record (EHR)-integrated applications (e.g., Epic Secure Chat and Cerner CareAware) exist.[8] In contrast to EHR-integrated email communication, secure messaging is designed for more interactive and conversational messaging between clinicians. Studies have reported a doubling in secure messaging use over the past 5 years.[4] [5] [6] [7] Despite the rapid growth of secure messaging in health care settings, little is known about the factors that contribute to its use for clinical communication.

Most studies evaluating the use of secure messaging have relied on qualitative techniques to assess clinician experiences and user-interface or workflow-related barriers to adoption.[9] [10] [11] [12] [13] [14] [15] [16] [17] [18] These previous studies have largely focused on individual clinical groups or specific clinical settings, such as communication between physicians and nurses in emergency departments.[11] [12] [14] [18] [19] Although several recent studies have described the growth in the volume of secure messaging use,[4] [5] [6] [20] [21] [22] few studies have quantified the uptake of secure messaging at the health professional-level across an organization. Therefore, variability in secure messaging use within an organization remains relatively poorly studied.

Objectives

The objective of this study was to assess the factors that contribute to health care professional use of EHR-integrated secure messaging across a large health care system involving several academic and community hospitals and many outpatient clinic locations. We identified secure messaging users and nonusers and investigated the contribution of clinical role, clinical unit, hospital or clinic location, and inpatient versus outpatient setting toward secure messaging use.

Methods

Study Setting

This study was conducted at Washington University School of Medicine and BJC Healthcare, a single large health care system consisting of 14 hospitals and 263 outpatient clinic locations. These hospital and clinic locations include both academic and community-based practice settings. Epic EHR (Epic Systems, Verona, Wisconsin, United States) was used across the health system.

The secure messaging platform used was Epic's Secure Chat, a virtual text messaging platform integrated within the Epic EHR, which was introduced to our health system in September 2019. Epic's Secure Chat allows EHR users (i.e., clinicians and support staff) to send messages to other users within the same institution using a text chat interface from within Epic's desktop and mobile clients. No other secure messaging applications are available at our institution. Other forms of clinical communication available included telephone, pager, and EHR-integrated email.

Participants and Data Sources

This study included all health care professionals (i.e., physicians, nurses, therapists, and social workers) who were clinically active between February 1, 2023 and February 28, 2023. For each participant, metadata on secure messaging use and EHR login information was collected from institutional data warehouses. This login data contained a trail of timestamps recording when users logged in to, logged out of, or timed out of a session within Epic. Login information was collected to determine whether participants were clinically active. Health care professionals with fewer than 20 EHR logins during the study period (i.e., fewer than an average of five logins per week) were excluded, as they were assumed to be clinically inactive during that period.

Primary Outcome

The primary outcome of this study was a binary indicator of whether a health care professional was a secure messaging user. Health care professionals were considered secure messaging users if they sent at least one message during the study period, regardless of whether it was a response to another individual's message or if it was the initiating message in a conversation. Automated messages were excluded from contributing toward a health care professional's secure messaging use.

Clinician-Level Measures

Additional metadata on health care professionals was collected to contextualize the clinical work environment. This included data on their clinical role, their most frequent practice setting (inpatient vs. outpatient), and the most frequent clinical unit where they worked (i.e., specific hospital ward or outpatient clinic) during the study period. Therefore, each health care professional had four measures: clinical role, practice setting, individual clinical unit, and hospital or clinical location. See [Fig. 1] for an illustration of the analytic strategy employed using these characteristics.

Fig. 1 Illustration of analytic strategy.

Individual clinical units were nested within larger organizational entities referred to as hospital or clinic locations, which reflected the general governance structure of the health care system ([Fig. 1]). For example, an inpatient nurse who worked on Ward 2 East at Hospital X would have “Ward 2 East” as their clinical unit and “Hospital X” as their hospital location. Similarly, an outpatient physician who worked at the Internal Medicine Clinic 1 at Medical Clinic Building Z would have “Internal Medicine Clinic 1” as their clinical unit and “Medical Clinic Building Z” as their clinic location. Each hospital or clinic location could have multiple clinical units nested within it ([Fig. 1], right).

Clinical roles were categorized into the following: pharmacist, physician, advanced practice provider (APP), therapist, medical assistant/technician, nurse, social worker/case management, others (e.g., paramedic, perfusionist, and study coordinator), or missing.

Statistical Analysis

All data were aggregated using the individual health care professional as the unit of analysis. In other words, our final analytic dataset had one row for each health care professional, with columns for their clinical role, clinical unit, hospital or clinic location, practice setting, and a binary outcome variable representing any secure messaging use during the considered study period.

Descriptive analyses in the form of frequencies and percentages were calculated to examine distributions of health care professional characteristics by secure messaging use. To better understand variability in secure messaging use, a multilevel mixed-effect logistic regression model was used. This model included hospital or clinic location and clinical unit nested within the hospital or clinic location as random effects to account for the clustering of individuals within these work settings. Clinical role and practice setting (inpatient vs. outpatient) were included as fixed effects in the models.

The proportion of variance accounted for by hospital or clinic location and clinical unit was calculated by approximating the intraclass correlation using the following equation: .[23] In this equation V_c refers to the variance attributable to the specific level of clustering (either hospital or clinic location or individual clinical unit) and V_{c tot} refers to the total variance accounted for by both levels. Odds ratios (ORs) and 95% confidence intervals were calculated for the association between the fixed effects and secure messaging use. All analyses were conducted in R (R Core Team, 2021), Rstudio (Rstudio Team, 2021), and Python 3.9.7.[24]

Results

The study included 33,195 health care professionals, 20,795 of whom worked across 800 inpatient clinical units in 14 hospitals; and 12,400 of whom worked across 1,055 outpatient clinical units within 263 outpatient clinic locations. For our cohort, 62% (20,576/33,195) were secure messaging users ([Tables 1]–[3]). Among inpatient health care professionals, 66% (13,688/20,795) were secure messaging users, whereas 56% (6,888/12,400) outpatient health care professionals were secure messaging users.

There was considerable variation in secure messaging use across clinical units ([Fig. 2A]). Among clinical units, 63% (1,172 /1,855) had more than 50% of their health care professionals using secure messaging. Additionally, 27% of clinical units (496/1,855) had over 90% of their health care professionals using secure messaging, whereas 20% of clinical units (375/1,855) had less than 10% of their health care professionals using secure messaging. The distribution of secure messaging use by clinical units was similar between inpatient and outpatient settings; 65% (520/800) of inpatient clinical units had more than 50% of their health care professionals using secure messaging, whereas 62% (652/1,055) of outpatient clinical units had more than 50% of their health care professionals using secure messaging.

Fig. 2 Percent secure messaging use within each clinical unit (A) or each hospital or clinic location (B) shown as caterpillar plots, stratified by inpatient versus outpatient practice setting. Each clinical unit (A), hospital (B, left), or clinic location (B, right) is plotted as a single dot, with the dot size indicating the number of health care professionals in that location.

There was considerable variation in secure messaging use across hospital or clinic locations ([Fig. 2B]). Among the hospital or clinic locations, 61% (168/277) had more than 50% of their health professional using secure messaging. Additionally, 16% of hospitals or clinic locations (44/277) had over 90% of health professionals using secure messaging, while 17% (46/ 277) had less than 10% of their health professionals using secure messaging. There was less variability in the prevalence of secure messaging use between hospitals compared with between clinic locations.

In multivariable analysis, 25.3% of the variance in secure messaging use was attributable to the clinical unit and 30.5% of the variance was attributable to hospital or clinic location. Compared with nurses (the largest clinician group, comprised of 8,823 secure messaging users among 11,919 nurses [74%]), APPs (OR: 1.65, 95% confidence interval [CI]: 1.37–1.98), pharmacists (OR: 2.77, 95% CI: 1.87–4.10), and physicians (OR: 1.50, 95% CI: 1.31–1.73) had significantly increased odds of secure messaging use, whereas medical assistants (OR: 0.20, 95% CI: 0.18–0.21), social work/case managers (OR: 0.57, 95% CI: 0.41–0.80), and therapists (OR: 0.53, 95% CI: 0.42–0.67) had significantly decreased odds ([Table 4]). On a population level, there was no significant difference in secure messaging use between outpatient and inpatient settings.

Table 1 Distribution of study participants and secure messaging use by clinical role. Clinical roles are ordered alphabetically after the reference group (nurses)

	Inpatient		Outpatient
	No. of secure messaging users/total users (%)	Total messages sent	No. of secure messaging users/total users(%)	Total messages sent
Nurse	7,578/10,055 (75.4%)	523,613	1,245/1,864 (66.8%)	117,037
APP	716/937 (76.4%)	92,613	585/695 (84.2%)	60,569
Medical assistants/technicians	1,260/3,264 (38.6%)	44,157	2,375/4,408 (53.9%)	139,751
Pharmacists	321/386 (83.2%)	32,318	77/88 (87.5%)	4,851
Physician	1,859/2,237 (83.1%)	237,868	1,706/2,057 (82.9%)	142,430
Social work/case management	238/268 (88.8%)	40,798	69/132 (52.3%)	3,757
Therapist	966/1,170 (82.6%)	47,959	285/566 (50.4%)	7,347
Other	133/424 (31.4%)	3,696	185/596 (31.0%)	7,872
Missing	617/2,054 (30.0%)	26,962	361/1,994 (18.1%)	20,583
Total	13,688/20,795 (66%)	1,049,984	6,888/12,400 (56%)	504,197

Abbreviation: APP, advanced practice provider.

Table 2 Among physicians (N = 4,294), the prevalence of secure messaging users for the 10 most common specialties in the inpatient versus outpatient setting

Inpatient (N = 2,237 physicians)			Outpatient (N = 2,057 physicians)
Specialty	No. of secure messaging users/total users (%)	Total messages sent	Specialty	No. of secure messaging users/total users (%)	Total messages sent
Emergency medicine	267/282 (95%)	28,822	Internal medicine	281/309 (91%)	61,610
General surgery	224/263 (85%)	20,476	Neurology	85/114 (75%)	3,915
Radiology	180/241 (75%)	3,954	Cardiology	77/92 (84%)	4,426
Anesthesiology	150/215 (70%)	3,964	Pediatrics	64/85 (75%)	2,772
Internal medicine	156/166 (94%)	51,103	Orthopaedic surgery	66/76 (87%)	2,013
Pediatrics	122/134 (91%)	22,575	Family medicine	32/69 (46%)	7,459
Obstetrics and gynecology	85/118 (72%)	8,480	Clinical pathology	32/69 (46%)	652
Critical care medicine	77/86 (90%)	11,817	Gastroenterology	58/63 (92%)	5,500
Cardiology	70/81 (86%)	5,820	Oncology	55/62 (89%)	2,847
Neonatology	45/54 (83%)	2,452	Ophthalmology	47/57 (82%)	1,414

Discussion

We found that secure messaging was widely used by a diverse range of health care professionals working in multiple clinical locations and practice settings. However, there was substantial variation in secure messaging use based on clinical roles and work context; surprisingly, these differences were approximately equally attributable to local work settings (represented by individual clinical units) and larger organizational structures (represented by hospital or clinic location). After accounting for these factors, working in inpatient versus outpatient practice settings did not contribute to secure messaging use. To the best of our knowledge, this is one of the largest studies on the contributors to secure messaging use.

We found that secure messaging use varied by clinical roles, with pharmacists, APPs, and physicians being the most likely users. The reasons for this are likely multifactorial; it is possible that these clinical roles have the highest communication needs for care coordination, especially when in-person communication is not practical and asynchronous communication can suffice. For example, physicians and APPs need to coordinate with consulting services, and pharmacists need to communicate with ordering health care professionals in multiple units with whom they do not share physical proximity. Previous studies have shown that use, adoption, and acceptance of health technology are influenced by its alignment with existing tasks and work processes.[25] [26] [27] [28] [29] [30] As such, our finding that there is variation in secure messaging use by clinical roles may relate to how secure messaging integrates within the clinical workflow or needs of these clinical roles.

It is perhaps not surprising that the individual clinical unit explained nearly a quarter of the variation in secure messaging use. Individual clinical units represented local work settings, often independently run, with each having their own unique cultures of communication and technology use practices. Previous qualitative research on communication behaviors has shown that there is a network externality effect in communication platform use; users tend to use communication platforms more when their messaging partners are also active on those platforms.[16] [29] [31] [32] Positive peer attitudes toward innovation have previously been shown to have a positive impact on behavioral intentions and facilitate successful health technology adoption.[26] [27] [33] It is likely that these network externality effects contributed to the clustering in secure messaging use by clinical units that we observed.

However, we also found that hospital or clinic location explained nearly a third of the variation in secure messaging use. These hospital or clinic locations represented larger organizational governance structures within our health care system, so we believe our findings suggest that leadership practices and informatics governance policies can meaningfully impact secure messaging behaviors. Previous studies on technology acceptance have found that health care organizations can support technology adoption by setting policies and expectations for use and by providing adequate training and technical support.[15] [25] [27] [29] The extent of secure messaging training and technical support was highly variable across the various hospitals and clinic locations within our health system, possibly explaining the strong organization-level clustering we observed.

We believe our results have implications for our understanding of secure messaging behaviors. Secure messaging has several potential advantages as a powerful mode of clinical communication; it may lower barriers to communication, improve collaboration, and decrease interruptions.[9] [10] [30] However, it may also increase the burden of communication and lead to less efficient communication when used ineffectively.[8] [16] [17] It is likely that health care systems will want to influence secure messaging uptake and modify messaging behaviors to best maximize secure messaging's benefits while minimizing potential harms. Our results suggest that successful implementation of secure messaging policies should be targeted at multiple—organizational levels to influence behaviors.[15] [33] Although the dissemination of organization-wide policies is important, such interventions should also be tailored to fit the cultural practices and needs of individual clinical units and clinical roles.

Limitations

This study had several limitations. Although we examined secure messaging use across 14 hospitals and a large number of outpatient clinic locations, all of these sites were affiliated with a single health care system with the same EHR and secure messaging platform; therefore, our results may not generalize to other health care systems or messaging platforms. However, we did repeat the analysis with a different study period spanning between December 2022 and January 2023 and observed similar results (data not shown). Because we did not have access to detailed work schedules, we only included health care professionals with at least 20 EHR logins in the study, with the assumption that health care professionals with at least this level of EHR activity over a single month would have been actively working; we believe this to be a reasonable proxy since EHR use is an essential part of clinical work, but we acknowledge that there may have been potential misclassification. Similarly, our assignment of clinical units was also based on login information and was potentially imprecise.

Finally, we can only speculate on the reasons for secure messaging use. No information on secure message content was collected during the study, and individual health care professionals were not interviewed to explore qualitative contributors to secure messaging use. In addition, organizational policies and guidelines for secure messaging use were highly heterogenous across our health care system during initial deployment and throughout the study period. Although we observed clustering in secure messaging use by clinical units and hospitals or clinic locations, as well as significant differences in use by clinical role, the exact reasons for this clustering and observed differences remain unknown.

Conclusion

We identified secure messaging users and nonusers across a large health care system and found that variance in secure messaging use was approximately equally attributable to the individual clinical unit (likely reflecting local culture and communication practices), and hospital or clinic location (likely reflecting leadership or policy-level guidance). These results suggest that future interventions intended to influence secure messaging use may need to not only involve organizational-level policies but also be tailored toward individual clinical units and health care professional workflows.

Clinical Relevance Statement

We found that more than half of health care professionals across inpatient and outpatient locations were secure messaging users. In addition, the variance in secure messaging use was roughly equally attributable to clinical units (i.e., a specific inpatient ward or outpatient clinic) and higher-level organizational structure (i.e., the hospital or clinic location of the clinical units). Therefore, policies for secure messaging use should be targeted at both levels of workplace organization to meaningfully influence secure messaging behavior.

Multiple-Choice Questions

1. What is a secure message?

   • an encrypted email

   • a HIPAA-compliant text-based message

   • a SMS text-based message on a mobile device

   • a message between pagers.

   Correct Answer: The correct answer is option b. Secure messaging was created due to security concerns among health care professionals who would send non-HIPAA compliant text-based messages containing patient information between mobile devices. Therefore, hospitals have implemented HIPAA-compliant solutions to allow for secure text-based communication between health care professionals.

2. Which factors were found to influence secure messaging use in this study?

   • clinical role, hospital location, time of day, and geographical location

   • clinical role only

   • clinical role, hospital location, and clinical unit

   • hospital location and clinical unit.

   Correct Answer The correct answer is option c. We were able to find variations in secure messaging use among three aspects of clinical work. This included clinical role, hospital location, and the clinical unit. Therefore, these three aspects of clinical work are potential factors influencing secure messaging use among health care professionals.

Table 3 Secure messaging use for 10 representative clinical units in the inpatient and outpatient setting

Inpatient			Outpatient
Inpatient clinical unit	No. of secure messaging users/total users (%)	Total messages sent	Outpatient clinical unit	No. of secure messaging users/total users (%)	Total messages sent
Hospital 1 emergency	355/410 (86.6%)	43,469	Oncology clinic 7 at location 1	151/303 (49.8%)	10,650
Hospital 1 radiology	195/276 (70.7%)	6,885	Revenue management	1/297 (0.3%)	7
Hospital 1 infection control	110/256 (43.0%)	3,252	Medicine resident clinic at location 2	128/143 (89.5%)	15,807
Hospital 1 internal medicine	185/215 (86.0%)	55,449	Radiation oncology clinic at location 1	64/137 (46.7%)	1,354
Hospital 2 ICU 1	114/125 (91.2%)	3,718	Pediatric clinic at location 3	72/89 (80.9%)	3,093
Hospital 1 catheterization laboratory	47/86 (54.7%)	1,422	Pediatric gastroenterology clinic 2 at location 4	43/45 (95.6%)	3,084
Hospital 3 ICU	69/85 (81.2%)	1,565	Dermatology clinic at location 5	15/35 (42.9%)	163
Hospital 4 internal medicine	20/73 (27.4%)	226	Nephrology clinic at location 1	31/35 (88.6%)	2,059
Hospital 1 psychiatry	38/57 (66.7%)	5,848	Pediatric clinic at location 6	6/23 (26.1%)	18
Hospital 1 palliative care	10/10 (100.0%)	522	Physical therapy clinic at location 7	10/16 (62.5%)	58

Abbreviation: ICU, intensive care unit.

Table 4 Odds ratio estimates for fixed effects from the multilevel logistic regression model, after controlling for clustering at the level of the individual clinical unit (ICC = 0.253) and hospital or clinic location (ICC = 0.305). Clinical roles are ordered alphabetically after the reference group (nurses)

Variable	Group	OR (95% CI)	p-Value
Provider type	Nurse	Reference	<0.001
	APP	1.65 (1.37–1.98)
	Medical assistant/technician	0.20 (0.18–0.21)
	Pharmacist	2.77 (1.87–4.10)
	Physician	1.50 (1.31–1.73)
	Social work/case manager	0.57 (0.41–0.80)
	Therapist	0.53 (0.42–0.67)
	Other	0.09 (0.07–0.10)
	Missing	0.08 (0.07–0.09)
Practice setting	Inpatient	Reference	0.453
	Outpatient	1.38 (0.60–3.19)

Abbreviations: APP, advanced practice provider; CI, confidence interval; OR, odds ratio.

• References

• 1 Edwards A, Fitzpatrick L-A, Augustine S. et al. Synchronous communication facilitates interruptive workflow for attending physicians and nurses in clinical settings. Int J Med Inform 2009; 78 (09) 629-637
• 2 Spencer R, Coiera E, Logan P. Variation in communication loads on clinical staff in the emergency department. Ann Emerg Med 2004; 44 (03) 268-273
• 3 Manojlovich M, Harrod M, Hofer TP, Lafferty M, McBratnie M, Krein SL. Using qualitative methods to explore communication practices in the context of patient care rounds on general care units. J Gen Intern Med 2020; 35 (03) 839-845
• 4 In US, SMS text messaging tops mobile phone calling. New York: NIELSEN.; Accessed October 3, 2008 at: https://www.nielsen.com/us/en/insights/news/2008/in-us-text-messaging-tops-mobile-phone-calling.html
• 5 Byrd TF, Fancher KG, Liebovitz DM, O'Leary KJ. Trends in secure mobile communication technology use among hospitalists in North America, 2016–2021. Health Policy Technol 2022; 11 (04) 100689
• 6 Liu X, Sutton PR, McKenna R. et al. Evaluation of secure messaging applications for a health care system: a case study. Appl Clin Inform 2019; 10 (01) 140-150
• 7 Steitz BD, Unertl KM, Levy MA. An analysis of electronic health record work to manage asynchronous clinical messages among breast cancer care teams. Appl Clin Inform 2021; 12 (04) 877-887
• 8 Hagedorn PA, Singh A, Luo B, Bonafide CP, Simmons JM. Secure text messaging in healthcare: latent threats and opportunities to improve patient safety. J Hosp Med 2020; 15 (06) 378-380
• 9 Patel N, Siegler JE, Stromberg N, Ravitz N, Hanson CW. Perfect storm of inpatient communication needs and an innovative solution utilizing smartphones and secured messaging. Appl Clin Inform 2016; 7 (03) 777-789
• 10 Chandra S, Oberg M, Hilburn G, Wu DT, Adhyaru BB. Improving communication in a large urban academic safety net hospital system: implementation of secure messaging. J Med Syst 2023; 47 (01) 56
• 11 Luu T, Spiegelman L, Nykin D. et al. Implementation of an electronic health record-based messaging system in the emergency department: effects on physician workflow and resident burnout. J Patient Saf 2022; 18 (02) e542-e546
• 12 Arscott WT, Gray K, Kuska D. et al. Evaluating the impact of secure mobile messaging on communication and cancer care team satisfaction in a large radiation oncology clinic. J Oncol Pract 2019; 15 (08) e733-e738
• 13 Przybylo JA, Wang A, Loftus P, Evans KH, Chu I, Shieh L. Smarter hospital communication: secure smartphone text messaging improves provider satisfaction and perception of efficacy, workflow. J Hosp Med 2014; 9 (09) 573-578
• 14 Wu R, Lo V, Morra D. et al. A smartphone-enabled communication system to improve hospital communication: usage and perceptions of medical trainees and nurses on general internal medicine wards. J Hosp Med 2015; 10 (02) 83-89
• 15 Haun JN, Hathaway W, Chavez M. et al. Clinical practice informs secure messaging benefits and best practices. Appl Clin Inform 2017; 8 (04) 1003-1011
• 16 Byrd IV TF, Speigel PS, Cameron KA, O'Leary KJ. Barriers to adoption of a secure text messaging system: a qualitative study of practicing clinicians. J Gen Intern Med 2023; 38 (05) 1224-1231
• 17 Aziz S, Barber J, Singh A, Alayari A, Rassbach CE. Resident and nurse perspectives on the use of secure text messaging systems. J Hosp Med 2022; 17 (11) 880-887
• 18 Feinberg J, Shaw S, Kashyap N. et al. Evaluating the impact of a new smartphone texting tool on patient care in obstetrics, an emergent healthcare setting. Appl Clin Inform 2019; 10 (05) 879-887
• 19 Hansen JE, Lazow M, Hagedorn PA. Reducing interdisciplinary communication failures through secure text messaging: a quality improvement project. Pediatr Qual Saf 2018; 3 (01) e053
• 20 Baratta LR, Harford D, Sinsky CA, Kannampallil T, Lou SS. Characterizing the patterns of electronic health record-integrated secure messaging use: cross-sectional study. J Med Internet Res 2023; 25: e48583
• 21 Small W, Iturrate E, Austrian J, Genes N. Electronic health record messaging patterns of health care professionals in inpatient medicine. JAMA Netw Open 2023; 6 (12) e2349136-e2349136
• 22 Kwan B, Bell JF, Longhurst CA, Goldhaber NH, Clay B. Implementation of an electronic health record-integrated instant messaging system in an academic health system. J Am Med Inform Assoc 2024; 31 (04) 997-1000
• 23 Merlo J, Chaix B, Ohlsson H. et al. A brief conceptual tutorial of multilevel analysis in social epidemiology: using measures of clustering in multilevel logistic regression to investigate contextual phenomena. J Epidemiol Community Health 2006; 60 (04) 290-297
• 24 Python Software Foundation. Python Language Reference, Version 3.9.16. 2021
• 25 Grossi A, Hoxhaj I, Gabutti I. et al. Hospital contextual factors affecting the implementation of health technologies: a systematic review. BMC Health Serv Res 2021; 21 (01) 407
• 26 Cresswell K, Sheikh A. Organizational issues in the implementation and adoption of health information technology innovations: an interpretative review. Int J Med Inform 2013; 82 (05) e73-e86
• 27 Gagnon M-P, Desmartis M, Labrecque M. et al. Systematic review of factors influencing the adoption of information and communication technologies by healthcare professionals. J Med Syst 2012; 36 (01) 241-277
• 28 Alexander G, Staggers N. A systematic review of the designs of clinical technology: findings and recommendations for future research. ANS Adv Nurs Sci 2009; 32 (03) 252-279
• 29 Boonstra A, Broekhuis M. Barriers to the acceptance of electronic medical records by physicians from systematic review to taxonomy and interventions. BMC Health Serv Res 2010; 10 (01) 231
• 30 Pourmand A, Roberson J, Gallugi A, Sabha Y, O'Connell F. Secure smartphone application-based text messaging in emergency department, a system implementation and review of literature. Am J Emerg Med 2018; 36 (09) 1680-1685
• 31 Byrd IV TF, Kim JS, Yeh C, Lee J, O'Leary KJ. Technology acceptance and critical mass: development of a consolidated model to explain the actual use of mobile health care communication tools. J Biomed Inform 2021; 117: 103749
• 32 Liebowitz SJ, Margolis SE. Network externality: an uncommon tragedy. J Econ Perspect 1994; 8 (02) 133-150
• 33 Yousef CC, Salgado TM, Farooq A. et al. Predicting health care providers' acceptance of a personal health record secure messaging feature. Appl Clin Inform 2022; 13 (01) 148-160