Engaging Patients and Other Stakeholders in “Designing for Dissemination” of Record Linkage Methods and Tools

Keywords

record linkage - dissemination - patient engagement - stakeholder engagement - patient-centered outcomes research

Background and Significance

In the past 15 years, there has been tremendous advancement in data science methods that leverage data collected in routine care for clinical and translational research.[1] Innovations in extraction, transformation, and loading,[2] data harmonization,[3] data quality assessment,[4] [5] and data governance[6] [7] have emerged and are available to guide this research, yet there are few experts qualified to apply these methods. Ensuring innovative data science methods are disseminated broadly and can be accurately and appropriately applied to clinical research is critical to ensuring research rigor.[8]

Record linkage (RL) methods are among the recent advances in secondary use of clinical and claims data for research.[9] [10] [11] RL refers to the technical and analytic methods for effectively and securely matching patient records from multiple distinct health systems and data platforms, such as electronic health records, administrative claims, patient-reported outcomes measures, and digital health devices.[12] [13] [14] RL has several benefits for research, such as enhancing the amount and types of information available about each patient (e.g., information about all services a patient has received, what prescriptions have been filled, how much physical activity they have done, and their quality of life), which are not typically all included in one data source. This facilitates answering patient-centered research questions about the extent to which certain treatments and interventions impact objective patient behavior, self-reported outcomes, and clinical outcomes, and the relationships among these factors. RL can also enable deduplication to ensure the research sample represents unique individuals (e.g., not double counting a patient who has been seen at more than one health care institution if a researcher has received records from multiple institutions). This helps ensure that the size of the population comprising the research sample is accurately described.

To realize the promise of RL methods for advancing patient-centered outcomes research (PCOR), investigators must adopt and apply RL methods to the testing of novel hypotheses using secondary health data from multiple sources. University of Colorado Record Linkage (CURL) is an electronic health RL software for linking electronic health data from disparate systems (e.g., clinical and claims data).[9] As with all research products, active dissemination to potential adopters and influencers is needed to support broad researcher adoption.[15] [16] Dissemination of data science and informatics methods may benefit from the design of materials that communicate value propositions and facilitate the use of novel health care informatics tools in research and, eventually, clinical care.[17]

As RL and other informatics tools are often used to research patient data[12] [18] (and results have direct implications for and potential value to patient care and health outcomes) patient perspectives are a critical component of positioning their value to promote dissemination and uptake.[19] [20] Further, as patient engagement throughout the research lifecycle is increasingly prioritized,[21] [22] communication materials for engaging with patients around the implications of research methods are required for effective dissemination of novel informatics tools and methodologies. Thus, engaging patients and other stakeholders in dissemination planning allows for the development of effective messages and materials for conveying the value of RL methods, which in turn will promote future adoption of RL methods and ensure transparency in how patient data are used in PCOR and resulting clinical informatics solutions.

Objectives

Our objective was to identify the needs and perspectives of patients and other stakeholders as potential adopters and influencers of RL methods and tools and to design messages and materials to guide the use of RL methods and tools (henceforth, “the RL dissemination package”).

First, we present the methods used to engage stakeholders in the iterative design and development of the RL dissemination package, including the application of user experience (UX) research and design methods. Next, we describe how the results were applied to iterate design choices. To conclude, we discuss the larger contributions of this work to patient engagement in informatics and dissemination science.

Methods

Overview of Dissemination Planning Framework

The dissemination planning framework ([Fig. 1]) begins with stating dissemination goals and objectives.[17] Stakeholders are engaged in addressing identified goals and objectives by delineating the dissemination plan components[23]: (1) Audience segments (potential adopters and influencers of RL methods); (2) Messages (language and imagery to convey the value and impetus for using RL methods); (3) Packaging (the format or medium of the messaging, such as print or video); and (4) Distribution channels (mechanisms to deliver the RL dissemination package to the audience, such as email or television).

Fig. 1 Dissemination planning framework. Figure depicts the processes and components involved in research dissemination planning.

Context

The current project describes patient and other stakeholder engagement in a Patient-Centered Outcomes Research Institute (PCORI)-funded Methods Award focused on the development of incremental privacy-preserving record linkage (iPPRL) methods.[24] [25] The iPPRL method represents one type of process for RL; it is available for use via the CURL system. Prior to launching the current project, we developed an explainer video[26] that describes how CURL addresses key challenges to RL, such as establishing unique identifiers, cleaning and normalizing data, and encrypting data to preserve privacy. This video and a corresponding survey evaluation served as a launching point for conversations with key stakeholders in the current project.

Design Team

The design team included a patient partner and five researchers with expertise in informatics, RL, stakeholder engagement, communication, dissemination and implementation science, user testing, and project management.

Goals and Objectives for Dissemination

Potential adopters of RL methods and tools include investigators conducting PCOR and making research methods decisions as well as data managers and analysts who apply those methods. Potential RL methods influencers include regulatory personnel who set and enforce policies about research data privacy and security. There are also numerous ways in which people whose health data are used in RL research (i.e., patients) represent RL influencers. Patient perspectives are directly relevant, given that their health care records are primary sources for RL research. Many patients are active participants in PCOR and other types of clinical and translational research (e.g., clinical trials, biobanks, registries). Dissemination goals and objectives were, therefore, to:

• Provide investigators and analysts with the tools and resources needed to plan and execute research using RL methods and the CURL platform.

• Provide patient stakeholders with the communication materials necessary to be comfortable and accepting of research using health data and RL methods.

Designing Messaging, Packaging, and Distribution Channels

To design the RL dissemination package, we employed patient engagement methods, customer discovery and value proposition design methods,[27] and UX design methods.[28] [29] [Fig. 2] provides an overview and timeline of all activities.

Fig. 2 Design timeline. Figure provides an overall timeline with brief descriptions of all design activities. CURL, The University of Colorado Record Linkage; RL, record linkage.

Patient Engagement Methods

Two methods were used for patient engagement in this project. First, we partnered with a community member who served as a member of the design team. The purpose was to integrate the patient perspective into all aspects of planning for future dissemination and implementation of RL methods developed in the iPPRL project. The patient partner has no training in research or informatics. She is a person who lives with multiple chronic health conditions and has served as a patient stakeholder representative on numerous PCOR projects. The patient partner met twice a month with other members of the design team and was a coequal partner in decision-making. Her role frequently involved suggesting plain language[30] for patient-oriented messages and materials and drafting scripts and graphic design concepts with the team's communication expert. Over several months, the design team conducted multiple colearning sessions[31]—a core principle of engagement—and discussed concerns patients may have about RL research methods and how we might distill the processes and importance of the research into a format accessible to a general audience.

Second, we engaged patients who serve on research advisory boards involved in conducting research using patient health data. Patient research advisory boards were identified via the design team's professional network. At key points during the design process, patient stakeholder representatives provided input to inform RL dissemination package revisions and identify distribution channels for the patient-oriented RL dissemination package materials.

Customer Discovery and Value Proposition Design

Based on established value proposition design methods,[27] we conducted customer discovery interviews to inform RL dissemination package features and resources needed to support the adoption and use of RL methods and tools. Value proposition design provides a framework whereby a product or service is positioned around what the customer values and needs. To begin, the design team emailed potential RL adopters within their professional networks with an invitation to participate in an interview about their work and RL-related challenges. Potential RL adopters included those known by or recommended to the design team as (1) people already using RL for health data in some capacity who were new to CURL; (2) people whose research aligned with the future use of RL methods (e.g., investigators who work with patient record data; (3) other stakeholders who informed RL adoption (data security and regulatory officials). Snowball sampling was used to identify additional people to interview by asking interviewees to nominate others. This sampling methodology was appropriate, as potential adopters of RL (and CURL, specifically) are a somewhat niche community that would be difficult to reach via broader sampling methods.

Customer discovery interview guides (tailored to stakeholder type) assessed the “jobs, pains, and gains” (i.e., the key elements of value proposition design) of using RL methods in research.[32] [33] [34] Questions focused on describing the work surrounding RL (e.g., “What is your job related to RL?”) and identifying related pains (e.g., “What are the major challenges you encounter?”) and gains (e.g., “What are some positive outcomes or improvements you'd like to achieve at work?”). Customer discovery interviews were conducted in June and July 2019 with 21 people. Interviews lasted 45 to 60 minutes and were conducted by two design team members via phone or in person. Interviews were audio recorded; interviewers took comprehensive notes. A matrix-based coding approach[35] was used to synthesize notes and identify themes related to “jobs, pains, and gains.” Themes were then reviewed by design team members to articulate the current and aspirational features of CURL that serve as “pain relievers” and “gain creators” for those using RL methods.

Insights from customer discovery informed value propositions for key audiences. Value propositions indicate how RL methods and tools align with the desired gains or relieve pains for audience segments. We then used value proposition statements to inform messaging about the benefits of using the CURL platform.

User Experience Design

Based on findings from the patient engagement and value proposition design activities, we developed a RL Planning Workbook for investigators and RL animated explainer videos for PCOR patient stakeholders. We used UX design principles and methods to iteratively develop and refine prototypes.[15] [36] [37] In UX design, designers apply both what they have learned (e.g., from patient engagement, customer discovery, and value proposition design) and the identified associated specifications to create sketches, mockups, and product prototypes.[29] The design phase typically requires multiple iterations before producing a high-fidelity product. We conducted user testing with both stakeholder groups to determine whether the products adequately met users' needs.

Record Linkage Explainer Videos for Patient Stakeholders

To address patient stakeholder information needs, two animated explainer videos focus on data security and privacy ([Fig. 3]).[38] [39] We adapted notes from colearning sessions to create video scripts. A graphic designer developed videos based on the scripts and visual ideas for representing concepts.

Fig. 3 Patient videos. Figure depicts two screen captures. One from the patient data and privacy regulations video and one from the privacy-protected record linkage video. HIPPA, Health Insurance Portability and Accountability Act.

We used two methods to test initial video prototypes with patients and other stakeholders. First, in July 2020, one member of the design team emailed members of existing patient stakeholder groups for other research projects asking them to view and evaluate the videos using a brief online survey. We also sent the survey to investigators who use stakeholder engagement methods. Survey participants comprised eight investigators and seven patient stakeholders. The survey consisted of three open-ended questions, plus one additional question specifically for investigators. Questions included: “What are your general impressions of the videos? What do you like or dislike?”; “What (if anything) did you find confusing or unclear?”; and “What questions or concerns do you have about RL after watching the videos?” Additionally, investigators were prompted to respond to: “As a researcher, we imagine that these videos would be helpful for engaging patient stakeholders and educating them about RL research. In thinking about your past or future/potential work, what else would you need in order to effectively engage patient stakeholders in research that involves RL?”

Next, we presented the videos to five members of an established patient research advisory board and asked: “What are your overall thoughts about the content of the videos?”; “What else (beyond the videos) would you like to know about RL research methods?”; and “How can we best educate patients about RL research methods?” We used this feedback to refine the video scripts and graphics.

Record Linkage Planning Workbook

The RL Planning Workbook[40] was developed to meet the needs of potential CURL users as identified during the customer discovery and value proposition design activities. The Workbook includes worksheets to be used by researchers planning research projects that incorporate RL methods ([Fig. 4]). The Workbook reflects steps in the RL process (as shown in the ideal process maps, [Fig. 4]) and integrates guiding questions for decision-making, such as: “What types of data do I plan to use?” and “Do my data meet the criteria performing RL?”

Fig. 4 Record Linkage Planning Workbook. Figure includes sample workbook pages. CURL, The University of Colorado Record Linkage.

Record Linkage Workshop

A workbook prototype was presented during a RL workshop in November 2019.[41] Workshop attendees were invited colleagues with knowledge and work related to health data and RL including data managers, investigators, and institutional leadership and regulatory officials. Many were stakeholders we also engaged during customer discovery interviews. Three RL subject matter experts (SMEs; outside of the design team) led small group sessions where participants used the workbook to talk through application in sample research contexts. RL SMEs were locally and nationally recognized for their work using RL, but who were not otherwise involved in CURL. Participants made notes indicating anything confusing or unclear in the workbook's contents. The full group then discussed the workbook's usefulness and opportunities for improvement. Overall, workshop participants expressed that the workbook was useful, had relevant prompts for RL planning, appropriately described the RL challenges, and offered pathways for RL solutions. However, participants also felt the layout was hard to follow, instructions were vague, and that the workbook's applicability to other aspects of the research planning process (e.g., grant proposals, institutional review board [IRB] protocols) was unclear. We revised workbook content and improved visual design by adding further instructions, flowcharts, and decision trees to illustrate RL requirements and guide RL methods selection.

In spring 2021, we conducted user testing of the revised RL Planning Workbook using a think aloud protocol[42] (a common user testing method) with 12 participants of varying levels of RL expertise. Prior to completing the think aloud session, participants were asked to consider a project involving RL. During the session, participants navigated the workbook with this project in mind while vocalizing cognitive processes and sharing decision-making considerations aloud. The sessions were facilitated by two researchers with usability expertise. Insights from usability testing informed final workbook changes.

Distribution of the Record Linkage Dissemination Package

Final Patient Advisory Board Meeting

In June 2021, we met with the patient research advisory board to share the revised patient stakeholder-oriented RL explainer videos and gather perspectives about distribution. The discussion was framed around four questions: “In what contexts should videos be used for patient engagement?”; “What additional information should accompany the videos?”; “How might this support patient stakeholder engagement in patient-centered research that uses RL?”; and “Based on these videos, as a patient stakeholder, what would be your expectations about how security and privacy of patient health data would be managed in an RL project?”

Record Linkage Town Hall

In July 2021, we presented the final workbook at a RL town hall and gathered suggestions for dissemination.[43] Town hall participants (N = 58) were patient stakeholders, data managers, data analysts and programmers, researcher investigators, technical professionals, and regulatory compliance representatives. Several final town hall participants were individuals previously engaged in this project.

Town hall attendees were provided with an overview of using novel RL research methods. Afterward, they were shown the workbook and the explainer videos and were asked: “As we finalize the workbook, to what extent should it be agnostic to RL tools such as CURL or specific to preparing to use CURL?”; “What types of local resources might be needed to support users of the workbook?”; and “How might you use these videos in the context of engaging stakeholders in research?” Additionally, we asked questions specific to distribution planning, such as, “With whom should we share these materials?; How might we reach them?”; and “When and where do people learn about RL methods?”

Results

Patient Engagement Insights

The patient partner recommended key design features of RL dissemination package components designed for patient stakeholder representatives. First, the “KIS” principle (keep it simple) was suggested as a guide for all plain language materials. Second, the patient partner emphasized that the content should convey two main points: why it was beneficial to use patient data in research (i.e., the types of research that could be conducted and the advances in health care that research could lead to) and what rules researchers must follow to protect patient data used in research (e.g., Health Insurance Portability and Accountability Act [HIPAA] rules, need for informed consent under certain circumstances). The broader set of patient stakeholders subsequently engaged during the design process corroborated the importance of these factors. Thus, the messages designed for patient stakeholder audiences focused on plain language messaging about the benefits to society of RL research and the protections in place to ensure patient data privacy and security.

Animated Videos: Patient Data and Privacy Regulations and Privacy-Protected Record Linkage

Patient videos included patient-oriented content and language, such as the benefits of RL methods and opportunities for answering important research questions with richer datasets enabled through RL. [Table 1] provides an overview of the RL dissemination package design strategies—including key messages, packaging, and distribution choices. Videos were designed to increase shared understanding between researchers and patients engaged in the planning and conduct of research studies that employ RL methods. The first video focuses on regulations, privacy and security, and processes for using patient health data in research.[38] The second video provides an overview of RL methods and how RL may be used for research.[39]

Table 1 RL dissemination package design elements

Audience segment	Key messages	Packaging (format)	Distribution (communication channels)
Patients engaged in research	• Benefits of research using RL of patient data • Rules governing use of patient data for research	Animated videos using plain language	Patient engagement research activities (in-person/virtual sessions, email, etc.)
Investigators	How to: • Establish opportunity and need for RL • Determine feasibility of RL • Select and describe linkage approach • Establish data-sharing agreements and approvals • Execute RL using CURL	RL Planning Workbook; findings from iPPRL project	Biostatistics and health data groups on campus, and academic societies (e.g., American Medical Informatics Association [AMIA], the North American Primary Care Research Group [NAPCRG], AcademyHealth)
Data analysts	Methods for improving data quality and efficiency	RL Planning Workbook; findings from iPPRL project	Findings published in research journals, conference presentations
Leadership and regulatory officials	Standard language for IRBs and data owners, highlighting how privacy-preserving RL methods protect data privacy and ownership	Resources accompanying RL Planning Workbook, including animated videos	(see above)

Abbreviations: CURL, The University of Colorado Record Linkage; iPPRL, incremental privacy-preserving record linkage; IRB, institutional review board; RL, record linkage.

Patient stakeholder engagement efforts suggested that rules governing patient data use—along with data privacy and security—were among their top concerns. Thus, we incorporated messaging around the rules and processes that govern patient data use for research into patient communication materials. Additionally, patient stakeholder engagement efforts revealed concerns about the initial pacing of the videos—including the use of up-tempo music, which overwhelmed patients, with one describing it as creating “a sense of panic.” Thus, in subsequent iterations, we slowed the pace of the videos to allow viewers more time to digest the information presented. We also made adaptations to meet the needs of viewers who might have vision or hearing impairments by (1) adding subtitles and (2) lowering the volume of the music to make the voiceover more easily heard.

Patient stakeholders found the final drafts of the videos to be clear and well-structured, noting that complex information was presented systematically so that viewers had the needed knowledge and familiarity before moving onto the next topic. Stakeholders felt the videos would be well suited for use as part of the informed consent process, especially for studies with a relatively small number of participants. They also suggested these videos could be used to generate more universal support for and confidence in health care research.

Customer Discovery and Value Proposition Design Insights

Results of the customer discovery interviews revealed jobs to be done, the principal pains and gains, and metrics for success for each of three customer segments. [Table 2] presents insights from the customer discovery process that were used to inform value proposition design. Value proposition statements focused on CURL's ability to standardize the process for sharing data and to establish best practices for RL ([Table 3]).

Table 2 Jobs, pains, gains, and metrics of success identified in customer discovery

	Main themes	Quotes
Jobs
Data managers	Principally perform work surrounding health data engineering, analytics, delivery, and reporting, as well as software and systems development	“I manage the analytics department that does majority of linkages and do some of the linkages myself as well. I oversee projects and staff that do the hands-on programming to complete linkage.”
Researchers	Work involves using health data to identify eligible patients, developing clinical and institutional informatics, and/or designing interventional studies to improve patient health	“I am often integrating solutions for how health data are generated, and then I also collect and use the data to look at outcomes. I am often working with minimum two sources of data and linking the data in a de-identified way.”
Leadership and regulatory officials	Responsible for managing clinical trial agreements and data requests and delivery; overseeing and providing infrastructure for HIPAA compliance; data governance and surveillance; and/or addressing regulatory and legal issues surrounding data privacy	“We enrich datasets using multiple data sources. These datasets are set up primarily for research. We provide secure analytics infrastructure preconfigured for HIPAA compliance standards.”
Pains
Data managers	● Current process for RL is inefficient and time-consuming ● Challenges when requesting others to pull data in the format needed ● Taking steps to ensure that they are not giving out bad information (such as false positives and false negatives)	“When we work with large datasets, we are going to introduce some false positives or false negatives. We don't have the capacity to review everything. We have to be open and honest about our assessment of the quality.”
Researchers	● Establishing data-sharing partnerships and contracts (data use agreements [DUA], business associate agreements [BAA], nondisclosure agreements); navigating legal hurdles ● Communicating privacy and security methods surrounding data sharing (e.g., IRB protocols) ● Determining the most viable RL approach given their research question, resources, and expertise	“The technical side of data sharing is easy; the governance side is hard.” Sharing encrypted personally identifiable information (PII) is the same as sharing PII in the view of a privacy/compliance officer (sentiment of one researcher)
Leadership and regulatory officials	● Understanding the process and how data security and privacy would be protected and conform to HIPAA and other policies ● Require assurances that data sharing is “not opening any back doors”	“Dealing with the health care culture around the paranoia of use of health data across institutional boundaries and getting executive support is key. Do not underestimate the security and compliance efforts; it is much more effort than you think it's going to be.”
Gains
Data managers	● Taking steps to ensure that they are not giving out bad information (such as false positives and false negatives) is important	“Advances in probabilistic matching and solid validated metrics of how certain you are that could be portrayed to clinicians would be helpful. It would be useful to give them a number or percentage of certainty. Standard rules are needed; it's currently very ad hoc.”
Researchers	● Establishing data-sharing partnerships and contracts (DUA, BAA, nondisclosure agreements); navigating legal hurdles ● Communicating privacy and security methods surrounding data sharing (e.g., IRB protocols) ● Determining which RL approach would be most viable given their research question, resources, and expertise available to them	“The goal is to alleviate concerns one might have about are we actually looking at data the correct way? Are we actually looking at the thing we're trying to look at with the data that we have? This would improve the validity of research and the conclusions we make.”
Leadership and regulatory officials	● Understanding the process and how data security and privacy would be protected and conform to HIPAA and other relevant policies ● Require assurances data sharing is “not opening any back doors”	“When I know that there are technical safeguards in place, because humans make mistakes. Computers don't unless we've programmed them wrong. When there are technical safeguards in place, that makes a privacy officer comfortable.”
Metrics of success
Data managers	● Ability to complete RL requests in a timely manner ● Confidence they are not giving out bad information (such as false positives and false negatives)	“Success is timeliness in our ability to do linkages, including timeliness of putting in data-sharing agreements.”
Researchers	● Confidence in navigating the data-sharing process ● Developing “well-trodden paths” for data sharing across universities and outside organizations	“Success is getting quality data. This means being able to get all the data elements you want—this is the challenging part—and low levels of missing data.”
Leadership and regulatory officials	● Research protocols that use CURL clearly demonstrate compliance with data privacy regulations ● Making it easy for leadership to say “yes” to data-sharing/RL research	“Probabilistic linkage is important to us. If CURL helps us keep up with the greatest and latest tech, that would be beneficial for us. We currently don't have the constant improvement that comes from a managed service like CURL.”

Abbreviations: CURL, The University of Colorado Record Linkage; IRB, institutional review board; HIPPA, Health Insurance Portability and Accountability Act; RL, record linkage.

Table 3 Value propositions

For investigators that need to perform record linkage in their research, CURL provides an integrated solution that enables them to perform data normalization, customization, obfuscation, and linkage better than custom, manually coded solutions (which require developing multiple processes and employing multiple tools) providing the benefit of a more efficient method for conducting record linkage with high-volume data

For data managers that need to perform record linkage tasks, CURL provides user-friendly, built-in linkage solutions and standard data specifications to assist the data linkage process, enabling them to perform data normalization, customization, obfuscation, and linkage better than custom, manually coded solutions (which require developing multiple processes and employing multiple tools), providing the benefit of accuracy, flexibility, efficiency, and data security

For institutional leadership and regulatory officials that need to ensure data privacy and security standards are met, CURL provides an IRB-approved solution that meets the federal regulatory requirements for human subjects research and data use that enables them to protect their institution from fiduciary risks associated with protected health information (PHI) data breaches better than resource-intensive manual data and protocol review processes, providing the benefit of state-of-the-art data security and the option for privacy-preserving data sharing

Abbreviations: CURL, The University of Colorado Record Linkage; IRB, institutional review board.

Customer discovery revealed several features to incorporate into the RL dissemination package. First, investigators who will incorporate RL into their protocols need guidance on how to determine which RL approach would be most viable given their research question, resources, and the expertise available to them, as well as data access permissions and constraints. Second, regulatory and security personnel who make decisions about allowing RL to be conducted need to fully understand the process and how the data security and privacy will be protected and conform to HIPAA and other relevant policies. Third, in terms of efficiency and quality of linkage and resulting datasets, analysts and data managers who ultimately perform RLs need to see the benefits of using novel RL methods rather than the manual, custom processes they have traditionally used.

Investigators needed communication materials to facilitate the research development process for those new to conducting research using RL methods and for investigators more experienced with RL who want greater clarity on how to integrate RL more comprehensively in their work. Regulatory personnel value standard language for IRBs and data owners, highlighting how privacy-preserving RL methods protect data privacy and ownership. Finally, for those performing RL, dissemination of the findings from this project should emphasize improvements in data quality and efficiency. These insights informed the development of the RL Planning Workbook.

Record Linkage Planning Workbook

The purpose of the RL Planning Workbook is to inform the integration of RL methods into research projects using CURL ([Table 1]). The workbook guides users through the steps to iteratively plan their RL project ([Fig. 4]). Each stage incorporates key questions—along with definitions and explanation of RL concepts—to help the user ensure they address the decisions critical to conducting research. The workbook is meant to be used in conjunction with other research team members, including statisticians or analysts who will conduct RL tasks and related analyses.

Town hall participants found the RL Planning Workbook useful for identifying solutions and navigating the numerous parameters and restrictions of datasets. They believe the workbook could be used to prepare investigators for productive consultations and collaboration with RL experts—which investigators expect they will need to successfully use RL in their research.

Discussion

A dissemination planning process involving patient engagement, customer discovery, and UX design methods yielded an RL dissemination package that met the needs of audience segments, including patients engaged in RL research and researchers, data managers, leadership, and regulatory officials. For patient stakeholders, animated videos create a shared understanding in a format that is accessible and able to be readily distributed via multiple channels (e.g., email, social media). To adopt RL methods, researchers need guidance and resources to support RL integration into research design. The RL dissemination package provides messaging with this guidance and points researchers to resources where needed. It also provides a common language to help researchers communicate with data managers, leadership, and regulatory officials. The RL workbook provides data managers—who need tools and metrics for assuring the validity of RL results—a pathway for navigating research design and analytic plans with their researcher counterparts. The packaging choice of a workbook allows users to interact with key messages and record their own RL plan. It also can be shared and used online (as a PDF) or printed to enhance distribution. Together the workbook and videos detail the methods for ensuring data privacy and security to meet the information needs of leadership and regulatory officials.

This work builds on prior research through application of a dissemination planning framework[15] and multiple stakeholder engagement in informatics research.[21] [43] Engaging patients and other stakeholders throughout the design process was valuable in designing effective messages and dissemination strategies for CURL. We demonstrate the value of applying patient engagement methods[31]; customer discovery and value proposition design methods[27]; and UX design[28] [29] to dissemination planning by identifying the intersection of stakeholder needs and the CURL platform's effectiveness at meeting those needs. For example, although our initial plans for the patient engagement videos focused on explaining how and why RL is used for health care research, feedback from patient stakeholders demonstrated a high level of concern about data privacy and security. Therefore, we developed one video that primarily focused on explaining the rules and regulations surrounding sharing and using patient data for research.

Limitations and Future Research

Included in the limitations of this work is that a relatively small set of perspectives informed RL dissemination package design. Next steps for this work are to evaluate the impact of the RL dissemination package on adoption and use of RL methods.

Additionally, as informatics tools such as CURL necessarily require iterative development to keep pace with evolving technology, subsequent efforts should be made to update dissemination package materials to reflect such changes.

Conclusion

Guided by the dissemination planning framework, the RL dissemination package was developed to meet the needs of diverse stakeholders. By using an iterative approach that incorporated user feedback throughout the development process, we were able to produce materials that fit the context of PCOR that employs RL methods. This work advances the science of patient engagement in informatics research by demonstrating the utility of engaging patients in RL methods dissemination design process to establish such methods as acceptable, appropriate, and useful.

Clinical Relevance Statement

PCOR and other types of clinical and translational research can be advanced through the use of novel RL methods, but investigators need resources to support the integration of these methods into their work. RL dissemination provides the resources necessary to facilitate the use of RL methods to enhance clinical outcomes research—ultimately, improving the translation of clinical outcomes data to health care practice. Patient engagement in the design process helps to ensure methods are developed and used with patient perspectives in mind. Furthermore, patient stakeholders engaged in research using RL methods may benefit from simple explainer videos to prepare them for this role.

Multiple-Choice Questions

1. Value proposition design methods include the identification of potential customers'_____?

   • Pains and gains

   • Needs and wants

   • Benefits and barriers

   • Challenges and facilitators

   Correct Answer: The correct answer is option a. Value proposition design focuses on product alignment with creating gains and relieving pains for customers.

2. How did patient stakeholder engagement improve the value of the RL Dissemination Package in this study?

   • Patient stakeholders generated ideas for how to best market CURL to researchers

   • Patient stakeholders submitted their data as test cases for using CURL

   • Patient stakeholders shared their concerns about the use of patient data and contributed ideas as to how best to communicate the value of using patient data for research

   • Patient stakeholders refused to support this work

   Correct Answer: The correct answer is option c. As promoting patient engagement in RL research was a key aim of this work, it was important to identify strategies for communicating with patient stakeholders about the methods and implications of RL.

• References

• 1 Meystre SM, Lovis C, Bürkle T, Tognola G, Budrionis A, Lehmann CU. Clinical data reuse or secondary use: current status and potential future progress. Yearb Med Inform 2017; 26 (01) 38-52
• 2 Ong TC, Kahn MG, Kwan BM. et al. Dynamic-ETL: a hybrid approach for health data extraction, transformation and loading. BMC Med Inform Decis Mak 2017; 17 (01) 134
• 3 Jiang G, Kiefer RC, Sharma DK, Prud'hommeaux E, Solbrig HR. A consensus-based approach for harmonizing the OHDSI common data model with HL7 FHIR. Stud Health Technol Inform 2017; 245: 887-891
• 4 Kahn MG, Callahan TJ, Barnard J. et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC) 2016; 4 (01) 1244
• 5 Richter A, Schmidt CO, Krüger M, Struckmann S. DataquieR: assessment of data quality in epidemiological research. J Open Source Softw 2021; 6 (61) 3093
• 6 Boyd JH, Randall SM, Ferrante AM. Application of privacy-preserving techniques in operational record linkage centres. Med Data Privacy Handbook Switzerland: Springer International Publishing; 2015
• 7 Brown AP, Ferrante AM, Randall SM, Boyd JH, Semmens JB. Ensuring privacy when integrating patient-based datasets: new methods and developments in record linkage. Front Public Health 2017; 5: 34
• 8 Kohane IS, Aronow BJ, Avillach P. et al; Consortium For Clinical Characterization Of COVID-19 By EHR (4CE). What every reader should know about studies using electronic health record data but may be afraid to ask. J Med Internet Res 2021; 23 (03) e22219
• 9 Ong TC, Duca LM, Kahn MG, Crume TL. A hybrid approach to record linkage using a combination of deterministic and probabilistic methodology. J Am Med Inform Assoc 2020; 27 (04) 505-513
• 10 Lim D, Randall S, Robinson S. et al. Unlocking potential within health systems using privacy-preserving record linkage: exploring chronic kidney disease outcomes through linked data modelling. Appl Clin Inform 2022; 13 (04) 901-909
• 11 Ross MK, Sanz J, Tep B, Follett R, Soohoo SL, Bell DS. Accuracy of an electronic health record patient linkage module evaluated between neighboring academic health care centers. Appl Clin Inform 2020; 11 (05) 725-732
• 12 Mason LG. A Comparison of Record Linkage Techniques. Statistical Survey Paper. Quarterly Census of Wages and Employment (QCEW). 2018 Accessed August 31, 2022 at: https://www.bls.gov/osmr/research-papers/2018/st180060.htm
• 13 Hall R, Fienberg SE. Privacy-preserving record linkage. In: Privacy in Statistical Databases. Berlin, Heidelberg: Springer; 2010: 269-283
• 14 Conway RBN, Armistead MG, Denney MJ, Smith GS. Validating the matching of patients in the linkage of a large hospital system's EHR with state and national death databases. Appl Clin Inform 2021; 12 (01) 82-89
• 15 Kwan BM, Brownson RC, Glasgow RE, Morrato EH, Luke DA. Designing for dissemination and sustainability to promote equitable impacts on health. Annu Rev Public Health 2022; 43 (43) 331-353
• 16 Hartzler A, McCarty CA, Rasmussen LV. et al. Stakeholder engagement: a key component of integrating genomic information into electronic health records. Genet Med 2013; 15 (10) 792-801
• 17 Baumann BL, Kolko DJ, Collins K, Herschell AD. Understanding practitioners' characteristics and perspectives prior to the dissemination of an evidence-based intervention. Child Abuse Negl 2006; 30 (07) 771-787
• 18 Zimmerman LP, Goel S, Sathar S. et al. A novel patient recruitment strategy: patient selection directly from the community through linkage to clinical data. Appl Clin Inform 2018; 9 (01) 114-121
• 19 Morrato EH, Hamer MK, Sills M, Kwan B, Schilling LM. Applying a commercialization-readiness framework to optimize value for achieving sustainability of an electronic health data research network and its data capabilities: the SAFTINet experience. EGEMS (Wash DC) 2019; 7 (01) 48
• 20 Concannon TW, Meissner P, Grunbaum JA. et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J Gen Intern Med 2012; 27 (08) 985-991
• 21 Lopez MH, Holve E, Rein A, Winkler J. Involving patients and consumers in research: new opportunities for meaningful engagement in research and quality improvement. In: EDM Forum,. Washington, DC: AcademyHealth; 2012
• 22 Lavallee DC, Wicks P, Alfonso Cristancho R, Mullins CD. Stakeholder engagement in patient-centered outcomes research: high-touch or high-tech?. Expert Rev Pharmacoecon Outcomes Res 2014; 14 (03) 335-344
• 23 McGuire WJ. McGuire's Classic Input–Output Framework for Constructing Persuasive Messages. Public Communication Campaigns. 2013. Thousand Oaks, CA: SAGE Publications; 133-145
• 24 Ong TC, Hill AC, Grannis S, Voong C, Kahn MG. Linkability assessment for demographics data. In: AMIA 2020 Virtual Informatics Summit Proceedings. American Medical Informatics Society; 2020
• 25 Ong TC, Hill AC, Kahn M, Grannis S, Voong C, Shilling L. Privacy-Preserving Transitive Record Linkage. In: AMIA 2021 Virtual Informatics Summit Proceedings. American Medical Informatics Society; 2021
• 26 CU record linkage. University of Colorado record linkage.
• 27 Osterwalder A, Pigneur Y, Bernarda G, Smith A. Value Proposition Design: How to Create Products and Services Customers Want. John Wiley & Sons; 2015
• 28 Gray CM. “It's More of a Mindset Than a Method”: UX Practitioners' Conception of Design Methods. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2016: 4044-4055
• 29 Hartson R, Pyla PS. The UX Book. Agile UX Design for a Quality User Experience. Morgan Kaufmann 2018
• 30 plainlanguage.gov / What is plain language?. Accessed January 1, 2023 at: https://www.plainlanguage.gov/about/definitions/
• 31 Kirwan JR, de Wit M, Frank L. et al. Emerging guidelines for patient engagement in research. Value Health 2017; 20 (03) 481-486
• 32 CU Record Linkage. iPPRL value proposition design customer discovery phone interviews: investigators, researchers, and clinical operations. CU Anschutz Digital Collections 2021
• 33 CU Record Linkage. iPPRL value proposition design customer discovery phone interviews: data manager/technician. CU Anschutz Digital Collections 2021
• 34 CU Record Linkage. iPPRL value proposition design customer discovery phone interviews: leadership and regulatory. CU Anschutz Digital Collections 2021
• 35 Scott KW, Howell D. Clarifying analysis and interpretation in grounded theory: using a conditional relationship guide and reflective coding matrix. Int J Qual Methods 2008; 7 (02) 1-15
• 36 Kepper MM, Walsh-Bailey C, Brownson RC. et al. Development of a health information technology tool for behavior change to address obesity and prevent chronic disease among adolescents: designing for dissemination and sustainment using the ORBIT model. Front Digit Health 2021; 3: 648777
• 37 Haddad R, Badke D'Andrea C, Ricchio A. et al. Using Innovation-Corps (I-Corps™) Methods to Adapt a Mobile Health (mHealth) Obesity Treatment for Community Mental Health Settings. Front Digit Health 2022; 4: 835002
• 38 CU record linkage: patient data and privacy regulations. CU Anschutz Digital Collections 2021
• 39 CU record linkage: privacy-protected record linkage. CU Anschutz Digital Collections 2021
• 40 CU Record Linkage. Record linkage for investigators. CU Anschutz Digital Collections 2021
• 41 CU Record Linkage. Health data linkage workshop materials. CU Anschutz Digital Collections 2021
• 42 Lewis C. Using the” thinking-aloud” method in cognitive interface design. Yorktown Heights, NY: IBM T.J. Watson Research Center; 1982
• 43 Meissner P, Cottler LB, Eder MM, Michener JL. Engagement science: the core of dissemination, implementation, and translational research science. J Clin Transl Sci 2020; 4 (03) 216-218