Patients as Agents in Behavioral Health Research and Service Provision: Recommendations to Support the Learning Health System

Abstract

Authentic inclusion and engagement of behavioral health patients in their care delivery and in the process of scientific discovery are often challenged in the health care system. Consequently, there is a growing need to engage with and better serve the needs of behavioral health patients, particularly by leveraging health information technologies. In this work, we present rationale and strategies for improving patient engagement in this population in research and clinical care. First, we describe the potential for creating meaningful patient–investigator partnerships in behavioral health research to allow for cocreation of knowledge with patients. Second, in the context of behavioral health services, we explore the utility of sharing clinical notes to promote patients' agency in care delivery. Both lines of inquiry are centered in a Learning Health System model for behavioral health, where patients are agents in enhancing the therapeutic alliance and advancing the process of knowledge generation. Recommendations include genuinely democratizing the health care system and biomedical research enterprise through patient-centered information technologies such as patient portals. In research and technology development, we recommend seeking and tailoring behavioral health patients' involvement to their abilities, promoting patient input in data analysis plans, evaluating research and informatics initiatives for patients and clinicians, and sharing success and research findings with patients. In clinical practice, we recommend encouraging patients to read behavioral health notes on portals, engaging in proactive communication regarding note content, assessing outcomes including stress and anxiety in response to note content, and working with technology providers to support note-sharing governance and deployment.

Keywords

mental health - patient participation - medical records - biomedical research - patient engagement - patient collaboration - OpenNotes - shared chart

Introduction

Nearly one in five U.S. adults lives with a behavioral health condition.[1] [2] During the past decade, there has been a growing awareness of the need to better utilize health information technologies to address the needs of this population, including broader access to care, improved outcomes, and effective research partnerships. Patients with behavioral health diagnoses are more likely to have experienced marginalization in the care delivery process.[3] Similarly, they are often excluded from the process of driving scientific discovery to treat the very disorders they experience.[4] At the same time, clinical professionals have recognized the value of using informatics tools and processes to rapidly improve care processes and, thereby, patient outcomes, through use of the Learning Health System model.[5] [6] A Learning Health System is defined as one in which “science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the care process, patients and families active participants in all elements, and new knowledge captured as an integral by-product of the care experience.”[5] Through a focus on person-centered care, inclusiveness, adaptability, and cooperative and participatory leadership, this model can not only support but also facilitate a more proactive role for patients and their caregivers in research and clinical practice. Clinicians and institutional researchers have an important role in advancing practice that is more patient-focused and research that is better aligned with patient needs and system goals, and that leverages clinical informatics to achieve these objectives.

The Learning Health System model frames the process of driving scientific discovery and knowledge creation as an organic outgrowth of patient care. In mental health, patient/client input, views, values, and engagement are essential to both provision of high-quality clinical care and successful formulation of appropriate and relevant research questions. It seems likely that the coproduction of patient notes, possibly with the physician-researcher assigning defined activities to the patient or caregiver (e.g., charting and tracking anxiety level via a mobile device to encourage proactive symptom identification) will facilitate a more nuanced picture of patient engagement in a Learning Health System. In addition, the availability of notes through patient portals enhances the value of informatics-based technologies to patients, and may encourage them to interact with such technologies more effectively.

In this article, we address two specific ways in which Learning Health Systems can utilize technology to improve patient engagement in this population to improve clinical outcomes and advance consumer and clinical informatics. Specifically, we describe patients as research partners and explore sharing clinical notes to promote patients as agents within a person-centered care model. These examples illustrate how approaches grounded in informatics support both research and patient care objectives.

Patients as Research Partners

Historically, inclusion of patients as partners has been perceived as a threat to the authenticity and validity of research, and the involvement of individual patients as representative of populations as suspect.[7] [8] Research involves several stages, including project design and development, execution, data analysis and interpretation, results dissemination, and practice change, and individual patients may be better suited to some aspects than others. However, some researchers remain unconvinced that patient involvement can make a positive difference.[9] [10] Underserved patients[11] can be difficult to recruit as research partners,[12] or even as research participants.[13] This challenge is substantially exacerbated in behavioral health research because of distrust in research and stigma around these illnesses.[14] [15] On the other hand, in patient-centered outcomes research (PCOR) initiatives, tokenistic involvement of patients is frequently perceived as a potential negative aspect.[16] [17] [18] [19] [20] Patient partners themselves have expressed the concern that too much patient influence can negatively affect research quality and study generalizability and lead to a discounting of researchers' perspective.[21] Additionally, patients who have the time and interest in collaborating with researchers may not represent the larger patient population.

Researchers, health care professionals, and patients have varying ideas about the meaning of patient participation in a behavioral health context.[22] Researchers cite multiple reasons to engage patients as research partners, such as improving capacity for PCOR among both researchers and patients[19] and developing better patient governance in research involving patients.[23] Patients are particularly well positioned to assist with research proposal development because they can identify topics that are meaningful and relevant to patients and their caregivers. To date, patient–researcher collaboration around identification of research questions has been scant; of more than 250 publications addressing research priority identification, patients and researchers worked together to identify questions in just 9 works.[24] However, patient involvement in most or all phases of research is not a universal practice. A survey of Canadian PCOR researchers reported less involvement of patient partners in data collection and analysis and dissemination of results.[19] In implementation research, investigators view patient involvement in research positively, though there is less support for patient involvement in advising on intervention sustainability poststudy.[20] This suggests that although researchers are open to engaging patients in some activities they are not necessarily on board with implementing patients' ideas about practice change.

Patients have taken partner roles as users of behavioral health services,[25] [26] but collaboration in behavioral health research has been more limited to date. This is partly due to the challenge of recognizing the difference between patients as research participants and as research partners.[19] Activities that involve engagement with patients, such as clinical trials, may create two-way communication that researchers mistake for partnering. For example, patient-reported outcome measure (PROM) use has become common in clinical trials, including trials that investigate mental health interventions.[27] [28] [29] [30] The use of PROMs can create a feedback mechanism that researchers interpret as collaborative communication rather than symptom reporting. The U.S. Department of Veterans Affairs has developed, tested, and evaluated 15 mobile health apps intended to address the needs and goals of veterans who have posttraumatic stress disorder (PTSD), including 7 apps designed for use with clinician-based treatment modalities.[31] In these examples, patient viewpoints are commissioned, but they are not considered truly co-collaborators, which limits the effectiveness of their participation.

Recruiting and retaining patient partners who are representative of a particular patient population and overcoming the stigma attached to some conditions make partnering with patients in research more difficult.[19] In general, recruiting and retaining enrollees in research addressing mental health concerns may be more difficult than for research related to other medical conditions, and this aspect of research merits particular consideration. Patients often have many roles and responsibilities to juggle in their daily lives and may be limited in the time they can devote to research partnerships, despite having a strong interest in doing so.[21] Online tools that allow behavioral health patients to provide feedback about services and issues of interest exists,[32] but this approach does not support one-to-one patient–partner engagement in research. True collaboration with patients may require researchers to cede some control over the work, which researchers may find uncomfortable or problematic in some circumstances.

The Patient-Centered Outcomes Research Institute expects investigators to include patients in research from the initial planning through the dissemination of findings as a requirement for funding of research proposals.[33] In practice, patients have assumed a broad range of roles including idea generation, study design, participant recruitment, data collection, data analysis, results dissemination, and manuscript preparation.[17] Patients have also taken an active role in designing large multicenter clinical trials.[34] In-person multistakeholder groups have emerged as an effective way to involve patients in prioritizing research topics and developing a research agenda.[35] [36] [37] As with any other relationships involving patients, building trust and relationships are critical to success.[21] Patient-partners in research value developing relationships with researchers that go beyond the typical patient–clinician relationship and involve greater collegiality. Partnering with patients is a long-term process in which value accrues over time as those involved gain experience working together and build trust.[38]

Much of what has been said of patients can be extended to family, caregivers, guardians, and even to advocates in certain circumstances. As in the case of pediatric research or in progressive degenerative conditions, family representatives or caregivers may share the role of research partner as the insights and needs of the immediate family or caregiver(s) are also of important value.

Novel Patient-Researcher Partnerships in Technology-Enabled Behavioral Health Research

Behavioral health researchers have sought to define the nature of user involvement in behavioral health services for some time.[39] [40] One such collaboration is underway at 25 emergency departments (EDs) in Ontario, Canada. In this cluster randomized controlled trial, researchers are designing an intervention to prevent suicide among men through collaboration with individuals who were treated in EDs after episodes of self-harm.[41] Patients were randomized by ED to receive either 7 sessions of in-person psychotherapy over 6 months with 12 months' use of a smartphone app designed for men who self-harm (10 hospitals) or the standard treatment (15 hospitals). The proportion of repeat visits to the ED for episodes of self-harm and/or suicide in the year following the opening of the study is the primary outcomes of the study. The researchers are also building a self-sustaining committee of people with lived experience who have interest in suicide prevention research and have agreed to act as coinvestigators.

People who have considered the use of mobile health (mHealth) apps for management of mental health conditions report both acceptance and rejection of specific features commonly included in apps,[42] suggesting that patient involvement in design, user experience testing, and evaluation can improve the utility of patient-facing technologies. Such involvement also helps to ensure that apps function appropriately and effectively with existing informatics tools such as electronic health records (EHRs) and computerized physician order entry. The Veterans Affairs (VA) incorporates human-centered design principles into the technologies it develops and includes patients and caregivers among those evaluating these tools to optimize tool usability and, thereby, promote use by patients and clinicians.[31] The American Psychiatric Association similarly facilitates user feedback from people with mental illness via its App Advisor Expert Panel.[43]

In another ongoing initiative, researchers are involving patients and their family members in the design and execution of a mixed methods study that will evaluate the value of behavioral health patient portals.[44] Using portal usage data, a literature review, patient/family focus groups, a daylong symposium, and outcomes data, researchers and their patient counterparts are cocreating a resource describing best practices for patient–researcher partnerships applicable to technology-based research initiatives in a broad range of clinical areas.[45] Such collaborations will improve the potential for developing patient- and clinician-friendly tools that support Learning Health System goals to align research and clinical care in an informatics-based ecosystem to drive a cycle of continuous advancement.

Sharing of Clinical Notes to Support Patients as Agents

As such, the role of patient engagement is just as crucial in practice and care-related activities. Just as in research, clinician regard of patients as true partners has failed to occur in some environments despite policies intended to support such collaboration.[46] [47] [48] In clinical practice, it is clear that the development of a therapeutic alliance is associated with improved behavioral health care outcomes.[49] [50] [51] Additionally, the shift toward a more person-centered care model, including an asset-based, collaborative process and shared decision-making, are thought to improve engagement. In turn, this increased engagement stimulates continuous improvement and innovation as part of the Learning Health System. Therefore, such models are being adopted as a clinical paradigm shift in several behavioral health settings.[49] [52]

Given this data, it can be postulated that increasing transparency by facilitating patients' access to their EHRs, particularly the clinical notes, would not only engender trust and strengthen the therapeutic alliance, but also encourage adoption of this person-centered care model, fostering patient engagement and adherence to a mutually agreed-upon plan of care. One example in which emerging technologies were leveraged to address this goal was the OpenNotes Project, a 2010 multicenter trial involving more than 100 primary care providers at three sites, allowed patients access to their EHR notes via the patient portal. More than 80% of patients accessed at least one note, with most reporting that doing so helped them to feel more in control of their care. Moreover, the authors reported an increase in medication adherence across all sites with the introduction of the novel framework.[53]

Although clinicians expressed concern that patients would find the notes confusing, worrisome, or offensive, few patients reported these concerns. Additionally, providing ready access to the notes via the portal rarely seemed to require more time for questions during the visit or lengthen the time spent documenting the visit, though many clinicians reported changing the way their notes addressed sensitive behavioral health issues such as substance abuse, as well as cancer and obesity. At the end of the study period, almost all patients expressed a desire to continue having access to their notes. None of the pilot clinicians opted out, and a majority indicated that opening their notes strengthened their relationships with some of their patients, noting improved trust, better communication, and shared decision making.[53]

In a follow-up survey 7 years later, patients who read their notes continued to report that doing so was very important to feeling in control of their care, remembering their plan of care, preparing for and making the most out of their visit, taking an active role in their care, and taking care of their health in general.[54] These benefits were more prevalent in patients who were less educated, older, non-white, and who usually did not speak English at home. Patients were more likely to read their notes if their clinicians encouraged them to do so, and few reported that the documentation sparked confusion or worry. Despite of these overall positive results, many institutions adopting the OpenNotes framework initially excluded behavioral health documentation due to concerns that access to such material would impede the therapeutic alliance and cause unnecessary anxiety or distress.

Clinicians' Perspectives on OpenNotes

In 2013, behavioral health notes were first made available to veterans through the Blue Button function in MyHealtheVet, the VA's online patient portal.[55] A survey of clinicians' attitudes around this development noted shifting power within the patient–clinician relationship toward more person-centered care. However, some clinicians expressed concerns about certain patients “dictating” what could or could not be written in the record. Many clinicians expressed concerns around the therapeutic process, including the efforts and nuance that go into rapport and trust-building. As clinicians may document aspects of the therapeutic process that often are not communicated to the patient, some clinicians worried that patients could negatively misinterpret these notations, which could adversely affect the therapeutic bond. However, some clinicians saw OpenNotes as an opportunity to enhance communication and therefore feelings of trust and transparency, which could positively impact the therapeutic relationship, even if there were disagreements regarding specific material. Finally, clinicians expressed a desire for system-level guidance on documentation practices to reduce the potential for adverse outcomes and legal consequences for both themselves and their patients. Specifically, they articulated ambivalence around the desire to document in a clinically accurate way while protecting patients from potential harm, and concern about the medicolegal risk in adjusting their practice accordingly.[56] [57]

Patients' Perspectives on OpenNotes

Interviews with VA behavioral health patients who had accessed their OpenNotes demonstrated that a diagnosis of PTSD, communication with their mental health clinician about the ability to view notes, and being more confident in the uses of Blue Button were independently associated with reading notes.[58] Those patients who did so felt that reading their notes improved their health knowledge and self-efficacy, fostered the relationship with their clinician(s), and was of high value. Patients also reported a belief that this relationship and mutual trust was critical to the therapeutic process. They felt this trust was strengthened by the perceived level of transparency and respect in clinicians' notes, but strained when transparency was low or when incongruences were noted in the absence of direct discussion. Additionally, although the proportion of behavioral health patients that reported being “upset” by their notes was similar to primary care, about a third of patients reported stress or worry. Those with PTSD had higher incidences of these types of negative emotional responses to OpenNotes, but also reported stronger positive alliances with their clinicians after reading their notes. Notably, a diagnosis of a psychotic, bipolar, or a personality disorder demonstrated no significant association with a negative emotional response to OpenNotes.[55] [59]

In March 2014, Beth Israel Deaconess Medical Center in Boston, United States, also began sharing behavioral health notes with 360 patients, publishing the organization's belief that doing so could benefit those with a spectrum of disorders as well as meaningfully influence the way clinicians frame their own thinking about their patients.[60] Since then, almost 60 organizations throughout the United States and Canada have started sharing behavioral health notes with patients.[61] Region Skåne in southern Sweden also began sharing its adult psychiatry notes in 2015, though still excluding information that could pose a risk to the patient or family through use of a special template, and delaying access to inpatient psychiatric notes for 4 weeks after discharge. Postimplementation surveys of clinicians demonstrated few adverse effects, with some clinicians believing their patients felt more in control of their health, that access to these notes improved patient trust, and a few even actively using the tool therapeutically.[62] [63]

Recommendations for Research and Practice

Based on the evidence to date and in support of both a person-centered care model and continuous advancement through the Learning Health System, we recommend the following approaches for including patients in behavioral health research:

• Seek opportunities to engage patients, families, and caregivers in research question identification, study development, research implementation, and informatics initiatives such as development of mobile apps, patient portals, and patient-centered clinical decision support tools. Patients and advocates may be able to explain more clearly and with greater conviction what outcomes should really matter, may note unconscious biases in research design, may offer effective ideas on recruitment, and may contribute other perspectives that promote the field. Existing approaches of patient and public involvement in health care offer narrow and/or limited opportunities for patients,[64] and their particular expertise is well-suited for improving patient-centered research. People living with mental health conditions have successfully completed multiphase research priority setting initiatives, and engagement from the earliest stages facilitates design of a research agenda that complements clinical efforts.[65]

• Embrace research retention practices that facilitate longer-term studies, which can provide outcomes that are more demonstrative of what patients may experience in the real world. Retention is particularly challenging in behavioral health research,[66] and patients can offer useful perspective on study design and implementation to support long-term retention.[67]

• Tailor patients' involvement to their abilities so patient engagement is possible.[68] A patient with mild dementia may not be able to read and absorb a long document or follow a conference call discussion, but may be able to participate in a focus group or provide valuable input about specific questions.

• Involve patients in data analysis, and listen to their ideas with an open mind. Career researchers bring a particular mindset to data analysis, even when such data takes qualitative forms. Patient involvement in analyzing qualitative data collected in a mental health research project resulted in an expanded interpretation of the referral process, more nuanced recommendations related to positive risk taking and punitive discharge in behavioral health care, and a novel observation about the effect of clinical support on decision-making related to medications.[69]

• Evaluate the success of research initiatives and the impact of increased engagement on health outcomes for both patient partners and clinical beneficiaries of these advances. Then, use this information to improve the design and implementation of future initiatives. Research methods in use to date are highly varied and nonstandardized,[70] which may delay their widespread adoption by researchers and Learning Health Systems and the integration of research results into clinical workflows.

• Share the findings of research with patients, families, and caregivers who were involved in the initiatives. Seeing the results of their efforts reinforces their role as contributing members of the research team and the value of their perspective to the outcome.

We additionally recommend opening behavioral health notes to patients in accordance with the following guidelines to promote better outcomes and increased clinical engagement in support of the Learning Health System.:

• Encourage patients to read their notes; this will both remind them of the note availability on the portal and affirm clinician endorsement of the process. Increasing the rate of patient note viewing can be a challenge, but it appears that clinician discussion of note availability can positively impact this.[54] [58] [71]

• Engage in proactive communication with patients regarding note content to improve transparency and trust and promote the therapeutic alliance, known to improve patient outcomes.[49] [50] [51] [55]

• Pay close attention to potential reactions of stress or anxiety in response to note content, particularly in patients with PTSD.[59]

• Where appropriate, use note content as a means of psychoeducation, explaining technical verbiage or content in lay terms.

• Avoid labels that may be perceived as judgmental and instead use language that highlights the patient's individuality and strengths. Including unique details from each session may help patients to feel heard and understood, and documenting progress may feel motivating.[55] [72]

• Pay attention to details. Patients may view spelling errors, incongruences (including a slight mismatch between Systematized Nomenclature of Medicine and International Classification of Diseases, Tenth Revision codes, which may need to be explained), and use of copy-forward functionality as a lack of professionalism that may impede the therapeutic relationship. Additionally, although clinicians may opt for less detailed documentation to minimize the risk for misinterpretation and to preserve patient privacy from others who may gain access to the chart, patients surveyed expressed a preference for more thorough notes.[72]

• When there is institutional concern that note content may adversely affect patient condition or concern for medicolegal risk, consider utilizing either an “opt-out” or an “opt-in” approach for clinicians sharing behavioral health notes. An “opt-out” approach will likely result in higher participation, but a more conservative “opt-in” approach may be preferable if staff have significant reservations. This allows clinicians control over the sharing of documentation, while fostering a cultural paradigm shift. We encourage utilizing this strategy as an adoption grace period toward default note sharing, rather than as a permanent solution.

• Urge EHR vendors to design additional technology to support specific note-sharing use cases, such as delayed release of notes after an inpatient behavioral health admission or if a patient screens positive for suicidality.[62] [63]

• Encourage behavioral health and informatics professional societies to release guidelines to support clinicians in appropriate and defensible documentation practices in a more transparent environment.

Conclusion

In behavioral health, patients' views, values, and engagement are essential for providing high-quality clinical care. In addition, collaboration with patients can serve to advance all aspects of research and quality improvement, including formulation of appropriate and relevant study questions to design, execution, and dissemination of research and related health informatics initiatives. In this article, we have given discrete examples of ways the health care ecosystem can better adopt frameworks and methodologies to partner with patients as key stakeholders and support patients as agents within a person-centered care model. This would serve to drive advancement of the field and enhancement of the therapeutic alliance to improve clinical outcomes. In doing so, this framework supports the continuous and seamless process of care, discovery, and innovation embodied by the Learning Health System. We recommend continued attention to opportunities for behavioral health patient engagement and partnership as well as guidelines from health professional societies to support these activities.

Clinical Relevance Statement

This work reviews the current state of patient engagement in behavioral health practice and research. It provides recommendations for engaging with patients in ways that support improved health outcomes and more meaningful and successful involvement in clinical research.

Multiple Choice Questions

1. Inclusion of patients as partners in behavioral health research has been minimal for all except the following:

   • Patient investigators are perceived as making research less authentic and valid.

   • Patients are unwilling to coproduce knowledge with researchers.

   • Individual patients are thought to be unable to represent patient populations.

   • Patients willing to act as research partners are hard to find.

   Correct Answer: The correct answer is option b. Not all patients have the time and/or interest to coproduce evidence through research with investigators, but patients act as research coinvestigators in many circumstances (e.g., in Patient-Centered Outcomes Research Institute-funded projects).

2. Clinical notes-sharing (e.g., the OpenNotes initiative) benefits patients through all of the following except:

   • Sharing clinical notes engenders trust on the part of patients.

   • Sharing clinical notes supports a patient-centered care model.

   • Availability of clinical notes makes it possible to see patients less frequently.

   • Sharing clinical notes helps patients better understand their condition and improve their self-efficacy.

   Correct Answer: The correct answer is option c. Patients may perceive decreased need to visit clinicians as a result of being able to view their notes, but simply being access to read clinical notes will not reduce the quantity of care patients need.

Conflict of Interest

A.S. reports grants from Patient-Centered Outcomes Research Institute (PCORI), grants from National Institutes of Health, during the conduct of the work. V.S. reports grants from National Science Foundation, during the conduct of the work.

Authors' Contributions

H.K.G., C.P., V.S., and A.S. wrote the first draft and revised the manuscript. This work is a collaborative effort between the AMIA Ethical, Legal, and Social Issues Working Group and the Mental Health Informatics Working Group.

Protection of Human and Animal Subjects

This work involved no humans or animals, and so was not subject to institutional review board oversight.

• References

• 1 National Institute of Mental Health. Mental Health. National Institute of Mental Health; 2019. Available at: https://www.nimh.nih.gov/health/statistics/mental-illness.shtml . Accessed February 3, 2019
• 2 United States Census Bureau. Quick Facts. United States Census Bureau; 2018. Available at: https://www.census.gov/quickfacts/fact/table/US/PST045218 . Accessed June 12, 2019
• 3 Rössler W. The stigma of mental disorders: a millennia-long history of social exclusion and prejudices. EMBO Rep 2016; 17 (09) 1250-1253
• 4 Faulkner A, Thomas P. User-led research and evidence-based medicine. Br J Psychiatry 2002; 180: 1-3
• 5 Institute of Medicine. Best Care at Lower Cost: The Path to Continuously Learning Health Care in America; 2013. Available at: https://www.nap.edu/catalog/13444/best-care-at-lower-cost-the-path-to-continuously-learning . Accessed July 16, 2019
• 6 Rubin JC, Silverstein JC, Friedman CP. , et al. Transforming the future of health together: the learning health systems consensus action plan. Learn Health Syst 2018; 2 (03) e10055
• 7 Roos EC. Patients in research: still many roadblocks. BMJ 2018; 363: k4387
• 8 Maguire K, Britten N. “How can anybody be representative for those kind of people?” Forms of patient representation in health research, and why it is always contestable. Soc Sci Med 2017; 183: 62-69
• 9 Oldfield BJ, Harrison MA, Genao I. , et al. Patient, family, and community advisory councils in health care and research: a systematic review. J Gen Intern Med 2019; 34 (07) 1292-1303
• 10 Salvetti M, Lubetzki C, Kapoor R. , et al. Steps toward collective sustainability in biomedical research. Trends Mol Med 2018; 24 (05) 429-432
• 11 Bonevski B, Randell M, Paul C. , et al. Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups. BMC Med Res Methodol 2014; 14: 42
• 12 Suurmond J, Torensma M. Involving underserved patient populations in research. BMJ 2018; 363: k4385
• 13 Paskett ED, Reeves KW, McLaughlin JM. , et al. Recruitment of minority and underserved populations in the United States: the Centers for Population Health and Health Disparities experience. Contemp Clin Trials 2008; 29 (06) 847-861
• 14 Ewalds Mulliez AP, Pomey MP, Bordeleau J, Desbiens F, Pelletier JF. A voice for the patients: evaluation of the implementation of a strategic organizational committee for patient engagement in mental health. PLoS One 2018; 13 (10) e0205173
• 15 Woodall A, Morgan C, Sloan C, Howard L. Barriers to participation in mental health research: are there specific gender, ethnicity and age related barriers?. BMC Psychiatry 2010; 10: 103
• 16 Domecq JP, Prutsky G, Elraiyah T. , et al. Patient engagement in research: a systematic review. BMC Health Serv Res 2014; 14: 89
• 17 Brett J, Staniszewska S, Mockford C. , et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect 2014; 17 (05) 637-650
• 18 Hahn DL, Hoffmann AE, Felzien M, LeMaster JW, Xu J, Fagnan LJ. Tokenism in patient engagement. Fam Pract 2017; 34 (03) 290-295
• 19 Bishop AC, Elliott MJ, Cassidy C. Moving patient-oriented research forward: thoughts from the next generation of knowledge translation researchers. Res Involv Engagem 2018; 4: 23
• 20 Gray-Burrows KA, Willis TA, Foy R. , et al. Role of patient and public involvement in implementation research: a consensus study. BMJ Qual Saf 2018; 27 (10) 858-864
• 21 Leese J, Macdonald G, Kerr S. , et al. ‘Adding another spinning plate to an already busy life’. Benefits and risks in patient partner-researcher relationships: a qualitative study of patient partners' experiences in a Canadian health research setting. BMJ Open 2018; 8 (08) e022154
• 22 Jørgensen K, Rendtorff JD, Holen M. How patient participation is constructed in mental health care: a grounded theory study. Scand J Caring Sci 2018; 32 (04) 1359-1370
• 23 Nowell WB, Curtis JR, Crow-Hercher R. Patient governance in a patient-powered research network for adult rheumatologic conditions. Med Care 2018; 56 (Suppl 10 Suppl 1): S16-S21
• 24 Stewart RJ, Caird J, Oliver K, Oliver S. Patients' and clinicians' research priorities. Health Expect 2011; 14 (04) 439-448
• 25 Gillard SG, Edwards C, Gibson SL, Owen K, Wright C. Introducing peer worker roles into UK mental health service teams: a qualitative analysis of the organisational benefits and challenges. BMC Health Serv Res 2013; 13: 188
• 26 Rose D, MacDonald D, Wilson A, Crawford M, Barnes M, Omeni E. Service user led organisations in mental health today. J Ment Health 2016; 25 (03) 254-259
• 27 Rollman BL, Herbeck Belnap B, Abebe KZ. , et al. Effectiveness of online collaborative care for treating mood and anxiety disorders in primary care: a randomized clinical trial. JAMA Psychiatry 2018; 75 (01) 56-64
• 28 Cohen RM, Greenberg JM, IsHak WW. Incorporating multidimensional patient-reported outcomes of symptom severity, functioning, and quality of life in the Individual Burden of Illness Index for Depression to measure treatment impact and recovery in MDD. JAMA Psychiatry 2013; 70 (03) 343-350
• 29 Steiner AJ, Recacho J, Vanle B. , et al. Quality of life, functioning, and depressive symptom severity in older adults with major depressive disorder treated with citalopram in the STAR*D study. J Clin Psychiatry 2017; 78 (07) 897-903
• 30 Yamaguchi S, Ojio Y, Koike J. , et al. Associations between readmission and patient-reported measures in acute psychiatric inpatients: a study protocol for a multicenter prospective longitudinal study (the ePOP-J study). Int J Ment Health Syst 2019; 13: 40
• 31 Owen JE, Kuhn E, Jaworski BK. , et al. VA mobile apps for PTSD and related problems: public health resources for veterans and those who care for them. mHealth 2018; 4: 28
• 32 Sylvia LG, Hearing CM, Montana RE. , et al. MoodNetwork: an innovative approach to patient-centered research. Med Care 2018; 56 (Suppl 10 Suppl 1): S48-S52
• 33 Patient-Centered Research Outcomes Institute. What You Need To Know To Apply. Patient-Centered Outcomes Research Institute; 2019 . Available at: https://www.pcori.org/funding-opportunities/what-you-need-know-apply . Accessed April 9, 2019
• 34 Faulkner M, Alikhaani J, Brown L. , et al. Exploring meaningful patient engagement in ADAPTABLE (aspirin dosing: a patient-centric trial assessing benefits and long-term effectiveness). Med Care 2018; 56 (Suppl 10 Suppl 1): S11-S15
• 35 Kim KK, Khodyakov D, Marie K. , et al. A novel stakeholder engagement approach for patient-centered outcomes research. Med Care 2018; 56 (Suppl 10 Suppl 1): S41-S47
• 36 Haynes SC, Rudov L, Nauman E, Hendryx L, Angove RSM, Carton T. Engaging stakeholders to develop a patient-centered research agenda: lessons learned from the Research Action for Health Network (REACHnet). Med Care 2018; 56 (Suppl. 10) S27-S32
• 37 Jonas D, Mansfield AJ, Curtis P. , et al. Identifying Priorities for Patient-Centered Outcomes Research for Serious Mental Illness. Rockville, MD: Agency for Healthcare Research and Quality (US); 2011 . Report No. 11–EHC066-EF
• 38 Lear SA. Patients in research: one step in a long path. BMJ 2018; 363: k4386
• 39 Truman C, Raine P. Experience and meaning of user involvement: some explorations from a community mental health project. Health Soc Care Community 2002; 10 (03) 136-143
• 40 Tambuyzer E, Pieters G, Van Audenhove C. Patient involvement in mental health care: one size does not fit all. Health Expect 2014; 17 (01) 138-150
• 41 MacLean S, MacKie C, Hatcher S. Involving people with lived experience in research on suicide prevention. CMAJ 2018; 190 (Suppl): S13-S14
• 42 Hendrikoff L, Kambeitz-Ilankovic L, Pryss R. , et al. Prospective acceptance of distinct mobile mental health features in psychiatric patients and mental health professionals. J Psychiatr Res 2019; 109: 126-132
• 43 American Psychiatric Association. App Advisor Expert Panel. American Psychiatric Association; 2019 . Available at: https://www.psychiatry.org/psychiatrists/practice/mental-health-apps/app-advisor-expert-panel . Accessed July 16, 2019
• 44 Strudwick G, Kuziemsky C, Booth RG. , et al. Engaging patients and family members in the evaluation of a mental health patient portal: protocol for a mixed-methods study. BMJ Open 2018; 8 (08) e025508
• 45 Strudwick G, Leung K, McLean D. , et al. (2019) Patient and Family Engagement in Health Information Technology Initiatives: Findings of a Literature Review, Focus Groups and Symposium. Centre for Addiction and Mental Health; 2019. Available at https://www.camh.ca/-/media/files/pdfs—reports-and-books—research/camh-health-it-initiative-28mar2018-6002.pdf . Accessed April 9, 2019
• 46 Rutter D, Manley C, Weaver T, Crawford MJ, Fulop N. Patients or partners? Case studies of user involvement in the planning and delivery of adult mental health services in London. Soc Sci Med 2004; 58 (10) 1973-1984
• 47 Laitila M, Nummelin J, Kortteisto T, Pitkänen A. Service users' views regarding user involvement in mental health services: a qualitative study. Arch Psychiatr Nurs 2018; 32 (05) 695-701
• 48 Storm M, Edwards A. Models of user involvement in the mental health context: intentions and implementation challenges. Psychiatr Q 2013; 84 (03) 313-327
• 49 Dixon LB, Holoshitz Y, Nossel I. Treatment engagement of individuals experiencing mental illness: review and update. World Psychiatry 2016; 15 (01) 13-20
• 50 Brady F, Warnock-Parkes E, Barker C, Ehlers A. Early in-session predictors of response to trauma-focused cognitive therapy for posttraumatic stress disorder. Behav Res Ther 2015; 75: 40-47
• 51 Jung E, Wiesjahn M, Rief W, Lincoln TM. Perceived therapist genuineness predicts therapeutic alliance in cognitive behavioural therapy for psychosis. Br J Clin Psychol 2015; 54 (01) 34-48
• 52 Stanhope V, Tondora J, Davidson L, Choy-Brown M, Marcus SC. Person-centered care planning and service engagement: a study protocol for a randomized controlled trial. Trials 2015; 16: 180
• 53 Delbanco T, Walker J, Bell SK. , et al. Inviting patients to read their doctors' notes: a quasi-experimental study and a look ahead. Ann Intern Med 2012; 157 (07) 461-470
• 54 Walker J, Leveille S, Bell S. , et al. OpenNotes after 7 years: patient experiences with ongoing access to their clinicians' outpatient visit notes. J Med Internet Res 2019; 21 (05) e13876
• 55 Cromer R, Denneson LM, Pisciotta M, Williams H, Woods S, Dobscha SK. Trust in mental health clinicians among patients who access clinical notes online. Psychiatr Serv 2017; 68 (05) 520-523
• 56 Dobscha SK, Denneson LM, Jacobson LE, Williams HB, Cromer R, Woods S. VA mental health clinician experiences and attitudes toward OpenNotes. Gen Hosp Psychiatry 2016; 38: 89-93
• 57 Denneson LM, Cromer R, Williams HB, Pisciotta M, Dobscha SK. A qualitative analysis of how online access to mental health notes is changing clinician perceptions of power and the therapeutic relationship. J Med Internet Res 2017; 19 (06) e208
• 58 Dobscha SK, Denneson LM, Pisciotta MK, Bourne DS, Chen JI, Philipps-Moses D, Woods SS. Predictors of viewing progress notes among users of VA's electronic health portal who receive mental health care. JAMIA Open 2018; 1 (01) 122-127
• 59 Denneson LM, Chen JI, Pisciotta M, Tuepker A, Dobscha SK. Patients' positive and negative responses to reading mental health clinical notes online. Psychiatr Serv 2018; 69 (05) 593-596
• 60 Kahn MW, Bell SK, Walker J, Delbanco T. . Let's show patients their mental health records. JAMA 2014; 311 (13) 1291-1292
• 61 OpenNotes. Mental health notes health systems list (direct communication). May 14, 2019
• 62 Petersson L, Erlingsdóttir G. OpenNotes in Swedish psychiatric care (part 1): survey among psychiatric care professionals. JMIR Ment Health 2018; 5 (01) e11
• 63 Petersson L, Erlingsdóttir G. OpenNotes in Swedish psychiatric care (part 2): survey among psychiatric care professionals. JMIR Ment Health 2018; 5 (02) e10521
• 64 Ocloo J, Matthews R. From tokenism to empowerment: progressing patient and public involvement in healthcare improvement. BMJ Qual Saf 2016; 25 (08) 626-632
• 65 Breault LJ, Rittenbach K, Hartle K. , et al. People with lived experience (PWLE) of depression: describing and reflecting on an explicit patient engagement process within depression research priority setting in Alberta, Canada. Res Involv Engagem 2018; 4: 37
• 66 Teague S, Youssef GJ, Macdonald JA. , et al; SEED Lifecourse Sciences Theme. Retention strategies in longitudinal cohort studies: a systematic review and meta-analysis. BMC Med Res Methodol 2018; 18 (01) 151
• 67 Crocker JC, Ricci-Cabello I, Parker A. , et al. Impact of patient and public involvement on enrolment and retention in clinical trials: systematic review and meta-analysis. BMJ 2018; 363: k4738
• 68 Bethell J, Commisso E, Rostad HM. , et al. Patient engagement in research related to dementia: a scoping review. Dementia 2018; 17 (08) 944-975
• 69 Gillard S, Simons L, Turner K, Lucock M, Edwards C. Patient and public involvement in the coproduction of knowledge: reflection on the analysis of qualitative data in a mental health study. Qual Health Res 2012; 22 (08) 1126-1137
• 70 Shippee ND, Domecq Garces JP, Prutsky Lopez GJ. , et al. Patient and service user engagement in research: a systematic review and synthesized framework. Health Expect 2015; 18 (05) 1151-1166
• 71 OpenNotes. Implementing OpenNotes: Improving Patient Access to Notes on Patient Portals: An OpenNotes White Paper. OpenNotes; 2018 . Available at http://www.opennotes.org/wp-content/uploads/2018/12/Implementing_OpenNotes_Improving_Patient_Access_to_Notes_on_Patient_Portals.pdf . Accessed January 19, 2019
• 72 Pisciotta M, Denneson LM, Williams HB, Woods S, Tuepker A, Dobscha SK. Providing mental health care in the context of online mental health notes: advice from patients and mental health clinicians. J Ment Health 2019; 28 (01) 64-70