
==== Front
Med Care
Med Care
MLR
Medical Care
0025-7079
1537-1948
Lippincott Williams & Wilkins Hagerstown, MD

MDC-D-23-00372
10.1097/MLR.0000000000002025
00009
3
Original Articles
Implementing Patient-Centered Outcomes Research Institute Stakeholder Engagement Principles in Models of Palliative Care Delivery and Advance Care Planning Research
Walling Anne M. MD, PhD *†awalling@mednet.ucla.edu

Verma Manisha MD, MPH ‡manisha.verma@jefferson.edu

Grudzen Corita R. MD, MSHS §grudzenc@mskcc.org

Enguidanos Susan PhD, MPH ∥enguidan@usc.edu

Barrett Nadine J. PhD, MA, MS ¶NJBarret@wakehealth.edu

Johnson Kimberly S. MD, MHS #kimberly.s.johnson@duke.edu

Combe Angela K. MS **combe@ohsu.edu

Johnston Fabian M. MD, MHS ††fjohnst4@jhmi.edu

Greer Joseph A. PhD JGREER2@mgh.harvard.edu
‡‡
* Department of Medicine, Division of General Internal Medicine and Health Services Research, University of California, Los Angeles, CA
† Department of Medicine, VA Greater Los Angeles Healthcare System, Los Angeles, CA
‡ Department of Medicine, Einstein Healthcare Network, Philadelphia, PA
§ Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY
∥ Leonard Davis School of Gerontology, University of Southern California, Los Angeles, CA
¶ Department of Social Science and Public Health Policy, Division of Population Health Sciences, Wake Forest University School of Medicine, Winston Salem, Durham, NC
# Department of Medicine, Division of Geriatrics, Duke University School of Medicine, Durham, NC
** Department of Medical Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, OR
†† Division of Surgical Oncology, Johns Hopkins University, Baltimore, MD
‡‡ Department of Psychiatry, Center for Psychiatric Oncology & Behavioral Sciences, Massachusetts General Hospital, Boston, MA
Correspondence to: Anne M. Walling, MD, PhD, Division of General Internal Medicine and Health Services Research, University of California, Los Angeles, 1100 Glendon Avenue, Suite 850, Los Angeles, CA 90095. E-mail: awalling@mednet.ucla.edu.
10 2024
6 9 2024
62 10 693700
Copyright © 2024 The Author(s). Published by Wolters Kluwer Health, Inc.
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal. http://creativecommons.org/licenses/by-nc-nd/4.0/

Background:

Given the many challenges of conducting research that addresses the palliative and end-of-life care needs of patients with serious illnesses, stakeholder engagement starting from the moment of study conceptualization and design is critical to ensure successful participant recruitment, data collection, intervention delivery, data analysis, and dissemination.

Methods:

Guided by a conceptual model published by the Patient-Centered Outcomes Research Institute (PCORI) entitled, “Measuring What Matters for Advancing the Science and Practice of Engagement”14 and with the support of a PCORI Engagement Officer, representatives from 9 PCORI-funded study teams formed a working group to survey team members and review, outline, and describe key lessons learned and best practices for promoting stakeholder engagement in palliative care research.

Results:

Almost all study teams engaged with patients/caregivers, clinicians, researchers, and health care system experts as stakeholder partners. About half the teams also included payers and training institutions as part of their stakeholder advisors as well as a range of content experts. Study teams relied on a variety of support structures and resources, and they employed 10 distinct methods for maintaining engagement. All engagement methods were generally considered to be effective by teams who used the method, though there was some variability in team-rated engagement quality of each method. Nine barriers to stakeholder engagement were identified across the 9 studies as well as 9 strategies (or facilitators) to overcome these barriers. We share examples of how stakeholder engagement impacted studies in all phases, including the preparatory phase, study initiation phase, execution phase, and data analysis/dissemination phase.

Conclusions:

Teams utilized a variety of resources and support structures as well as capitalized on multiple engagement methods for fostering stakeholder engagement, resulting in a high level of collaboration and integration.

Key Words:

stakeholder engagement
palliative care
advance care planning
SDCT
OPEN-ACCESSTRUE
==== Body
pmcThere is a growing recognition of the importance of meaningfully engaging stakeholders, including patients and family members, in every phase of health care research, particularly for translational, comparative effectiveness, and implementation science.1–3 Underscoring this priority, all awards funded by the Patient-Centered Outcomes Research Institute (PCORI) have patients and other key stakeholders as part of the investigator study teams. Guided by the PCORI engagement principles, researchers provide a prospective engagement plan as part of their study proposals and have access to the support and guidance of a PCORI engagement officer throughout the study period.4

The PCORI engagement principles include reciprocal relationships, co-learning, and partnerships, as well as transparency, honesty, and trust.4 Scholarly literature is limited with respect to specific steps and best practices for implementing these principles in the context of real-world research. We therefore sought to describe the experience of 9 PCORI-funded palliative care and advance care planning (ACP) studies aiming to implement these principles. In 2018, PCORI created the Palliative Care Learning Network (PCLN), a research consortium and learning collaborative representing these 9 PCORI-funded multisite studies of palliative care and ACP interventions for diverse populations. The protocols for most of these trials have been published previously and include the following:Introducing Palliative Care within the Treatment of End Stage Liver Disease: A Randomized Controlled Trial (PIs: Navarro and Verma).5

Emergency-Department Initiated Palliative Care in Older Adults with Advanced Illness (PIs: Grudzen and Goldfeld).6,7

Expanding Access to Home-Based Palliative Care through Primary Care Medical Groups (PI: Enguidanos).8

Reducing Disparities in the Quality of Palliative Care for Older African Americans through Improved Advance Care Planning (PI: Johnson).9

A Cluster-Randomized Trial Comparing Team-Based versus Primary Care Clinician-Focused Advance Care Planning in Practice-Based Research Networks (PIs: Totten and Légaré).10

A Multi-Center Randomized Controlled Trial of Perioperative Palliative Care Surrounding Cancer Surgery for Patients and their Family Members (PI: Aslakson).11

Population-Based Comparison of Evidence-Based, Patient-Centered Advance Care Planning Interventions on Advance Directive Completion, Goal Concordant Care and Caregiver Outcomes for Patients with Advanced Illness (PIs: Wenger and Walling).12

Comparative Effectiveness of Early Integrated Telehealth Versus In-Person Palliative Care for Patients with Advanced Lung Cancer (PIs: Temel and Greer).13

Specialty Compared to Oncology Delivered Palliative Care for Patients with Acute Myeloid Leukemia (PIs: El-Jawahri and Temel).

Given the many challenges of conducting research that addresses the palliative and end-of-life care needs of patients with serious illnesses, stakeholder engagement starting from the moment of study conceptualization is critical to ensure successful participant recruitment, data collection, intervention delivery, data analysis, and dissemination. In this manuscript, we detail the key processes for engagement as well as common barriers and useful facilitators in conducting stakeholder-engaged palliative care research.

METHODS

As part of the PCLN initiatives, representatives from the 9 PCORI-funded study teams formed a working group to review, outline, and describe key lessons learned and best practices for promoting stakeholder engagement in palliative care research. Guided by a conceptual model that describes PCORI’s theory of action for engagement in research14 and with the support of a PCORI Engagement Officer, from February 2023 through August 2023, this working group convened multiple meetings to discuss the different approaches to stakeholder engagement across the 9 studies. In this conceptual model, ideally, engagement activities lead to a higher quality of engagement, thus resulting in better research and partner outcomes.14 The working group also disseminated an online survey for each group member to complete in collaboration with their respective study teams about their experiences with stakeholder engagement. The 25-item survey included questions regarding the stakeholder target constituents, processes and methods for promoting engagement, barriers, and facilitators of stakeholder engagement, and impacts of engagement across the different phases of study implementation. After collating this information, the working group members met again to review the survey results to clarify and resolve any incomplete or unclear responses. The members then completed a follow-up 5-item survey to confirm methods for fostering stakeholder engagement and to rank order their top 3 barriers and top 3 facilitators for engaging their stakeholder partners among engagement methods, as well as barriers and facilitators to engagement identified during the initial 25-item survey. Finally, a follow-up survey tailored to the engagement strategies utilized by each team gathered practical guidance on how to adopt these strategies, and teams rated the perceived engagement quality of each engagement activity or method on a 5-point Likert scale (1–5, with 5 being the highest quality engagement). Surveys are available in the appendix, Supplemental Digital Content 1, http://links.lww.com/MLR/C851. To ensure the accuracy and validity of the survey results, all working group members reviewed the manuscript before submission for publication.

RESULTS

Stakeholder Engagement Groups and Support Structures

As shown in Table 1, almost all study teams engaged with patients/caregivers, clinicians, researchers, and health care system experts as stakeholder partners. Approximately half the teams also included payers and training institutions as part of their stakeholder advisors. Given the unique aims of each trial, several teams identified other groups, such as professional society and disease-based organizations, faith leaders/chaplains, hospice center leadership, experts in serious illness communication and ACP, leaders in telehealth care delivery and research, and dissemination and implementation experts, as stakeholders for their respective studies.

TABLE 1 Types of Stakeholder Groups Across the 9 PCLN Studies

Study team	Patients/caregivers	Clinicians	Researchers	Payers	Hospitals/ health care systems	Policymakers	Training institutions	Other	
Navarro and Verma	X	X	X		X		X		
Grudzen and Goldfeld	X			X		X		Professional Society and Disease-based Organizations	
Enguidanos	X	X	X	X	X	X			
Johnson	X	X	X		X	X		Faith Leaders, Chaplain, Hospice Center Leadership	
Totten and Légaré	X	X	X		X	X	X	Serious Illness Communication Experts (Ariadne Labs)	
Aslakson	X	X	X		X	X	X	Professional Society and Disease-based Organizations	
Wenger and Walling	X	X	X	X	X	X	X		
Temel and Greer	X	X	X	X	X	X		Telehealth Clinicians and Researchers	
El-Jawahri and Temel	X	X	X	X	X	X		Dissemination/ Implementation Experts	

To develop comprehensive stakeholder engagement plans, study teams relied on a variety of support structures and resources. Specifically, patient and family advisory councils, local community organizations, support groups, religious communities, and practice-based research networks were instrumental in identifying patient and caregiver stakeholders. Clinicians, researchers, and health care system stakeholders often came from the participating centers, whereas payers and national advocacy and professional organizations (eg, palliative care organizations and disease-specific groups) with vested interests in the trial outcomes helped to identify representatives to serve as stakeholders. Finally, several study teams noted that PCORI Engagement Officers and resources, as well as NIH-funded resources, such as the Palliative Care Research Cooperative and Clinical & Translational Science Awards Program, were essential to supporting their stakeholder engagement efforts.

Some teams drew on various frameworks and population-based models of patient-centered care to inform their selection of stakeholder groups and delineate stakeholder roles. For example, one team developed a specific framework to detail the roles and responsibilities of a stakeholder advisory member (PI: Johnson). Specifically, this framework included the following: (1) overview of the study and goals, (2) explanation of the purpose of the Stakeholder Advisory Council (SAC), (3) clarification of who can be a stakeholder (eg, patients and caregivers of diverse racial and ethnic backgrounds, community organizations, faith-based communities/religious leaders, hospice and home care workers, health care sector representatives, etc.), (4) roles and responsibilities of the stakeholder advisory member, (5) key qualifications needed to serve as a stakeholder advisor, (6) meeting frequency of the SAC, and (7) compensation for serving on the SAC. In addition, to enhance the diversity of stakeholder perspectives, most study teams conducted purposive sampling of patients, caregivers, and other key stakeholders based on a range of factors, including sociodemographic characteristics (ie, age, gender, race, ethnicity, socioeconomic status, immigration status, and preferred language); geographic region (ie, urban, rural, and across the United States or a particular state); clinical characteristics (ie, medical diagnoses such as cancer or heart disease, treatment phase, type of surgery, experience with health care, and advance care planning); health care role (ie, patient, caregiver, clinician, administrator, and policymaker); research experience; and representativeness of the populations receiving care at the study sites.

Stakeholder Engagement Methods and Processes

Table 2 details the methods study teams used to foster stakeholder engagement. The primary means of collaboration between study teams and stakeholders was through regular meetings and/or working retreats. Most teams also endorsed the inclusion of site-level stakeholders to collaborate with study staff at each institution to ensure the successful implementation of their trials across multiple sites, as well as an overall Stakeholder Advisory Board to provide guidance on the stakeholder engagement processes. Teams highlighted the importance of bidirectional communication with summary reports after stakeholder meetings to outline collected feedback and corresponding study changes. Other methods to foster engagement included joint agenda planning for meetings, academic presentations and publications with stakeholders, newsletters, online surveys of the stakeholder experience, invited guest speakers to present to stakeholders, study websites highlighting patient and caregiver voices, and journal clubs.

TABLE 2 Engagement Method Implementation and Quality

Engagement method	Implementation strategies (Study N)	Engagement quality of method*	
Regular meetings with stakeholders (N=9)	• Met with patient/caregiver stakeholders monthly (1), quarterly (6), or biannually (2)
• Met with Study Advisory Group monthly (2), quarterly (3), biannually (3), or annually (1)
• Patient/caregiver stakeholder group contained <6 members (1), 6–10 members (6), 11–20 members (1), or 21–50 members (1)
• Study Advisory Group contained 6–10 members (3), 11–20 members (3), or 21–50 members (3)
• Topics discussed during meetings include study updates, milestones, challenges, ways to overcome challenges, and feedback relevant to study phase	4.9	
Inclusion of site-level stakeholders (N=8)	• Site research staff met with their local stakeholders quarterly (3), biannually (2), left to the discretion of each site (2), and once at the beginning of the study (1)
• Site PIs reached out to their patients to participate as stakeholders (3)
• Selected patients and caregivers who represent the study population at their local site (2)
• Site-level stakeholders were recruited through networks, local community partnership, patient advisory groups, faith organizations, etc. (1)
• Held facilitation training at each study site to know how to identify and engage diverse stakeholders (1)	5	
Bidirectional communication with summary reports after stakeholder meetings (N=8)	• Provided summary reports sent to stakeholders after each stakeholder meeting (8)
• Summary reports included study updates, challenges and facilitators, action items, recommendations and feedback provided by stakeholders and their impact, copies of slides presented at the meeting, and copies of any peer-reviewed publications	4.5	
Joint meeting agenda planning with stakeholders (N=5)	• Pre-meetings with stakeholders before larger meetings were conducted to discuss proposed agenda items (5)
• Topics included strategies to encourage interactions during larger meetings, preparing for stakeholder presentations, and identifying pressing topics for discussion	5	
Academic presentations and publications with stakeholders (N=6)	• Presentations were given at national conferences (3) and an international conference (1)
• Stakeholders were included as authors in published study protocols (3)5,9,10 and papers about engagement with stakeholders (4)15–20	4.8	
Newsletters sent to stakeholders (N=5)	• Newsletters were sent to stakeholders monthly (2), quarterly (2), and biannually (1)
• Newsletters contained project updates (accomplishments, upcoming dates, milestones, etc.) and stakeholder and study staff spotlights with pictures and bios	5	
Online surveys of stakeholder experience (N=6)	• Surveys were conducted after each stakeholder meeting (3), biannually (1), and annually (1), or at least once during the study (1)	4.3	
Invited guest speakers (N=3)	• Guest speakers were invited to each stakeholder meeting (1) and a mix of annually and biannually throughout the study (2)
• Guest speakers included national organizations, content experts, and often stakeholders themselves
• PIs, Co-Is, and study team members invited guest speakers via email
• Guest speakers were invited to increase knowledge of a particular issue, increase participation and interest among a wide array of stakeholders, and engage organizations that had not participated regularly in meetings	4.7	
Webpage featuring stakeholder voices (N=2)	• PI created and maintained the webpage
• Staff and patient/caregiver stakeholders contributed to the content
• Staff and volunteers supported webpage maintenance and design	5	
Journal clubs (N=2)	• Journal club started based on a suggestion from a patient stakeholder after reading a newspaper article on palliative care and continued quarterly
• Discussions centered on news and blogs that patients and caregivers found interesting and relevant in addition to journal articles	4	
* Engagement quality was measured using a 5-point Likert scale (1–5, with 5 representing the highest quality engagement).

Table 2 also details how the study teams implemented the engagement methods and rated the effectiveness of supporting engagement quality. All engagement methods were generally considered to be effective by teams who used the method, though there was some variability in team-rated engagement quality of each method. For example, regular meetings with stakeholders were an engagement strategy that was used by all 9 teams surveyed, and this strategy was rated highly in terms of effectiveness at supporting engagement quality by teams (4.9/5), while journal clubs were only used by 2 teams and rated lower (4/5) in terms of effectiveness at supporting engagement quality. In terms of logistics for regular meetings, the frequency of meetings between the study teams and their stakeholder partners varied, ranging from monthly to biannually for both patient/caregiver stakeholders and other study advisory group members, though the majority met quarterly for ~60 to 120 minutes. These meetings took place either in person or via telephone or video conference, most often consisting of PowerPoint presentations to provide study updates and guide discussion of key questions. Topics discussed during the meetings included study updates, milestones, challenges, ways to overcome challenges, and feedback relevant to study phase. The meeting frequency was higher during the launch of the trials, with some teams also hosting annual in-person gatherings at their institutions or other venues (eg, national conferences). After the onset of the COVID-19 pandemic, most stakeholder meetings took place virtually. All 9 study teams provided reimbursement or incentive payments for stakeholders, ranging from $50 to $100 per meeting, paid either by check or gift card. Managing reimbursement documentation for tax purposes per institutional policies was a challenge for some teams, and many stakeholders opted to forego reimbursement to support the projects. Additional details for how other engagement methods were implemented are included in Table 2.

Barriers to Stakeholder Engagement

We identified 9 barriers to stakeholder engagement across the 9 studies. The top 3 barriers, as ranked by teams (top 3 listed below per rank order), were common across 5 or more studies.Stakeholder turnover due to health (particularly patients), relocation, employment changes, and personal challenges (7 studies)

COVID-19 preventing in-person meetings (6 studies).

Maintaining the same level of engagement across geographic sites (5 studies).

Ensuring the content presented was relevant to all groups (challenge of meetings that included patients, organizational leaders, payers, etc. all together) (3 studies).

Need to educate IRB that stakeholders were study collaborators, not participants (3 studies).

Clarification of roles and responsibilities of stakeholders (3 studies).

Waning engagement/attendance as the study progressed (2 studies).

Changing priorities of organizational leaders and secular trends (2 studies).

Discerning which stakeholder recommendations were feasible to implement (2 studies)

Facilitators of Stakeholder Engagement

We identified 9 facilitators of stakeholder engagement that were used across the 9 studies to overcome barriers. The top 3 facilitators, as ranked by teams (top 3 listed below per rank order), were common across 6 or more studies.Dedicated resources such as staff/funding for engagement (6 studies).

Ensuring time for listening to and discussion of feedback, including premeeting preparation with patient/caregiver team members (6 studies).

Flexibility and openness to change approach/processes based on feedback (7 studies).

Personal interactions between stakeholders and the PI and study team (6 studies).

Infrastructure for communication (eg, Google groups, video teleconferencing for stakeholder meetings, weekly meetings, and newsletters) (6 studies).

Providing education as part of an advisory group participation, such as for palliative care policy and research (6 studies).

Developing camaraderie among participants (5 studies).

Regular measurement and feedback regarding engagement expectations/processes (5 studies).

Prior longitudinal relationships with patient stakeholders (4 studies).

Impact of Stakeholder Engagement

All but one study agreed or strongly agreed with the statement that stakeholder engagement with the research team was optimal. These teams highlighted that a high level of engagement and feedback were key aspects of study success resulting in strong ties with stakeholder groups. One team that neither agreed nor disagreed with this statement had a successful engagement with insurance partners and caregiver organizations but struggled with health care systems and national policy organizations, ultimately leading to challenges with recruitment.21 Teams provided clear examples of engagement and the impact of stakeholder feedback on study procedures (Table 3). These examples show the high level of integration that stakeholders had in all aspects of the study, from the preparatory to the data analysis and dissemination phases.

TABLE 3 Examples of Engagement and Impact of Stakeholders on Study Phases Across 9 PCLN Studies

Preparatory phase	• Identified strategic way to inform and choose comparators and interventionists.
• Helped ensure the research question is relevant (needed) to the potential study population and community.
• Assisted with identifying other potential stakeholders needed to help contribute to the success of the study.
• Helped shape the research design and choice of outcomes, including measures to assess those outcomes (both primary and secondary). Most importantly, optimizing the choice of important outcomes resonates across studies.
• Patients and caregivers shared their own stories to inform the training materials and how they would have valued palliative care at those critical times.
• Involved clinical centers during the preparatory phase to inform the structure and processes for intervention delivery.	
Study initiation phase	• Developed strategies and language to introduce the concept of palliative care and ACP.
• Conducted role plays with research staff for recruitment, informed consent, and intervention delivery procedures.
• Assisted/guided the development of patient-facing study materials.
• Assisted/guided modification of survey instruments, including qualitative interview guides, to ensure patient friendliness.	
Execution phase	• Provided ongoing feedback on recruitment challenges (e.g. patient-facing materials, acceptability, and health literacy).
• Provided feedback on improved data collection (patient-facing language and materials; gift card type; surveys: design, mixed mode, and pilot testing; reminder postcards; and COVID challenges).
• Provided feedback on key study outcomes.
• Provided feedback to expand data elements in chart review.	
Data analysis and dissemination phase	• Participated in writing/journal publications as well as oral and poster presentations and plenaries.
• Provided guidance regarding dissemination of findings to a broad audience, including to other potential stakeholders and use of website/digital media.
• Provided insight into study results from the patient’s point of view.
• Obtained feedback that allowed for the development of more robust list of subgroup analyses (e.g. added insurance status and type to the chart review data that we collected and conducted a companion implementation survey regarding the delivery of palliative care via telehealth)	

Preparatory Phase

In developing their study proposals, teams noted that involving stakeholders during the preparatory phase helped shape the research design, study aims and measures, as well as training materials and structure and processes for intervention delivery. One example from the PAL LIVER study5 was identifying the study comparative groups and development of the research hypothesis. For this study, the research team formed a research advisory board (RAB) with patients diagnosed with end-stage liver disease and their caregivers, a few of whom had been a part of a prior feasibility trial of palliative care for similar patients undergoing evaluation for liver transplants. The RAB members voted for a study design to compare the effectiveness of palliative care services delivered by their own hepatologists versus palliative care subspecialists. The RAB members also proposed a superiority comparison, given the fact that hepatologists have a pre-existing relationship with their patients and a better understanding of their disease trajectory, which may benefit the model of hepatologist-delivered palliative care versus having a new provider. These discussions supported the refinement of comparative models of palliative care and proposed data analysis plan at a very early stage.22

Study Initiation Phase

Study teams described how stakeholders assisted in the training of research staff, the development of strategies for how to introduce the concept of palliative care or ACP, as well as the development of patient-facing study materials. One example from the UC Health Care Planning Study12 shows how the community advisory board helped meet recruitment targets by recognizing early on that the 3-page HIPAA form required by the institution would hinder enrollment, which was later confirmed with quantitative data.23 During the initiation phase, several studies involved stakeholders to assist with developing and piloting survey instruments (including qualitative interviews), reviewing and refining recruitment scripts, and role-playing with research coordinators to explain the role of palliative care during the enrollment process, given the historical misconception of palliative care services referring only to end-of-life or hospice care.

Execution Phase

Study teams also described how stakeholder engagement helped overcome challenges with participant recruitment, intervention delivery, and data collection throughout the execution phase. For example, stakeholders of the early integrated telehealth versus in-person palliative care trial13 recommended conducting a survey of the study clinicians across the 22 sites involved in the trial to learn about the barriers and facilitators in providing palliative care via video visits and to identify any interventions that clinicians and support staff used to overcome the barriers in intervention delivery.

Data Analysis and Dissemination Phase

Stakeholders provided key insights on study results, including community advisory board members from the patient’s perspective. These insights often led to more robust analyses. For example, the policymaker and health care system stakeholders in one of the teams suggested considering different types of subgroup analyses that may be meaningful based on participants’ sociodemographic characteristics and social determinants of health, such as insurance status and type. Finally, study teams further described how stakeholders provided guidance on the dissemination of results to a broad audience (broader than would have been reached with the study team alone) and participated in the writing of journal articles as well as poster and oral (even plenary) presentations.

DISCUSSION

Stakeholder engagement can enhance research efficiency and its true impact on study processes and outcomes. All 9 studies included comprehensive stakeholder engagement plans that incorporated the perspectives and voices of patients and families, clinicians, and researchers. Policymakers and representatives from health care systems, payers, and training institutions also served as key stakeholders, in addition to a range of content experts depending on the focus of the study. Teams utilized a variety of resources and support structures as well as capitalized on multiple engagement methods for fostering stakeholder engagement, resulting in a high level of collaboration and integration. All engagement methods were perceived to be effective in supporting engagement quality (lowest score 4/5 on 5-point Likert scale, with a score of 5 reflecting the highest quality engagement), though there was some variability in the frequency of use across teams as well as the perceived effectiveness in supporting engagement quality. While our teams perceived that engagement methods were effective in achieving engagement quality, future research should formally measure if increased engagement quality is associated with differences in research and partner outcomes.

We conclude that broad and well-integrated stakeholder engagement is essential for the success of real-world research focused on palliative care and ACP. Despite efforts to improve messaging and communication about these topics,24–27 confusion persists regarding the semantics and philosophical focus of palliative care and ACP, including among physicians, even among those in our own field.28–30 Stakeholder engagement is therefore essential, especially in palliative care and ACP research, to support overcoming misperceptions and ensure that interventions are implemented and evaluated successfully.

Interestingly, while barriers varied across our diverse studies, with the majority of barriers identified by less than half of the studies, facilitators were common across studies, with 8/9 identified facilitators utilized by the majority of the participating studies. This consistency in reported facilitators likely represents the benefits of a learning collaborative. Although the studies had different designs, covered different geographic areas, and encountered different challenges, many of the approaches to overcome barriers were shared and utilized across studies in the PCLN.

Successful stakeholder engagement requires a well-conceived prospective approach, and the methods for stakeholder engagement and facilitators of successful engagement described in this paper can serve as a reference of best practices for other palliative care and ACP studies. We hope the experiences and shared knowledge that we collaboratively collated will support teams considering engaging with stakeholders for their research, enhance understanding of the benefits this collaboration can bring, provide a simple guide on steps and processes for successful engagement, and assist with identifying and overcoming challenges that might occur throughout the different study phases.

Supplementary Material

ACKNOWLEDGMENTS

The authors would like to acknowledge staff at all study sites who were integral contributors to the studies, as well as the core research team members and all study patient, caregiver, and other advisors, particularly The UC Health Care Planning study team members: Neil Wenger, Juan Carlos Antonio Lopez, Douglas Bell, Kirsten Buen, Anna DePaolis-Dickey, Ron Hays, Victor Gonzalez, Javier Sanz, Katherine Santos, Chi-hong Tseng, Rebecca Sudore, Axel Hererra, Leah Karliner, Jonathan Lee, Brookelle Li, Kanan Patel, Christine Ritchie, Andrew Robinson, Gabriela Vanegas, Aiesha Volow, Lisa Gibbs, Maryam Rahimi, Jamie Anand, Aaron Chau, Valerie George, Rick Marshall, Eileen Sabino-Laughlin, Megan Whalen, Keeta Scholl, and Judy Thomas; UC Patient/Caregiver Advisory Board members: Imelda Aguilera, Irene Conway, Nabi Khorrami, Naz Khorrami, Jason Kogan, Patricia Levenberg, Arnold Porath, Wanda Reynolds, Tom Reynolds, and Esme Seto; and Study Advisory Group members: Romilla Batra, Teri Boughton, Kimberly Bower, David Debono, Torrie Fields, Jennifer Fuller, Marvin Gordon, Shawn Hamilton, Jean Kutner, Kate Meyers, James Mittelberger, Dan Osterweil, Steve Pantilat, David Pryor, Elyse Salend, Amy Vandenbroucke, and Charlene Zettel. Executive Committee members: Victor Navarro, Andrzej Kosinski, Tamar Taddei, Michael Volk, Kavitha Ramchandran, and Marie Bakitas; Research Advisory Board members: Kenneth and Lisa Green, Patricia Ramos, Richard Ramos, Phyllis Fleming, Dan Howard, and Bitsy Tatnall; and all clinical centers and Duke Clinical Research Institute’s team members participating in the PAL LIVER study. Core research team members: Keith S. Goldfeld, Allison M. Cuthel, Kaitlyn Van Allen, and Mara Flannery; and Study Advisory Committee Members Romilla Batra, Juanita Booker-Vaughns, Garrett K. Chan, Patrick Dunn, Robert Galvin, Ernest Hopkins, Eric David Isaacs, Constance L. Kizzie-Gillet, Margaret Maguire, Neha Reddy Pidatala, Dawn Rosini, William Vaughan, Sally Welsh, Pluscedia G. Williams; Angela Young-Brinn, and Martha Navarro for the EMPallA study. Anna Rahman, Torrie Fields, Richard Brumley, Michael Rabow, and Wendy Mack for the Expanding Access to Home-Based Palliative Care study. Kenisha Bethea, Marisette Hasan, members of the 5 Community Advisory Councils across all sites, members of the EQUAL ACP Community Advisory Board, the Clinical Research Coordinators who facilitated the Community Advisory Council meetings, and the AME Zion HEAL Partnership. The Meta-LARC ACP study patient and family advisors: Jacqueline Alikhaani, B. Angeloe Burch Sr., Deb Constien, Kenneth Dailey, Gail Drey, Kathy Kastner, Judy Katz, Susan Lowe, James Pantelas, Olga Petrova, and Keith Provin; the Meta-LARC ACP Trial Engagement Consultants from the Institute for Patient- and Family-Centered Care, including Deborah Dokken and Mary Minniti; and the Meta-LARC ACP trial team.

This work was (partially) supported through the following Patient-Centered Outcomes Research Institute (PCORI) Awards: IHS-1609-36518-IC, IHS-1602-34070, PLC-1609-36306, PLC-1609-36381, PLC-1609-36714, PLC-1609-35995, PLC-1609-36277, PLC-1609-36291, and PLACER-2020C3-21247. All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the view of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or the Methodology Committee. This work was also supported by the Memorial Sloan Kettering Cancer Center Support Grant/Core Grant (P30 CA008748). The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.

S.E. received grant funding for palliative care-related research/work from the California Healthcare Foundation, Blue Shield of California, and Stupski Foundation. A.K.C. received honoraria payments for participating on a national Advisory Committee that began in June 2023. J.A.G. has received research funding from Blue Note Therapeutics, remuneration from BeiGene for serving as an advisory board member, and royalties from Oxford University Press. The remaining authors declare no conflict of interest.

Supplemental Digital Content is available for this article. Direct URL citations are provided in the HTML and PDF versions of this article on the journal's website, www.lww-medicalcare.com.
==== Refs
REFERENCES

1 Yarborough M Edwards K Espinoza P . Relationships hold the key to trustworthy and productive translational science: recommendations for expanding community engagement in biomedical research. Clin Transl Sci. 2013;6 :310–313.23919367
2 Goodman MS Sanders Thompson VL . The science of stakeholder engagement in research: classification, implementation, and evaluation. Transl Behav Med. 2017;7 :486–491.28397159
3 Bell-Brown A Watabayashi K Kreizenbeck K . An evaluation of stakeholder engagement in comparative effectiveness research: lessons learned from SWOG S1415CD. J Comp Eff Res. 2022;11 :1313–1321.36378570
4 Patient-Centered Outcomes Research Institute. PCORI engagement rubric for applicants. 2016. Accessed February 22, 2024. https://www.pcori.org/sites/default/files/Engagement-Rubric.pdf
5 Verma M Kosinski AS Volk ML . Introducing palliative care within the treatment of end-stage liver disease: the study protocol of a cluster randomized controlled trial. J Palliat Med. 2019;22 (suppl 1 ):34–43.
6 Grudzen CR Shim DJ Schmucker AM . Emergency Medicine Palliative Care Access (EMPallA): protocol for a multicentre randomised controlled trial comparing the effectiveness of specialty outpatient versus nurse-led telephonic palliative care of older adults with advanced illness. BMJ Open. 2019;9 :e025692.
7 Grudzen CR Schmucker AM Shim DJ . Development of an outpatient palliative care protocol to monitor fidelity in the emergency medicine palliative care access trial. J Palliat Med. 2019;22 (suppl 1 ):S-66–S-71.
8 Enguidanos S Rahman A Fields T . Expanding access to home-based palliative care: a randomized controlled trial protocol. J Palliat Med. 2019;22 (suppl 1 ):S-58–S-65.
9 Ejem DB Barrett N Rhodes RL . Reducing disparities in the quality of palliative care for older African Americans through improved advance care planning: study design and protocol. J Palliat Med. 2019;22 (suppl 1 ):S-90–S-100.
10 Totten AM Fagnan LJ Dorr D . Protocol for a cluster randomized trial comparing team-based to clinician-focused implementation of advance care planning in primary care. J Palliat Med. 2019;22 (suppl 1 ):82–89.31486729
11 Aslakson RA Chandrashekaran SV Rickerson E . A multicenter, randomized controlled trial of perioperative palliative care surrounding cancer surgery for patients and their family members (PERIOP-PC). J Palliat Med. 2019;22 (suppl 1 ):44–57.31486730
12 Walling AM Sudore RL Bell D . Population-based pragmatic trial of advance care planning in primary care in the University of California Health System. J Palliat Med. 2019;22 (suppl 1 ):72–81.31486723
13 Chua IS Zachariah F Dale W . Early integrated telehealth versus in-person palliative care for patients with advanced lung cancer: a study protocol. J Palliat Med. 2019;22 (S1 ):S-7–S-19.
14 Patient-Centered Outcomes Research Institute. Measuring what matters for advancing the science and practice of engagement. April 24, 2023. Accessed February 23, 2024. https://www.pcori.org/resources/measuring-what-matters-advancing-science-and-practice-engagement
15 Rahman A Fields T Shirsat N . with Patient and Caregiver Advisors to the USC PCORI Study on Palliative Care, “Expanding Access to Home-Based Palliative through Primary Care Medical Groups”. Patient and caregiver perspectives on how to introduce community-based palliative care. J Palliat Med. 2018;21 :1544–1545.30383509
16 Combe AK Dokken DL Minniti MM . Meaningful engagement of patients and families in a complex trial of advance care planning in primary care. Patient Exp J. 2023;10 :57–73.
17 Booker-Vaughns J Rosini D Batra R . What’s in this for you? What’s in this for me?: a win-win perspective of involving Study Advisory Committee Members in Palliative Care Research. J Patient Exp. 2024;11 :23743735231224562.38188534
18 Zhao N Cuthel AM Storms O . Advancing patient-centered research practices in a pragmatic patient-level randomized clinical trial: a thematic analysis of stakeholder engagement in Emergency Medicine Palliative Care Access (EMPallA). Res Involv Engagem. 2024;10 :10.38263088
19 De Forcrand C Flannery M Cho J . Pragmatic considerations in incorporating stakeholder engagement into a palliative care transitions study. Med Care. 2021;59 (suppl 4 ):S370–S378.34228019
20 Sudore RL Walling AM Gibbs L . Implementation challenges for a multisite advance care planning pragmatic trial: lessons learned. J Pain Symptom Manage. 2023;66 :e265–e273.37098388
21 Enguidanos S Rahman A . Early termination of a palliative care trial: perspectives of multiple stakeholders on barriers to palliative care and research. J Palliat Med. 2022;25 :54–59.34191594
22 Verma M Tapper EB Singal AG . Nonhospice palliative care within the treatment of end-stage liver disease. Hepatol Baltim Md. 2020;71 :2149–2159.
23 Walling AM Sudore R Gibbs L . Randomized trial of the effect of a HIPAA authorization form on survey response for a research cohort in a cluster-randomized advance care planning trial. Abstracts from the 2021 Annual Meeting of the Society of General Internal Medicine. J Gen Intern Med. 2021;36 (suppl 1 ):1–469.
24 Back AL Wax JS Rossi RD . What patients and caregivers experience when they receive palliative care: a study eliciting metaphors that could shape public messaging. J Palliat Med. 2023;26 :751–756.37126403
25 Back AL Warner MR Beard KM . Use of messaging principles to design a Facebook ad promoting public engagement in serious illness care for National Healthcare Decisions Day. J Palliat Med. 2021;24 :1762–1765.34668792
26 Massachusetts Coalition for Serious Illness Care. Advancing the language of advance care planning: a messaging research project. Presented at November 19. Accessed February 23, 2024. https://www.maseriouscare.org/sites/g/files/csphws2336/files/2021-11/messaging-research-overview-updated-november-2019.pdf
27 McInturff B, Harrington L. Presentation of 2011 Research on Palliative Care. Center to Advance Palliative Care. Published May 20, 2011. Accessed June 12, 2024. https://www.capc.org/documents/download/868/
28 Grant MS Back AL Dettmar NS . Public perceptions of advance care planning, palliative care, and hospice: a scoping review. J Palliat Med. 2021;24 :46–52.32614634
29 Arney J Gray C Walling AM . Two mental models of integrated care for advanced liver disease: qualitative study of multidisciplinary health professionals. BMJ Open. 2022;12 :e062836.
30 Sudore RL Hickman SE Walling AM . Controversies about advance care planning. JAMA. 2022;327 :685.
