
==== Front
Med Care
Med Care
MLR
Medical Care
0025-7079
1537-1948
Lippincott Williams & Wilkins Hagerstown, MD

MDC-D-23-00371
10.1097/MLR.0000000000002042
00007
3
Original Articles
Conducting Comparative Effectiveness, Multisite Palliative Care and Advance Care Planning Trials
Lessons Learned and Future Directions From PCORI-Funded Studies
El-Jawahri Areej MD *ael-jawahri@mgb.org

Waterman Brittany MD †brittany.waterman@osumc.edu

Enguidanos Susan PhD, MPH ‡enguidan@usc.edu

Gibbs Lisa MD §lgibbs@hs.uci.edu

Navarro Victor MD ∥Victor.Navarro@jefferson.edu

Olsen Maren PhD ¶maren.olsen@duke.edu

Temel Jennifer MD *jtemel@mgb.org

Totten Annette PhD #totten@ohsu.edu

Grudzen Corita R. MD, MSHS, FACEP grudzenc@mskcc.org
**
* Division of Hematology-Oncology, Massachusetts General Hospital, Boston, MA
† Department of Internal Medicine, The Ohio State University Wexner Medical Center, Columbus, OH
‡ Leonard Davis School of Gerontology, University of Southern California, Los Angeles, CA
§ Department of Medicine, University of California, Irvine, Orange, CA
∥ Department of Medicine, Thomas Jefferson University, Philadelphia, PA
¶ Department of Biostatistics and Bioinformatics, Duke University, Durham, NC
# Informatics and Clinical Epidemiology, Oregon Health and Science University, Portland, OR
** Division of Supportive and Acute Care Services, Department of Medicine, Memorial Sloan Kettering Cancer Center, New York, NY
Correspondence to: Areej El-Jawahri, MD, Division of Hematology-Oncology, Massachusetts General Hospital 55 Fruit Street, Boston, MA 02114. E-mail: ael-jawahri@mgb.org.
10 2024
6 9 2024
62 10 671679
Copyright © 2024 The Author(s). Published by Wolters Kluwer Health, Inc.
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal.

The Patient-Centered Outcomes Research Institute (PCORI) funded multiple large-scale comparative effectiveness clinical trials evaluating palliative care (PC) and advance care planning (ACP) healthcare delivery models. This article provides an overview of the most common barriers our investigative teams encountered while implementing these trials and the strategies we utilized to overcome these challenges, with particular attention to identifying research partners for multisite trials; addressing contracting and regulatory issues; creating a team governance structure; training and engaging study staff across sites; recruiting, consenting, and enrolling study participants; collecting PC and ACP data and study outcomes; and managing multisite collaborations. The goal of this article is to provide guidance on how to best plan for and conduct rigorous trials evaluating PC and ACP healthcare delivery interventions moving forward.

Key Words:

Palliative Care
Pragmatic Trials
Advance Care Planning
Palliative Care Research
OPEN-ACCESSTRUE
==== Body
pmcOver the past decade, there has been increasing awareness of the importance of conducting rigorous clinical trials to build the evidence base and define the role of palliative care (PC) and advance care planning (ACP) healthcare delivery models for patients with serious illness.1–5 The goal of these trials is to inform clinical practice by evaluating the efficacy and effectiveness of healthcare interventions, as well as how to best incorporate PC and ACP interventions into the routine care of patients with serious illnesses. Core components of PC and ACP trials include integrating specialty PC clinicians in the care of patients with serious illnesses, addressing the palliative care needs of particular patient populations, and promoting ACP and high-quality patient-clinician-communication, to ensure the delivery of high-quality, goal-concordant at the end-of-life (EOL) for patients with serious illness. Yet, several meta-analyses of PC and ACP trials have highlighted important methodological limitations of prior research in the field.4–7 Multisite clinical trials are essential for providing high-quality evidence to set the standard of care for clinical practice but are often difficult to implement8,9 due to the challenges of implementing study procedures across multiple sites, ensuring adequate engagement from various stakeholders, and overcoming barriers to recruitment, intervention delivery, and data collection.10 Importantly, there are specific challenges inherent to PC and ACP research that increase the complexity of conducting multisite trials that can hamper the rigorous research needed to advance the science in this field.

The Patient-Centered Outcomes Research Institute (PCORI) is an independent, nonprofit organization that seeks to empower patients, caregivers, and other stakeholders with actionable information about their health and healthcare choices by funding comparative effectiveness research to help patients, families, clinicians, health system leaders, and policymakers make informed decisions. Over the last decade, PCORI has funded multiple large-scale comparative effectiveness trials evaluating PC and ACP healthcare delivery models for patients with serious illness and their caregivers, establishing a learning network among investigators to foster co-learning across the studies, which has played a critical role in the development of this article. The goal of this article is to describe the complexities of conducting large-scale multisite comparative effectiveness trials evaluating PC and ACP healthcare delivery models with an emphasis on the lessons learned during the process of executing these studies to inform future PC and ACP research efforts.

WHAT SHOULD INVESTIGATORS KNOW BEFORE PROPOSING A MULTISITE PC OR ACP CLINICAL TRIAL?

Investigators designing multisite PC or ACP trials should be prepared for challenges in identifying research partners, completing the contracting and regulatory processes, developing a strategy for training study staff, engaging study sites, recruiting and consenting participants, delivering interventions, and collecting and managing data.10 Importantly, strategies used to ensure the rigorous conduct of single-site or pilot PC or ACP trials may not translate fully to the challenges of multisite trial implementation. Thus, before proposing a multisite PC or ACP trial, investigators should consider conducting feasibility work at the participating sites to learn more about challenges and potential solutions that can be executed for large-scale implementation. Table 1 highlights the key strategies for success by focusing on important aspects of conducting multisite PC and ACP trials, which we discuss in detail below.

TABLE 1 Key Strategies for Success in All Aspects of Multisite Palliative Care (PC) and Advance Care Planning (ACP) Trials

Aspects of PC/ACP multisite trial	Strategies for success	
Identifying research partners for multisite PC/ACP trials	Conduct a rigorous potential study site review process, with attention to research infrastructure and prior experience in PC/ACP research.Obtain estimates of eligible patients to ensure adequate volume.
Obtain data on sites’ sociodemographic diversity to ensure generalizability.
Identify and engage clinical champions who can serve as advocates.
Engage hospital or health system leaders as champions for large-scale implementation and dissemination trials.
Leverage existing infrastructure within cooperative networks (eg, Practice-Based Research Networks) or other research consortia.
Identify backup study sites in case additional sites are needed.	
Contracting and regulatory issues	Plan a single Institutional Review Board (IRB) process to minimize regulatory burden
Partner with a single IRB experienced in PC/ACP.
Initiate the contracting process with study sites as early as possible and anticipate delays.
Consider start-up and milestone-based payment model for site reimbursement.
Address regulatory issues and safety concerns pertaining to consenting seriously ill patient population in PC/ACP trials in the study protocol.	
Team governance structure	Create a team governance structure with clear delineation of responsibilities and oversight.
Consider dividing responsibilities between a Clinical Coordinating Center and a Data Coordinating Center.
Hire a project manager with experience managing large-scale studies and exceptional communication skills.
Set up a clear meeting structure for the study team and its various subcommittees, including the roles of patient/family advisors.
Anticipate and prepare for study staff turnover.
Delineate expectations and benefits for co-investigators early including access to data, ability to develop ancillary studies, and collaborative verses independence publication.
Establish group authorship guidelines.	
Site and study staff training and engagement	Train site investigators, study interventionists, and research staff in standard operation procedures for the study.
Provide comprehensive training manuals and templates for study tracking logs.
Video record all training sessions for onboarding new study staff in the future.
Leverage patient stakeholders and specialty palliative care clinicians when training research staff to present PC/ACP studies.
Facilitate weekly or bi-weekly meetings with study staff at participating sites to address challenges and provide ongoing support.
Provide incentives such as payment or continued education credit to enhance engagement of interventionists when possible.
Provide ongoing support for research staff focused on addressing the challenges of working with seriously ill population in PC/ACP trials.	
Recruitment, consenting, and enrolling study participants	Utilize systematic recruitment to ensure generalizability and reduce bias.
Rely on the study staff to identify and consent eligible participants rather than a referral approach, which may result in bias in PC/ACP trials.
Leverage the electronic health record and technology when possible to streamline and reduce screening burden on study sites.
Test electronic health record screening algorithms across sites.
Integrate screening into the study site workflows; permitting as much flexibility as allowed by the study protocol, for implementation and dissemination trials.
Ensure equitable recruitment of participants who are underrepresented in clinical research.
Provide rigorous study staff training on how to present PC/ACP studies.
Leverage patient and caregiver stakeholders when crafting recruitment materials, conducting mock consents, and providing feedback on how to present PC/ACP to eligible patients and caregivers.
Provide study staff with language about how to talk about PC and ACP and address commonly asked questions by patients and families.
Monitor screening, recruitment, and consenting numbers by reviewing the study CONSORT on at least a monthly basis and troubleshoot specific challenges with each site.
Leverage study staff meetings to share learning regarding screening and recruitment process; allow sites to share experiences with each other.	
Data collection and management	Build a study-specific database and tracking log to be used across all study sites.
Diversify methods of data collection to minimize missing data in PC/ACP trials where attrition due to death is anticipated to be high.
Track methods of data collection (in-person, email, phone) to ensure rigor and capacity to conduct sensitivity analyses.
Prioritize administering important primary and secondary participant-reported outcomes in case participants experience fatigue and survey burden.
Align data collection with clinical care in implementation and dissemination trials.
Conduct rigorous training of study staff on how to obtain necessary data from the electronic health record.
Review CONSORTs regularly with careful attention to missing data.

Ensure appropriate oversight to identify modifiable reasons for missing data that can be addressed in PC/ACP trials.
Train study staff for challenging situations specific to obtaining data in PC/ACP trials (eg, contacting a caregiver of a deceased participant).
Anticipate more missing data in multisite PC/ACP trials than what is seen in pilot studies.	
Managing multisite trials	Facilitate regular meetings with the site investigators and staff to monitor enrollment, intervention delivery, and data collection.
Ask sites about and monitor potential disruptions that could impact the trial (eg, Organizational mergers, EHR changes, key clinician leave of absence, etc.).
Track site performance and provide transparent feedback relative to other sites.
Communicate through monthly newsletters to share best practices, disseminate lessons learned, and encourage collaboration.
Engage local patient and caregiver stakeholders to implement study procedures at respective sites effectively.
Share site performance metrics regularly to enhance performance across all sites.
Build a community around the study with shared mission and goals by celebrating study accomplishments, boosting morale, and provide site-to-site support.
Engage sites in collaborations and contributions that are meaningful to them.
Leverage the Scientific Advisory Committee and stakeholders to help overcome barriers	

Identifying Research Partners for Multisite PC and ACP Trials

The main limitation of prior PC and ACP trials is the lack of generalizability of study findings across diverse patient populations. Ensuring adequate geographic, racial, and ethnic diversity is critical when identifying research partners for multisite PC and ACP trials. Several of our PCORI-funded PC and ACP trials leveraged cooperative groups and organizations such as the Palliative Care Research Cooperative, Primary Care Practice-Based Research Networks (PBRNs), and professional academic societies in identifying study sites with substantial diversity to ensure the generalizability of study findings.11,12 Leveraging cooperative groups and healthcare organizations may help identify sites with prior experience conducting PC and ACP trials that have the research infrastructure to support the study. The investigative team should meet with sites interested in participating in the trial to ensure they have the experience and infrastructure to conduct PC and ACP trials, prior experience with this type of research, and adequate numbers of potentially eligible patients, including those with diverse backgrounds. Given the lack of familiarity and common misconceptions about PC and ACP research, it is also essential to identify clinical champions at participating sites to advocate for this research and to assess institutional readiness and commitment. For large-scale pragmatic comparative effectiveness trials integrating PC and ACP healthcare delivery model interventions, investigators should engage administrators, leaders, and additional stakeholders early [eg, information technology, electronic health record (EHR) integration experts, and technical analysts] to ensure commitment, as well as the capacity to implement such interventions in the proposed timeframe. Institutional readiness to implement PC and ACP interventions can often drive investigators decision-making regarding the optimal study design for pragmatic clinical trials. For example, a stepped wedge cluster randomized design can be employed when healthcare systems have decided to implement an intervention into routine care, yet still provides investigators an opportunity to study the effectiveness of such large-scale intervention efforts.13,14 The main advantage of the stepped wedge design is that the intervention would be rolled out to all participants across the healthcare systems without the need to randomize patients or healthcare systems to an unexpected control condition.13,14

Contracting and Regulatory Process for PC and ACP Clinical Trials

We faced regulatory challenges when implementing PCORI-funded PC and ACP trials, included difficulties dealing with multiple institutional review boards, as well as heterogeneity in the regulatory bodies’ experience with overseeing these types of trials. Many of our studies utilized a single Institutional Review Board (IRB) to reduce the regulatory burden on the participating site.15,16 Yet, the single IRB process can be challenging based on sites’ experience with reliance agreements, as well as the extent of local restrictions applied to the study conduct even when reliance to a single IRB is granted.17 Partnering with an IRB that has experience in overseeing PC and ACP trials can be especially useful to overcome regulatory challenges regarding understanding the relatively low risk of these interventions on study participants, applying various methods to identify seriously ill patients who meet eligibility criteria, and considering a documented waiver of written informed consent in minimal risk trials. Importantly, we recommend starting the regulatory process as early as possible since obtaining IRB approval across sites may take 4–6x months.

Contracting with study sites also requires significant time and planning. With the current contracting staff shortage across the country,18,19 it is important to start the contracting process as early as possible for multisite trials, understanding that this process may take up to 6 months. Funding organizations should also recognize the need to develop realistic timelines for study start-up activities that takes into account time needed for contracting. When developing contracts, investigators should consider the advantages and disadvantages of various payment models. Several of our studies used a fixed payment model with sites reimbursed for their research team’s effort over the study period.16,20 This model does not allow for modulation of payment based on study site performance and may create challenges for the study team if certain sites are underperforming. Some of our studies leveraged a milestone-based payment model where sites were reimbursed based on their performance milestones.15 Although this model can be advantageous to allocate the study budget most effectively based on the trial performance, it can be challenging for sites to hire adequate staffing to initiate the clinical trial without start-up funding. Many of our studies used a hybrid reimbursement model that included start-up funding followed by milestone-based payments to provide optimal flexibility for the research team while also prioritizing study performance metrics.11,21

Team Governance Structure for Multisite PC and ACP Clinical Trials

Given the complexity of multisite PC and ACP trials, creating a team governance structure with delineation of various responsibilities and oversight will facilitate implementation and decision-making. Figure 1 provides an example of a study team governance structure from one of our PCORI-funded clinical trials. In this example, the study included a Clinical Coordinating Center as well as a Data Coordinating Center that are distinct in their roles and responsibilities. Establishing a Data Coordinator Center that is independent of the clinical coordinating team also can enhance the integrity of the trial and reduce the potential for bias. In all of our PCORI-funded trials, we established a Scientific Advisory Committee, or Executive Committee, as well as various stakeholder groups specific to the PC and ACP interventions being implemented to provide ongoing guidance and feedback throughout the course of the trial. Stakeholder groups can play a critical role in providing strategies to overcome cultural barriers to the integration of PC and ACP interventions into clinical practice. Once a team governance structure is established, investigators should set clear guidelines for study team meetings as well as a reporting structure to ensure adequate oversight over the trial operations. Developing clear standard operating procedures will help overcome the challenges often experienced with study staff turnover. Finally, it is critical to have a strategy to address potential conflicts within the study team. The Study Executive Committee can often play that role for these large-scale clinical trials, or even outside entities such as cooperative groups or the Scientific Advisory Committee.

FIGURE 1 Example of team governance structure.

It is important to hire a project manager with prior experience in managing large-scale studies and exceptional communication skills to manage study staff across participating sites. PC and ACP clinical trials often enroll seriously ill patients and their caregivers (ie, family or friends) during the most trying times in their lives. Interacting with seriously ill patients and their caregivers daily can be especially challenging for study staff, who may have little experience dealing with this population. A compassionate project manager can play a pivotal role in supporting study staff by helping them process difficult patient encounters and emotions when patients die during the study.

Site and Study Staff Training and Engagement

Training for multisite PC and ACP trials is an important task throughout the study conduct to ensure meticulous implementation of the study protocol and procedures across sites (Table 1). It is helpful to conduct an initial in-person study training for the site investigators, study interventionalists, and research staff to provide a detailed overview of the study’s standard operating procedures and to fuel excitement about the importance of the study to advance the science of PC or ACP and help our patients and their caregivers. By cultivating a cohesive sense of a mission-driven community around the study, investigators can garner commitment and dedication from all study staff to ensure the success of the trial. This can also reduce the inherent challenges of asserting the authority of the leading investigators over study staff at participating sites. Working closely with the site investigators to ensure adequate oversight of the study staff also is instrumental.

All our PCORI-funded trials developed comprehensive training manuals, standard operating procedures, and/or templates for study tracking logs to streamline and standardize the study implementation across sites. For large pragmatic trials, procedures for leveraging the electronic health record to identify eligible patients and collect study outcomes must also be incorporated in the training process. As study staff turnover is inevitable, we often video-recorded the training sessions and used them to onboard new study staff who joined later. Ongoing supervision of study activities requires weekly or bi-weekly meetings with study staff at participating sites to address challenges with study implementation and provide ongoing support. It is helpful during these meetings to provide data on site performance across various metrics such as recruitment, retention, intervention delivery, and data collection, which can identify challenges and areas for improvement.

The training must also address common misconceptions about PC and ACP to ensure the study staff are well-prepared to discuss these often difficult topics with patients, clinicians, hospital leaders, and other stakeholders. Leveraging patient representatives and/or PC specialists during the training can provide the research team with the tools to address common issues such as “How do I present this study to my patient without scaring them?” or “This patient is not dying, so they do not need PC or ACP.” As noted earlier, working with seriously ill patients during what may be the most difficult period in their lives can be particularly challenging for study staff. Thus, it is important to acknowledge these challenges during the initial training and provide ongoing support for study staff to process difficult patient encounters and experiences throughout the study period.

Recruiting, Consenting, and Enrolling Study Participants

Recruiting, consenting, and enrolling study participants is one of the primary challenges in PC and ACP clinical trials. Some of our studies used a systematic screening approach to identify all potentially eligible patients and approached them consecutively to assess their willingness to participate in the study.11,16,20,21 A systematic screening and enrollment approach prioritizes study generalizability and reduces the risk of referral bias in clinical trials. This is particularly pertinent in PC and ACP trials as clinicians may have misconceptions about PC and/or ACP that impact their referrals for the trial. Alternatively, some of our PCORI-funded studies relied on pragmatic designs and practice-based or clinician referrals to identify patients for study participation.12 In pragmatic trials, leveraging the EHR to integrate systematic screening into the workflow can be especially helpful for reducing the screening burden on study staff as well as ensuring a sustainable infrastructure for intervention implementation into routine clinical practice. When utilizing EHR algorithms, investigators should test these algorithms and refine their implementation process at each site before initiating enrollment, especially when sites are using different EHRs.

Providing rigorous training for study staff on how best to introduce PC and ACP clinical trials to potential study participants is critical for successful recruitment in these trials. Many of our studies leveraged patient and caregiver stakeholders for mock consents to train the study staff on how to discuss PC and ACP and the overall goals of the trial. Providing educational materials and scripts for study staff also can help standardize the language used to describe PC and ACP. For many of our studies, recruitment challenges were exacerbated by the COVID-19 pandemic and the need to transition recruitment efforts from in-person to remote procedures. Our investigative teams worked closely with the research staff across participating institutions to enhance their skills for remote recruitment and consenting procedures. Involving clinicians in introducing studies can also be a powerful tool to overcome misconceptions about PC and ACP among potential study participants. For many pragmatic comparative effectiveness trials, the goal is to implement and disseminate evidence-based ACP and PC interventions into clinical practice. In these studies, informed consent can often be waived, subject to the approval of the IRB, which can guarantee that everyone who is eligible is included and help enhance the generalizability of the results.22

Some of our studies incorporated systematic strategies to ensure equitable recruitment of participants who are underrepresented in clinical research.20 These strategies include setting goals for the recruitment of underrepresented populations, developing a plan that included translating all necessary study materials into multiple languages, engaging stakeholders from underrepresented backgrounds to train and troubleshoot the recruitment process, hiring diverse research staff that reflect the diverse backgrounds of the patients receiving care at the participating sites, involving patient/family advisors in crafting recruitment messaging, monitoring recruitment targets, and developing a corrective action plan as needed to achieve recruitment goals.

We learned through our studies that meticulous oversight is necessary to ensure successful screening, recruitment, and enrollment procedures. The study team must regularly monitor the number of eligible patients at all sites as well as recruitment rates. For studies leveraging pragmatic screening with the EHR, ensuring the consistency and accuracy of the algorithm in implementing the screening procedures is paramount.22 The study team must carefully review the consort diagram to monitor screening, recruitment, and enrollment numbers on a monthly basis. In pragmatic trial designs that leverage cluster randomization, attention must be paid to patient factors that might be imbalanced between study groups when reviewing the consort on a monthly basis. We often used the study team meetings to discuss the site performance and troubleshoot screening and recruitment challenges. Comparing performance across sites can help identify areas for improvement. Furthermore, engaging in ongoing discussions that leverage the knowledge gained from high-performing sites can be useful in overcoming challenges in recruitment at low-performing sites.

Intervention Delivery and Fidelity

A detailed discussion of how to monitor intervention fidelity is addressed in another article in this issue.

Data Collection and Analysis

Standardizing and streamlining the data collection process for multisite trials is instrumental to successfully implement the study across multiple sites. Our studies worked collaboratively with the site investigators and data coordinating center (if applicable) to build an easy-to-use study database with clear definitions of the data being collected. In many PC and ACP clinical trials, participant-reported outcomes are used to measure clinical outcomes. Investigators should provide maximal flexibility for data collection methods, including in-person, email, and phone administration of participant-reported outcomes, which can reduce the risk of missing data. Many of our studies aligned the data collection time points with the clinical care of these patients, which can further reduce the risk of missing data.

Similar to recruitment monitoring, we learned that ongoing oversight is needed to monitor data collection and missing data rates across study sites by reviewing the consorts, comparing missing data rates across sites, systematically collecting reasons for missing data, and intervening when problems related to missing data can be modified. Similarly, we often discussed strategies with study staff across sites to optimize data collection. Unfortunately, seriously ill study participants are often hospitalized or too ill to complete participant-reported outcomes. Thus, we trained research staff on when to approach seriously ill patients for data collection and address such scenarios in the context of PC and ACP trials. Being cognizant of the length of the participant-reported outcome assessment battery, as well as using adaptive survey design and prioritizing the most important outcomes in survey packets, can help reduce participant burden and optimize data collection for the most critical trial outcomes. Given challenges with the response rate in the context of serious illness, attrition due to death, and other reasons for missing data in multisite PC and ACP trials, investigators must be thoughtful in estimating the rate of missing data, anticipating higher attrition than often seen in single-site pilot trials, and ensuring adequate power for the study.

Although patient-reported outcomes are increasingly recognized by regulators, clinicians, and patients as critical endpoints to collect patient-centered data,23,24 healthcare systems and policymakers continue to require data on healthcare utilization and EOL outcomes to invest in effective PC and ACP healthcare delivery models. As a result, many PC and ACP clinical trials collect data healthcare utilization and end-of-life outcomes for study participants, including hospitalizations, death date, place of death, and hospice utilization. Most of our studies, including pragmatic trials, used the EHR to identify these important study outcomes. However, there are inherent limitations to this approach, given the lack of access to EHR data outside of the participating sites. Contacting bereaved caregivers or clinicians also is a strategy that can be used to confirm and provide additional details on end-of-life outcomes. Study staff should be sensitive when collecting data from bereaved caregivers in PC and ACP clinical trials and should receive extensive training on how to contact family or caregivers and collect these data. For example, providing study staff with scripted language to express their condolence and thank the caregivers for the patients participation in the study can be especially helpful. Ensuring appropriate timing of collecting bereavement outcomes is also important by waiting at least 3–4 weeks after death when appropriate.25,26 In addition, confirming death dates in real-time can be challenging in PC and ACP trials. We often trained study staff to search for obituaries before contacting study participants and empowering them with language to use if they encountered a bereaved family without being aware of the death.

Managing Multisite PC and ACP Clinical Trials

We found several stratregies to be helpful in managing the complexity of multisite PC and ACP trials. Building a sense of community around the study purpose, mission, and overall goals among all study staff across participating sites will create a recipe for success. Many of our studies used regular newsletters and highlighted study accomplishments across sites to boost morale (Fig. 2). Engaging sites in collaboration and contributions that are meaningful to them, such as authoring manuscripts, serving on study subcommittees, or presenting at local community meetings describing their experiences in implementing the trial at their site, can also help enhance engagement. Providing mentorship and support to junior investigators across participating sites is also a helpful strategy to build a networking community. Our studies used regular meeting cadence to build a sense of community among research staff, clinicians delivering the PC or ACP interventions, and site investigators. Providing local site investigators with slide decks to education clinicans about the study procedures and to brainstorm solutions for challenges, such as recruitment can be helpful.

FIGURE 2 An example of a study newsletter sent to all study sites, investigators, and clinicians delivering interventions in one of our ACP trials. ACP indicates advance care planning.

Managing multisite trials also requires oversight and constant monitoring of site performance across numerous metrics, including screening, recruitment, intervention delivery and fidelity, and data collection.. At regular meetings with study sites, it is useful to ask about and document disruptions that could impact the trial such as mergers or restructuring, EHR changes that are especially critical for pragmatic trials, or leaves of absence of key clinicians. In large studies, the responsibility to visit sites in person can be delegated among the executive committee. Furthermore, a member of the leading investigators should strive to visit a site on short notice in response to significant changes in recruitment; often, firsthand conversations and observations of the study environment and team interpersonal dynamics can allow the study leaders to identify barriers that might impact recruitment. The study team must also create a welcoming environment to allow study staff from across sites to reach out for additional help and support. Figure 2 provides a sample newsletter from our PCORI-funded studies, depicting site performance metrics that can be helpful to share with sites to provide additional motivation to meet study goals. We also often leveraged our stakeholders, as well as a scientific advisory committee, to discuss challenges and overcome barriers in implementing these comparative effectiveness trials across sites.

CONCLUSIONS

Although multisite PC and ACP trials pose many challenges, our experiences in conducting these trials are informative and may shape current and future approaches to leading studies. There are numerous lessons learned and strategies that can help ensure the success of these trials and advance the science of PC and ACP. Close and attentive monitoring of the sites and frequent collaboration with them can overcome many of the challenges in conducting this research. We are fortunate in the PC and ACP community to be collaborating with colleagues who are dedicated to enhancing our understanding of how to best serve our patients living with serious illness and their caregivers. This dedication is instrumental in building a cohesive sense of community around the most critical research questions that PC and ACP trials attempt to answer. Our experience underscores the need to expect and anticipate challenges across numerous domains of study conduct, some of which are specific to conducting PC and ACP clinical trials. By anticipating these challenges, investigators, funding agencies, and professional organizations can proactively set strategies, realistic timelines, and goals for our studies to ensure their success.

ACKNOWLEDGMENTS

The authors would like to acknowledge the patients, caregivers, and other advisors who were integral partners in the studies, as well as the core research team members and staff at all study sites, particularly: Anna Rahman, Torrie Fields, Richard Brumley, Michael Rabow, and Wendy Mack for the Expanding Access to Home-Based Palliative Care study.

The UC Health Care Planning study team members Neil Wenger, Anne Walling, Juan Carlos Antonio Lopez, Douglas Bell, Kirsten Buen, Anna DePaolis-Dickey, Ron Hays, Victor Gonzalez, Javier Sanz, Katherine Santos, Chi-hong Tseng, Rebecca Sudore, Axel Hererra, Leah Karliner, Jonathan Lee, Brookelle Li, Kanan Patel, Christine Ritchie, Andrew Robinson, Gabriela Vanegas, Aiesha Volow, Maryam Rahimi, Jamie Anand, Aaron Chau, Valerie George, Rick Marshall, Eileen Sabino-Laughlin, Megan Whalen, Keeta Scholl, and Judy Thomas

Executive Committee members Manisha Verma, Andrzej Kosinski, Tamar Taddei, Michael Volk, Kavitha Ramchandran, and Marie Bakitas, and all clinical centers and Duke Clinical Research Institute’s team members participating in the PAL LIVER study.

Statistical and data team members Avi Alkon, Alyssa Platt, and Linda Sanders; co-investigator Marisette Hasan; Kenisha Bethea and Nadine Barrett; and site PIs Tammie Quest, Ronit Elk, Raegan Durant, Marie Bakitas, Deborah Ejem, Ramona Rhodes, and Anu Gangavati for the EQUAL ACP study.

The REACH PC study site PIs Joseph Greer, James Tulsky, Toby Campbell, Leslie Blackhall, Stacy Fischer, Mike Rabow, Finly Zachariah, Eytan Szmuilowicz, Amelia Cullinan, Laura Shoemaker, Laura Hanson, Maria Bakitas, Maria Silveira, Tammie Quest, Sara Martin, Kostas Leventakos, Josephine Feliciano, Carl Grey, Sean O’Mahony, Lori Spoozak, and Jessica Bauman.

The Meta-LARC ACP trial team.

Core research team members Keith S. Goldfeld, Allison M. Cuthel, Kaitlyn Van Allen, and Mara Flannery; Site Principal Investigators Anar Shah, Kei Ouchi, Carolyn K. Holland, Chinwe Ogedegbe, Erin Zimny, Scott Dresden, Jason Bischof, Carter Neugarten, Richelle Cooper, Christopher Coyne, and Karen Jubanyik; co-Investigators Tiny Varghese, Susan E. Cohen, Abraham Brody, Brenda Matti-Orozco, Joshua Lakin, Paige Barker, Jose Contreras, David Henkin, Erin Stevens, Melanie Smith, Elena Kuzin-Palmeri, Arum Kim, Joseph Lowy, Jennifer S. Scherer, Nancy E. Bael, Jeffrey Berger, Christopher Pietras, Kira Skavinski, and Dmitry Kozhevnikov for the EMPallA study.

This work was (partially) supported through the following Patient-Centered Outcomes Research Institute (PCORI) Awards: IHS-1609-36518-IC, IHS-1602-34070, PLC-1609-36306, PLC-1609-36381, PLC-1609-36714, PLC-1609-35995, PLC-1609-36277, PLC-1609-36291, and PLACER-2020C3-21247. All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the view of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or Methodology Committee. This work was also supported by the Fan Fox & Leslie R. Samuels Foundation award 108995.

A.E.-J. serves as a consultant for Incyte Corporation, GSK, and Tuesday Health. S.E. received grant funding for palliative care-related research/work from California Healthcare Foundation, Blue Shield of California, and Stupski Foundation. The remaining authors declare no conflict of interest.
==== Refs
REFERENCES

1 Morrison RS Meier DE Arnold RM . What’s wrong with advance care planning? JAMA. 2021;326 :1575–1576.34623373
2 Verma S Hingwala J Low JTS . Palliative clinical trials in advanced chronic liver disease: challenges and opportunities. J Hepatol. 2023;79 :1236–1253.37419393
3 Robbins-Welty GA Webb JA Shalev D . Advancing palliative care integration in hematology: building upon existing evidence. Curr Treat Options Oncol. 2023;24 :542–564.37017909
4 Kavalieratos D Corbelli J Zhang D . Association between palliative care and patient and caregiver outcomes: a systematic review and meta-analysis. JAMA. 2016;316 :2104–2114.27893131
5 McMahan RD Tellez I Sudore RL . Deconstructing the complexities of advance care planning outcomes: what do we know and where do we go? A scoping review. J Am Geriatr Soc. 2021;69 :234–244.32894787
6 Gaertner J Siemens W Meerpohl JJ . Effect of specialist palliative care services on quality of life in adults with advanced incurable illness in hospital, hospice, or community settings: systematic review and meta-analysis. Brit Med J. 2017;357 :1756–1771.
7 Jimenez G Tan WS Virk AK . Overview of Systematic Reviews of Advance Care Planning: Summary of Evidence and Global Lessons. J Pain Symptom Manage. 2018;56 :436–459 e25.29807158
8 Burman WJ Reves RR Cohn DL . Breaking the camel’s back: multicenter clinical trials and local institutional review boards. Ann Intern Med. 2001;134 :152–157.11177319
9 Goodlett D Hung A Feriozzi A . Site engagement for multi-site clinical trials. Contemp Clin Trials Commun. 2020;19 :1–6.
10 Greer JA Temel JS . Pitfalls and prospects—lessons Learned from early palliative care research. NEJM Evid. 2022;1 :2766–2782.
11 Chua IS Zachariah F Dale W . Early integrated telehealth versus in-person palliative care for patients with advanced lung cancer: a study protocol. J Palliat Med. 2019;22 :(S1 )7–19.
12 Totten AM Fagnan LJ Dorr D . Protocol for a cluster randomized trial comparing team-based to clinician-focused implementation of advance care planning in primary care. J Palliat Med. 2019;22 (S1 ):82–89.31486729
13 Hemming K Haines TP Chilton PJ . The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. Brit Med J. 2015;350 :1791–1812.
14 Hemming K Taljaard M . Reflection on modern methods: when is a stepped-wedge cluster randomized trial a good study design choice? Int J Epidemiol. 2020;49 :1043–1052.32386407
15 Grudzen CR Schmucker AM Shim DJ . Development of an outpatient palliative care protocol to monitor fidelity in the emergency medicine palliative care access trial. J Palliat Med. 2019;22 (S1 ):66–71.31486726
16 Walling AM Sudore RL Bell D . Population-based pragmatic trial of advance care planning in primary care in the University of California Health System. J Palliat Med. 2019;22 (S1 ):72–81.31486723
17 Green JM Goodman P Kirby A . Implementation of single IRB review for multisite human subjects research: Persistent challenges and possible solutions. J Clin Transl Sci. 2023;7 :e99 1–5.37250991
18 Oversight P . Key Insights: Health Care Staffing Shortages. United States Government; 2023.https://www.pandemicoversight.gov/oversight/our-publications-reports/health-care-staffing-shortages.
19 Glatter R Papadakos P Shah Y. American health care faces a staffing crisis and it’s affecting care. 2023. Accessed February 25, 2023. https://time.com/6291392/american-health-care-staffing-crisis/
20 Ejem DB Barrett N Rhodes RL . Reducing disparities in the quality of palliative care for older african americans through improved advance care planning: study design and protocol. J Palliat Med. 2019;22 (S1 ):90–100.
21 Aslakson RA Chandrashekaran SV Rickerson E . A multicenter, randomized controlled trial of perioperative palliative care surrounding cancer surgery for patients and their family members (PERIOP-PC). J Palliat Med. 2019;22 (S1 ):44–57.31486730
22 Ford I Norrie J . Pragmatic trials. N Engl J Med. 2016;375 :454–463.27518663
23 Kluetz PG O’Connor DJ Soltys K . Incorporating the patient experience into regulatory decision making in the USA, Europe, and Canada. Lancet Oncol. 2018;19 :e267–e274.29726391
24 Mercieca-Bebber R King MT Calvert MJ . The importance of patient-reported outcomes in clinical trials and strategies for future optimization. Patient Relat Outcome Meas. 2018;9 :353–367.30464666
25 Steeves R Kahn D Ropka ME . Ethical considerations in research with bereaved families. Fam Community Health. 2001;23 :75–83.11401625
26 Beck AM Konnert CA . Ethical issues in the study of bereavement: the opinions of bereaved adults. Death Stud. 2007;31 :783–799.17886410
