
==== Front
Med Care
Med Care
MLR
Medical Care
0025-7079
1537-1948
Lippincott Williams & Wilkins Hagerstown, MD

MDC-D-23-00385
10.1097/MLR.0000000000002037
00008
3
Original Articles
Palliative Care and Advance Care Planning Intervention Fidelity Monitoring
Methods and Lessons Learned From PCORI-Funded Large-Scale, Pragmatic Clinical Trials
Verma Manisha MD, MPH *Manisha.Verma@jefferson.edu

Grudzen Corita R. MD, MSHS †grudzenc@mskcc.org

Izumi Seiko PhD, RN ‡izumis@ohsu.edu

Wenger Neil MD, MPH §nwenger@mednet.ucla.edu

El-Jawahri Areej MD ∥ael-jawahri@partners.org

Ejem Deborah PhD ¶tejem@uab.edu

Aslakson Rebecca A. MD, PhD rebecca.aslakson@uvmhealth.org
#
* Department of Medicine, Jefferson Einstein Hospital, Thomas Jefferson University, Philadelphia, PA
† Memorial Sloan Kettering Cancer Center, New York, NY
‡ School of Nursing, Oregon Health and Science University, Portland, OR
§ UCLA Department of Medicine, Los Angeles, CA
∥ Division of Hematology-Oncology, Massachusetts General Hospital, Boston, MA
¶ University of Alabama at Birmingham School of Nursing, Birmingham, AL
# Department of Anesthesiology, University of Vermont Medical Center, Burlington, VT
Correspondence to: Manisha Verma, MD, MPH, Department of Medicine, Einstein Healthcare Network, 5501 Old York Road, Philadelphia, PA 19141. E-mail: manisha.verma@jefferson.edu.
10 2024
6 9 2024
62 10 680692
Copyright © 2024 The Author(s). Published by Wolters Kluwer Health, Inc.
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution-Non Commercial-No Derivatives License 4.0 (CCBY-NC-ND), where it is permissible to download and share the work provided it is properly cited. The work cannot be changed in any way or used commercially without permission from the journal. http://creativecommons.org/licenses/by-nc-nd/4.0/

Over the past decade, the Patient-Centered Outcomes Research Institute (PCORI) funded multiple large-scale, comparative effectiveness clinical trials evaluating palliative care and advance care planning interventions. These are complex multicomponent interventions that need robust but flexible fidelity monitoring. Fidelity is necessary to maintain both internal and external validity within palliative care intervention research and to ultimately evaluate the real-world impact of high-quality interventions. Different trials not only took varying approaches to fidelity monitoring but also uncovered both unique and common challenges and facilitators. This article summarizes 8 of these trials and highlights approaches, adaptations, barriers, and facilitators for intervention fidelity monitoring. Identifying and delivering core elements while simultaneously allowing adaptations of noncore elements is a vital part of fidelity monitoring. Dissemination of such experiences can inform both future palliative care research as well as ongoing implementation of palliative care and advance care planning interventions across diverse clinical practices. Adoption of rigorous intervention fidelity methods is critical to advancing the science and reproducibility of palliative care interventions.

Key Words:

palliative care
advance care planning
adaptations
implementation
fidelity monitoring
SDCT
OPEN-ACCESSTRUE
==== Body
pmcClinical trials focused on palliative care (PC) delivery and advance care planning (ACP) involve complex interventions delivered by diverse practitioners in varied settings to patients with heterogeneous serious illnesses. These interventions are comprised of multiple components working independently or interdependently to impact patient, caregiver, and health care system outcomes, hence requiring flexible intervention fidelity monitoring with adaptations in the given context.1,2 Fidelity monitoring is “the ongoing assessment, measurement, and enhancement of the reliability and internal validity of a study” and is essentially understood as “whether an intervention is delivered as planned by the developers.”3,4 Fidelity monitoring includes both treatment integrity and treatment differentiation.5,6 Treatment integrity is the degree to which an intervention is delivered as intended, while treatment differentiation ensures that comparators consistently differ during study conduct. Adaptations to the intervention are almost always natural in PC and ACP intervention trials as it is critical to tailor care to specific individual patient needs at that point in time and study context. In fact, fidelity and adaptation are essential in PC and ACP intervention trials.7 There is, however, a fine line between intervention fidelity and adaptations, as it can impact the results of the trial. The decline in fidelity or unaccountable adaptations can contribute to Type III errors and even underestimate the effectiveness of the interventions.8,9 Hence, the robustness of maintenance of intervention fidelity with built-in allowable and trackable adaptations within pragmatic PC clinical trials becomes even more critical.10

Over the past 8 years, the Patient-Centered Outcomes Research Institute (PCORI) funded multiple large models of PC-focused and ACP-focused comparative effectiveness clinical trials for patients with serious illnesses and their family members. Of the multiple trials, 7 were funded under a common large, palliative care–focused funding announcement and are completed or near completion. Another 3 trials were funded as part of other PCORI funding opportunities. All trials tested the effectiveness of models of PC and population-level ACP interventions in real-world settings. Since 2017, the Palliative Care Learning Network (PCLN) established by PCORI brought these trial investigators together to refine their approaches and outcomes, fostering shared learning among them. Examples include developing payment plans, strategizing recruitment and site management, and aligning outcome measures. In this context, the partnerships within the PCLN have also allowed multiple teams to optimize their approaches to intervention fidelity monitoring.

This article discusses approaches to intervention fidelity monitoring across 8 of the 10 PCORI-funded clinical trials of models of PC (Section A; N=5) and ACP interventions (Section B; N=3) participating in the PCLN. Two of these trials were paused before completion due to local systematic barriers and are not included in this paper. We summarize approaches, adaptations, barriers, and facilitators and make recommendations based on key lessons learned.

METHODS

Since 2017 and through the Palliative Care Learning Network, PCORI has organized periodic biannual to quarterly group meetings of principal investigators (PI), project managers, and project biostatisticians, during which each team provided study updates on enrollment, challenges, adaptations, and study progress. Each study team used a structured mechanism to describe their approach and adaptations for fidelity measurement; commonalities and differences among the trials were identified, assessed, and collected for this article. In addition, each trial PI nominated one of their team investigators to represent their team and study approaches to intervention fidelity measurement and adaptation. The manuscript was generated by summating the different study approaches and reflecting on differences and similarities across studies. (Appendix A, Supplemental Digital Content 1, http://links.lww.com/MLR/C865).

Conceptual Framework of Intervention Fidelity

FIGURE 1 Conceptual framework for intervention fidelity. EHR indicates electronic health records.

Evidence-based interventions need to quantify the fidelity with which the interventions were implemented during the experimental phase (Fig. 1). There is no consistent conceptual framework for intervention fidelity in palliative care research. Rather, most scientists draw upon a combination or create one specific to the focus of the intervention using process/delivery model or clinical practice guidelines.11 We draw upon the general frameworks (eg, CFIR/Consolidated Framework for Implementation Research) and palliative care–specific insights from Ang et al’s systematic review on conceptualizing the key elements for intervention fidelity in PC trials included here. The recommended 5 core intervention fidelity elements include context, intervention design, interventionists/practitioners, delivery of intervention, and participant’s responsiveness (receipt and enactment).12 We have added adaptations to this list of core fidelity elements based on our experiences and observations. The elements are described below as relevant to PC and ACP interventions.Context: Refers to the structures and culture of the organization, social, political, and environmental forces, and the linkages established as a part of intervention planning. For models of PC trials, context related to access and availability of PC providers to deliver the intervention and processes of collaboration between them, and the primary physicians of study populations play an important role. In addition, methods of approaching and enrolling patients who are proposed to benefit from PC services, but PC is not generally a part of their routine care, require culture transformation through education and collaboration. This is also a critical element for pragmatic ACP trials involving population-based approaches, particularly those requiring integration of the intervention within the framework of Information Technology. The linkages and communication between the study and IT teams are necessary for such interventions.

Design: Encompasses 4 key domains: theoretical foundation of the intervention and its targeted primary outcome (ie, how and why the intervention is designed to impact its primary outcome), protocol/ manual of the intervention outlining the core and adaptable components (ie, guide to the interventionist on what are the key elements which cannot be missed), dosage (ie, frequency and duration of the intervention visits), and content/focus of each intervention visit.

Interventionists (i.e., staff delivering the intervention): Given the shortage of specialty PC clinicians, there is increasing focus on nonspecialty providers and other health professionals (together referred to as health care practitioners) to acquire additional primary PC skills. This requires standardized interventionist training and ways to assess competence and skill acquisition. Initial training coupled with ongoing refreshers can serve as the foundation for nonspecialty providers to be trained to deliver PC interventions. In addition, health care team members from other professions (eg, nurses) and community health workers can also be trained to deliver these interventions.

Delivery: Includes the approach for intervention implementation (what, who, where, and how) and its delivery to enrolled participants (patients and caregivers). Ultimately, the intervention should be delivered as planned and proposed by the research investigators, including the mode of delivery of the intervention (such as in-person or telehealth-based) and provider adherence to recommendations. The study processes from enrollment to study completion can vary based on individual system workflows within a standardized protocol. The delivery system needs to be modified during the experimental/evidence-generation phase for both PC and ACP trials. Clear understanding and documentation of what was changed to allow the intervention to happen is critical for future replication. The collaborative partnerships within and between all care providers to ensure the quality, access, and conduct of these trials are paramount.

Participant’s responsiveness: Refers to measurement of the understanding of receiving intervention by the enrolled patients and family members. This is difficult to ascertain, given the serious illness, care settings (which can be acute), and patients with dismal prognosis. Hence, we consider this an optional fidelity element in PC and ACP trials. This can be assessed as an outcome related to satisfaction with the intervention received or change in patient or caregiver-reported understanding of palliative care, receipt of goal concordant care, and/or completion of advance directives. These can be assessed separately using validated surveys and embedded or nested qualitative interviews of enrolled participants (as a part of the summative evaluation). It is important to try to include this element in PC and ACP research (when doable) as this will help assess how much participants can enact upon a PC intervention irrespective of the other core fidelity elements. Research shows that recipients with higher enactment potentially have better outcomes related to intervention delivery.12 This hypothesis needs further exploration in PC and ACP trials.

Adaptations: Refer to the flexibility in intervention delivery tailored to individual needs, which are pivotal for PC and ACP intervention trials. The adaptations encompass variability in mode, dose, and content of the intervention, determined by the patient and provider interactions and decisions at the point of care/service.14 Adaptations are an essential feature to be built within the intervention implementation and actively tracked/monitored, as these can serve as critical facilitators for both providers and participants.13 Although fidelity and adaptation may seem contradictory to each other, in PC and ACP pragmatic trials they both are necessary. The key is to keep the core components of intervention intact while simultaneously capturing adaptations as and when made. In pragmatic trials, adaptations often emerge during the study as the trial proceeds, as observed in all the trials discussed here. Commonly the adaptations intend to help with recruitment and retention and facilitate the delivery of intervention. Ideally, the core elements of an intervention and a codified pathway for naturally adapting those elements within different contexts are outlined before a trial is launched. However, natural adaptation is also a component of successful implementation in palliative care research.

Table 1 outlines the details of core elements and adaptations within each of the 8 trials included here.

TABLE 1 Core and Adaptable Elements of Intervention Fidelity Across Models of Palliative Care Delivery and Advance Care Planning Trials

Study	Intervention fidelity-core elements	Intervention fidelity-adaptable elements	
Models of care	
 PERIOP-PC	- Four core domains of PC (education of patient and family, use of standardized symptom assessments, and availability of multidisciplinary and interprofessional resources) encouraged and tracked, covered within ~60 min.
- Standard dose of PC (monthly visits) based on empirical data.
- PC intervention delivered by PC subspecialists and subspecialist teams.
- Standardized training materials for PC clinicians in both study arms, to either activate surgical teams about PC consultation guidelines or educate palliative teams about early-stage upper GI cancer prognoses and treatments.
- Completion of post-visit surveys assessing the delivery of PC by the Interventionists.
- Weekly site coordinator meetings to encourage co-learning regarding ongoing challenges in intervention delivery- Spot-checking EHR review of de-identified PC notes to assess content.
- EHR review to track PC visits in both study arms.	- PC subspecialists are encouraged to use a full spectrum of skills to assess and respond with treatments tailored for specific patient and family needs and wishes.
- Delivery of PC could be in-person or telephonic, per patient/family preference.
- Surgeons are encouraged to follow NCCN Guidelines for Palliative Care consultation but PC consultation in the control arm was ultimately a surgeon-specific and patient-specific decision.	
 PAL LIVER	- There are 4 study visits (initial, 1, 2, and 3 mo) in both comparative groups (PC delivered by Subspecialists vs. Trained Hepatologists).
- Standardized symptom, depression, and distress assessment coupled with key NCP domains to be discussed over any of the intervention visits.
- The providers are required to complete the checklist to document what was actually done during that visit, followed by a routine clinical note documentation.
- The executive committee reviews the PC checklist and clinical notes for a select number of cases, to assure consistency and fidelity.	- Providers are given flexibility to cover the content of the PC checklist over separate visits based on the patient’s individual needs and readiness. For example, an ACP discussion can happen at any of the 4 study visits.
- Delivery of PC intervention can be in person or via telehealth at any of the study visits.
- Intervention visits can occur as outpatient or inpatient, depending on the patient’s situation. However, the majority of these visits have occurred as Outpatient consults.	
 EMPallA	Registered Nurse (RN) arm
- All RNs worked centrally, certified in Hospice and Palliative Nurse (CHPN), and received in-house, standardized training on Motivational Interviewing, Problem Solving Theory, Center to Advance Palliative Care (CAPC) palliative care modules, and Respecting Choices Last Steps
- All RNs were licensed in all 9 states where enrollment occurred.
- Initial assessment completed between 48 and 72 h after enrollment, followed by 3- and 6-month visits.
- RNs conducted standardized assessments and followed standardized protocols for visits.
- Notes documented in Epic were structured and templated forms used to capture key data elements such as if contact was made (yes/no), duration of calls, call frequency, and what standard toolkit was used, etc.
- Nurses also completed 3- and 6-month intervention completion checklist in REDCap.
- Oversight by palliative medicine physician, including mock calls, listening in periodically to calls with patients, and discussion at weekly interdisciplinary team (IDT) meetings to discuss each patient and ensure care was consistent across nurses.
Outpatient Physician (OP) arm
- OP physician or nurse practitioner was employed at the respective enrollment site. Physicians were either board-eligible or board-certified in hospice and palliative medicine.
- OP providers conducted an initial standard assessment and completed a REDCap checklist after each visit (initial, 3, and 6 mo) for the domains addressed.
- Central research team held bimonthly meetings with all OP providers to promote best practices and troubleshoot any scheduling needs or other barriers to attendance.	- Nurses (RNs) and Outpatient (OP) providers were able to adapt the visit content and domains addressed to fit each patient’s individual needs (eg, providers were given the flexibility to cover the content of checklists over separate visits).
- In the OP arm, after 2020, the delivery protocol was adapted to include delivery of PC via in-person or telehealth appointments, per patient, family or health system preference.	
 REACH-PC	- Standard dose of PC (monthly visits) until the patient dies.
- Standard training on intervention delivery
- Core domains of palliative care identified and reinforced via PC clinician survey and documented notes.
- The study team reviews intervention fidelity data and addresses any issues and challenges in terms of ensuring fidelity to the core PC domains.
- Refresher training to reinforce the core palliative care domains and address any challenges in intervention delivery	- PC clinicians have the flexibility of focusing on different PC domains during the visits based on patients’ clinical needs and illness trajectory.
- Joint visits with oncology can be used based on the judgment of the clinicians.
- Patients can be seen during their chemotherapy infusion or as a separate clinic visit based on the care setting.
- Palliative care clinicians may refer patients to additional supportive care services at their discretion.
- Timing of serious illness conversation or documentation of end-of-life care preferences is left at the discretion of the PC clinician.
- Caregivers are encouraged but not required to attend PC visits.	
 SCOPE Leukemia	- Minimal dose of PC based on empirical data (twice weekly visits during hospitalization).
- Standard training on intervention delivery (similar to REACH- PC).
- Core domains of palliative care identified and reinforced via PC clinician survey and documented notes.
- The study team reviews intervention fidelity data and addresses any issues and challenges in terms of ensuring fidelity to the core PC domains.
- Refresher training to reinforce the core PC domains and address any challenges in intervention delivery	- Clinicians can increase the frequency of PC visits beyond twice weekly based on the patient’s clinical needs.
- Clinicians have the flexibility of focusing on different PC domains during the visits based on patients’ clinical needs and illness trajectory.
- Clinicians may refer patients to additional supportive care services at their discretion.
- Timing of serious illness conversation or documentation of end-of-life care preferences is left at the discretion of the PC clinician.
- Caregivers are encouraged but not required to attend PC visits	
ACP studies	
 UC Health Planning	- Develop an automated mechanism to identify seriously ill patients for whom ACP is needed, build a mechanism within EHR to prime patients to engage in ACP during an upcoming clinic visit, and prepare primary care clinicians for the interaction.
- Three interventions programmed into the EHR, one including outreach from a health navigator. Standardized messaging across sites.
- Automated analysis of diagnostic codes, encounters, and clinical data, performed weekly.
- Standardized training of the health navigators and education of the primary care physicians.
- Standardized ACP documentation by health care navigators across sites.
- Standardized workflows within health systems, esp., scanning Advance Directives to ensure they are available at the point of care.	- Despite the automated algorithms, manual chart abstraction at each site was needed to ensure the fidelity of the patient identification algorithm.
- COVID-19 induced major changes in clinic attendance that required modification of messaging (within EHRs) and including telehealth-based visits. This was suggested by the study advisory group and required by health systems.	
 EQUAL ACP	- Standardized training of the lay community health workers
- Respecting Choices: Participants receive general ACP information materials and information about choosing a health care agent in the mail. Within two weeks, the lay ACP facilitator conducts an ACP discussion with participants on surrogate decisions, in-person or by phone. The lay ACP facilitator follows up with a phone call 2 weeks after the meeting to answer questions.
- Five Wishes: Participants receive the Five Wishes advance directive form and the Five Wishes Conversation Guide for Individuals and Families. Two weeks after receipt, lay ACP facilitators called to review materials. Four weeks later, lay ACP facilitators called again to follow up.	All interactions of Respecting Choices lay ACP facilitator shifted to Zoom (preferred) or phone during the pandemic.	
 ACP in PBRNs	- Training: (1) standardized in-person Serious Illness Care Program training, (2) online modules to learn about how to promote ACP in primary care settings, (3)use of Serious Illness Conversation Guide (SICG) to facilitate the conversations, and (4) 90 min in-person training with role-play.
- Develop workflow (1) to identify patients, (2) to prepare patients for conversations, (3) to set up a reminder for clinicians to have SIC, and 4) how to document SIC.
- Interventionists in individual-arm are limited to physicians or advance practice providers (APPs), while team members could determine who would be the interventionists in the comparative arm.	- Each clinic decided how to identify patients for ACP based on the existing structure and available resources. (Some clinics chose to use an automated algorithm based on EHR, while others used manual search or team huddle depending on resources available).
- Each clinic decided workflow on how to schedule visits, remind clinicians, document, bill, and follow-up on ACP conversations based on existing structure and available resources.
- Delivery of additional or refresher training after the initial training was determined depending on the clinics’ needs and staff turnover.	
ACP indicates advance care planning; EHR, electronic health records; NCCN, National Comprehensive Cancer Network.

Approaches to Intervention Design, Implementation, and Fidelity Monitoring

Section A: Trials to Evaluate Models of Palliative Care Delivery

Five of the 8 trials compared the effectiveness of different models of PC delivery across diverse care settings. Key evaluated components of intervention include—level of integration (eg, consultative/subspecialist PC vs. primary PC delivered by non-PC providers or health care practitioners), site of palliative care delivery (eg, home-based, outpatient, inpatient, and emergency department), and mode of care (eg, in-person and telehealth). Guidelines for most PC intervention visits were adapted from the National Consensus Project for Quality Palliative Care.15–17 Key components of PC interventions included:education of patients and family caregivers about the disease and its prognosis;

management of symptoms and treatment options;

formal symptom and distress assessments, and

access to a multidisciplinary, interprofessional PC team.

Key fidelity measures include standardized training with refreshers, structured PC checklists to document what was discussed, clinical note documentation, quality assurance checks of clinical documentation with checklists by the research team, routine sharing of best practices among the interventionists, capturing dosage over time and allowing adaptations to workflows, and mode of delivery given the pragmatic and individualized nature of the interventions. Some trials considered conducting audio recordings of the intervention visits to assess fidelity but did not pursue it. This was primarily due to strict privacy issues and physician concerns about recording sensitive information. Notably, many trials included qualitative study aims involving interviewing patients, caregivers, and interventionists/ providers to better understand patients’, family members,’ and practitioners’ lived experiences during intervention delivery.

Fidelity monitoring has been conducted throughout these trials by the investigators and their teams. Table 2 outlines the challenges, facilitators, and lessons learned from these trials. All the standardized checklists utilized in these trials are compiled as a Supplementary Appendix, Supplemental Digital Content 1, http://links.lww.com/MLR/C865. This appendix includes details from each trial, developed by researchers and clinicians with experience in PC and ACP research.

TABLE 2 Approaches to Intervention Design and Implementation in Multisite Comparative Effectiveness Palliative Care Trials.

	Intervention design	Actual implementation	Challenges and facilitators	Lessons learned	
Study Section A	
Models of care	
 PERIOP-PC	PC by a trained subspecialist and delivered in 5 visits over 3 mo vs. enhanced usual care with primary team encouraged to follow PC consultation guidelines.
Minimal protocolization of content/ focus of PC.
Unit of randomization: Patient.	Intervention delivered as planned. Patients highly preferred telephonic PC visits, as opposed to in-person.	Minimally protocolized visits by PC subspecialists required less training but visit content and focus varied across providers and sites.	Difficult to attribute which components of PC contributed to study outcomes.
Inherent need for PC clinicians to tailor the intervention to the unique needs and circumstances of patients.	
 PAL LIVER	PC delivered by trained subspecialist vs. hepatologists who received primary PC training through online and virtual live sessions. The intervention is delivered over 4 monthly visits within 3 mo.
Content of PC in both study arms is guided by the same standardized checklist.
Unit of randomization: Cluster by Clinical Center.	Intervention delivered as planned. Modifications included the addition of a telehealth option for all intervention visits (due to the COVID-19 pandemic and stakeholder suggestions during year 1)	Virtual training program connected and trained hepatologists across geographically diverse centers. The training was tailored to liver disease–specific scenarios (based on an Oncology PC program)
Caregiver-related intervention was difficult to document in EHR.	Online training works well for practicing physicians. Refresher courses with live sessions facilitated difficult patient scenario discussions.
Majority of the intervention checklist was covered during the initial study visit; monthly visits could be better tailored to unique patient and clinician needs.	
 EMPallA	PC delivered by RN with certification in palliative nursing via phone, vs. in-person PC subspecialist during outpatient visits over 3 visits within 6 mo.
Content of PC service guided by the National Consensus Project and tracked across both arms.
Unit of randomization: Patient.	Substitution of telehealth, instead of an in-person visit by PC subspecialists (outpatient intervention arm) due to the COVID-19 pandemic.	Telehealth and telephonic care are well received by patients and caregivers.
High staff turnover required repeated training.	Important to have a methodology and infrastructure to measure the “dose” and components of palliative care.
Standardized training curriculum required significant upfront effort yet had multiple downstream benefits.	
 REACH-PC	PC is delivered by subspecialists monthly, either in-person or via telehealth, until the patient dies or discontinues. Patients can be in the study for up to 5 y.
Minimal protocolization of content/ focus of PC.
Unit of randomization: Patient.	Increased drop-out rates when patient oncology visits were not monthly.
A major challenge when the COVID-19 pandemic limited in-person care, which led to pausing enrollment in the in-person arm.	Intervention allows for the potential avoidance of in-person PC visits.
Heterogeneity in PC delivery due to variations in subspecialist clinician practices.	Critical need to train specialty PC clinicians on PC intervention.
The challenges of conducting an in-person vs. telehealth intervention trial during the COVID-19 pandemic
The importance of aligning the PC intervention with oncology care plans.	
 SCOPE Leukemia	PC delivered by a subspecialist or hematologic oncologist, with both clinician groups receiving specialized training, during the hospitalization.
Content of PC or subspecialist guided by standardized PC domains.
Unit of randomization: Cluster by Clinical Center.	Intervention Clinicians receive training every 6 wk for the first year and then every 12 wk thereafter.	PC intervention guide and standardized training reduced variability between providers.
Intervention requires training all clinicians, which can be practically challenging.	Important to engage diverse stakeholders at participating sites to ensure trial success, including designating both palliative care and hematologic oncologist champions.	
Section B	Intervention design and providers	Population identification	Challenges and facilitators	Lessons learned	
Planned	Actuality	
ACP studies	
 UC Health Planning	Compared:
(1) AD and targeted ACP messaging,
(2) AD, targeted ACP messaging, and an ACP website, and
(3) AD, targeted ACP messaging, an ACP website, and outreach from a care coordinator.
Unit of randomization: Cluster by Clinic.	Weekly automated analysis of diagnostic codes, encounters, and clinical data.	As planned with further manual chart abstraction.	Though intervention content was standardized, implementation required intensive interaction with the health system and EHR stakeholders.
Inability to determine whether participants opened mailed interventions.	Benefit from standardized training and materials but also need for flexibility and patience when interacting with health system and IT/EHR stakeholders.
Challenges around outcomes and fidelity of administrative EHR data.	
 EQUAL ACP	ACP delivered by patient-guided self-management or a structured facilitator-guided approach.
Unit of randomization: Participant.
Duration of intervention: 3 mo	Review of EMR from clinics associated with an academic medical center and/or federally qualified health centers	As planned	Required video conferencing that was often difficult for some participants to use.
A central IRB streamlined process for involving diverse research sites.	Challenges with identification of participants and ongoing training of research team staff and lay facilitators, particularly at federally qualified health centers.	
 ACP in PBRNs	Comparison of 2 approaches to implement ACP as an intervention in primary care clinics: (1) Serious Illness Conversation (SIC) facilitated and delivered by individual providers; and (2) SIC facilitated and delivered by interprofessional team.
Unit of randomization: Cluster by Clinic
Duration of intervention: 12 mo	Clinicians’ intuition: “no” to the “surprise question.”*	Manual chart review, team huddle discussion, and/or EMR algorithm.	Standardized training materials were helpful.
Having clear descriptions of what are core vs. adaptable intervention components was helpful.
Diversity of each clinic setting results in great variations in how to adapt the intervention.
Adaptation happened organically over time, and accurate tracking of the process and degree of adaptation was challenging.
Difficult to maintain treatment differentiation between arms due to the great degree of variations in adaptation by each clinic.	High turnover of leadership, clinicians, and staff caused the loss of champions and experienced interventionists. To maintain momentum and expected quality of intervention, efforts to provide additional training for new members and continuous engagement of staff were critical.
Lack of appropriate outcome measures to capture clinician and patient receipt of the intervention is a major challenge not only to evaluate the impact of the intervention but fidelity and integrity of the intervention.	
* The “surprise question”—the clinician would not be surprised if the patient died within the next 6 mo.

ACP indicates advance care planning; AD, advanced directive; EMR, electronic medical record; IT, information technology; PBRNs, practice-based research networks; PC, palliative care; RN, nurse.

Below is a brief overview and key fidelity challenge faced by the clinical trials included in this article:Perioperative Palliative Care Surrounding Cancer Surgery (PERIOP-PC)—This multisite randomized controlled trial compared surgical-PC subspecialist team co-management to surgical team alone, for patients pursuing curative-intent surgeries for upper gastrointestinal cancers.18,19 The co-management intervention arm involved 5 visits over 3 months by PC subspecialists either by telephone or in person. The standard of care surgical team met with enrolled patients as needed. The specialist tailored intervention content to the patient’s unique circumstances and fidelity measurement were through the completion of surveys by the palliative care specialist about each patient-specialist interaction.

Palliative Care for Patients with Liver Diseases (PAL LIVER)—This cluster randomized trial compares PC delivered by a PC subspecialist versus PC provided by hepatologists specifically trained to provide PC for patients with end-stage liver disease and their caregivers.20,21 Both intervention arms involved 4 visits over 3 months, and providers completed a standardized checklist to document visit content followed by routine clinical note documentation. The key challenge was to include PC physicians at sites with limited resources to deliver the intervention. However, the engagement of PC physicians by the hepatologists at each site facilitated these collaborative efforts at a very early stage. In addition, standardized training of hepatologists on primary PC and a common checklist used in both models facilitated the maintenance of fidelity in both Models.

Emergency Medicine Palliative Care Access (EMPallA)—This is a multicenter, parallel, 2-arm randomized controlled trial in the Emergency Department comparing 2 established models of palliative care: nurse-led telephonic case management to specialty outpatient PC for older adults with serious, life-limiting illness following Emergency Department discharge.22 The nurse(s) contacted enrolled patients once a week or more (per patient needs) for 6 months. The PC subspecialist physicians met enrolled patients once a month for 6 months. Rigorous training of any new interventionists and completion of checklists after each visit, coupled with regular meetings with interventionists, helped maintain fidelity throughout the study.

Integrated Telehealth versus In-Person Palliative Care for Patients with Advanced Lung Cancer (REACH-PC)—This equivalence randomized trial compares early integrated in-person PC versus telehealth-based PC for patients with newly diagnosed advanced lung cancer and their caregivers.23 All study participants received a monthly PC specialist visit (either in-person or over telehealth) from enrollment until death. All intervention visits are followed by clinical note documentation. Intervention fidelity was challenged during COVID-19, when all nonessential patient-provider interactions suddenly shifted to telehealth-based visits. The study paused enrollment during the acute surge in the in-person arm and then resumed when the situation in the country was better. However, given the sudden circumstances of COVID, the research hypothesis being tested in this model has become even more critical. Fidelity is maintained and assessed using checklists completed at the end of each study visit.

Specialty Compared to Oncology delivered Palliative Care for Patients with Acute Myeloid Leukemia (SCOPE-Leukemia)—This is a cluster randomized clinical trial where participating institutions are randomly assigned to either a specialty PC model or a primary PC model (ie, training oncology clinicians to incorporate palliative care skills in their practice) in hospitalized patients with acute myeloid leukemia (AML). Patients receiving care in specialty PC sites would see specialty PC clinicians at least twice weekly during their hospitalizations for AML. Patients receiving care at primary PC sites would receive care from an oncology clinician trained to incorporate PC skills into their practice.23 The fidelity is challenged by adding more clinical time of the oncologists. However, this is overcome by additional training and a better understanding of primary palliative care among oncologists. In addition, the completion of checklists to document the focus of each study visit maintains fidelity.

Section B: Trials to Evaluate Advance Care Planning (ACP)

Three trials are evaluating varying approaches to facilitate and promote ACP discussions and documentation across diverse care settings and populations. Key evaluated ACP components include patient and/or health care practitioner–directed approaches to facilitate ACP conversations (eg, facilitated ACP communication and standardized serious illness conversations); delivery of information (eg, brochure in the mail or electronically through the patient portal, web-based information, and care coordinator/navigators or care teams connecting with patients); and documentation of goals of care and ACP in the EHR.

Key fidelity measures include standardized training and clinical documentation of ACP, standardized workflows, and automated algorithms and reports built within the EHR to trigger patients or provider messaging (Table 2). Utilizing standardized frameworks such as Consolidated Framework of Implementation Research (CFIR) and Reach, Effectiveness, Adoption, Implementation and Maintenance (REAIM), and capitalizing on existing EHR can promote fidelity to EHR-based ACP interventions.25 Of note, all studies created their own fidelity instrumentation depending on the content and focus of the intervention. In addition, qualitative methods are utilized to better explore how the intervention was implemented within the practice workflow.26 Below is an overview of these trials:University of California (UC) Health Care Planning Trial—This is a cluster randomized trial comparing 3 different approaches to promoting ACP—(1) distribution of an advance directive (AD) with targeted ACP messaging, (2) distribution of the AD, targeted ACP messaging, and prompting patients to engage with an ACP-focused website, and (3) distribution of the AD, targeted ACP messaging, and prompting patients to engage with an ACP-focused website and outreach from a trained care coordinator.27 The target population is all patients who lacked ACP documentation, suffered from a serious illness, and were cared for in 50 cluster-randomized primary care clinics. Conducted within 3 large health care systems, the study used an automated mechanism to identify seriously ill patients for whom ACP was deemed appropriate and built a mechanism within the EHR to prime patients and primary care clinicians to have ACP conversations. Fidelity was constantly assessed using the REAIM framework metrics, including—the number of patients eligible for intervention, number of intervention messages sent, appointment-based versus batch messaging, tracking number of patients active in the patient portal and how many opened the ACP message, and number of clicks onto the prepare for your care webpage.

Quality of palliative care for older African Americans through improved advance care planning (EQUAL-ACP)—EQUAL-ACP is a multisite, matched pair (patient and surrogate decision maker), cluster randomized trial comparing a patient-guided self-management ACP document (five wishes form) with a facilitator-guided and a structured approach for ACP communication (respecting choices first steps) on increasing rates of formal and informal ACP.28 The study population includes older, community-dwelling African American and White adults with serious illness who receive care in 10 primary care clinics across 5 states. In addition, the study examines how racial concordance between participants and interventionists (trained lay community health workers) affects ACP outcomes. Fidelity was maintained by rigorous training of the interventionists coupled with live feedback. However, due to in-person restrictions during the COVID pandemic, the facilitator-guided approach became virtual, which was challenging for fidelity assessment. In addition, due to high staff turnover, repeated training of research staff and lay facilitators made fidelity maintenance a time-intensive process.

ACP in primary care Practice Based Research Networks (PBRNs)—This study compares 2 (individual-focused and team-based) approaches to implementing ACP conversations in primary care clinics within 7 PBRNs in the United States and Canada.29 Participating clinics are randomly assigned to individual-arm or team-arm. Clinics in individual-arm received standardized training for an individual practitioner to facilitate ACP conversations, while clinics in team-arm received the same training modified to facilitate ACP conversations by an interprofessional team.30 The study population was adult patients receiving care in the participating clinics and identified as living with serious illness by clinicians. Participating clinics varied by size, geographical location, and organizational structure. Therefore, there were wide variations in how they adapted and implemented the intervention into their practice. Tracking these individual adaptations to processes was challenging. However, the intervention fidelity was maintained for the core components.

Challenges and Facilitators for Intervention Fidelity Monitoring

The mechanisms to conduct intervention fidelity monitoring within the realm of good palliative care research are diverse and challenging (Table 2). Based on the experiences of the 8 trials discussed here, several common themes that have led to success are shared below. Developing standardized training and checklists: Many study teams noted the benefit of creating a standardized curriculum/training upfront for all the interventionists. This was a challenging task, and many utilized existing resources tailored to study specific populations and settings. Input and early involvement of educators in training development was necessary, for example, PAL LIVER trial partnered with Stanford’s online training program on palliative care. The study team worked rigorously during the initial 6 months to tailor the entire content of an oncology-focused training program to end-stage liver disease–specific case scenarios and clinical applications.21 Standardized checklists and clinical note templates were developed and used to ensure the consistency of intervention content. Early in a study, the generation of such tools often involves significant time and effort but ultimately is beneficial, particularly over a long study and during times of staff turnover. Such tools also ensure internal and external validity for the study intervention and could be used by other researchers, health care practitioners, clinics, and/or health care systems wishing to adopt and implement similar interventions in future studies and/or clinical practice settings (Appendix, Supplemental Digital Content 1, http://links.lww.com/MLR/C865).

Team culture and effectiveness: From the beginning of proposal development and submission, developing a team culture can help lay a strong foundation for intervention fidelity across multiple sites within large-scale trials. Developing a culture of shared goals among the participating institutions with effective team meetings to outline the expectations to maintain the fidelity of palliative care interventions helps conduct trials to maintain fidelity throughout the intervention phase. Considering the intervention as a whole can allow for the development of processes that aim to maintain the intervention’s core elements for each enrolled participant.

Lack of palliative care–specific fidelity measurement tools—Nearly all trials noted significant challenges in selecting and utilizing PC-related and ACP-related fidelity measurement tools that challenged intervention fidelity monitoring and effectiveness evaluation.31 Each trial developed its own toolkit (as detailed in the appendix, Supplemental Digital Content 1, http://links.lww.com/MLR/C865) and process measures. The lack of validated outcomes or process measures for participant understanding of PC and/or ACP principles significantly impaired hypothesis testing across many teams and trials. Many studies attempted to explore this hypothesis instead, through concurrent and nested qualitative research. Some study teams also tracked the number and focus of discussions between clinicians and patients/family members using study checklists (completed by the interventionists after each study visit). Even when study teams used EHR administrative data for outcomes (ie, presence or absence of an advance directive or a goals-of-care conversation), there was still difficulty discerning the reliability, validity, and timestamping of these outcomes. Finally, many teams noted the importance and benefit of study infrastructure and protocols to determine key fidelity-related process measures, such as duration of palliative care visits or phone calls, number of outreach attempts, and frequency of patient “no-shows.” In EQUAL ACP, ACP completion was evaluated by the completion of (1) a formal document [health care power of attorney, living will, Five Wishes, Physician Orders for Life Sustaining Treatment (POLST), or other legal directive] or an informal document (ie, letter) naming a decision maker or describing preference; (2) discussion with clinician documented in the medical charts; and (3) patient report that he/she asked someone to make decisions for him/her or discussed values, goals, or preferences for future medical care with family, friends, or other surrogate decision-makers. Similarly, the enactment of ACP discussion was assessed with (open-ended) survey items as was implemented in the UC Health Care Planning study and ACP in PBRN study.

Competing priorities for interventionists and interprovider variability— Intervention providers often had competing priorities related to their routine work responsibilities, which limited their availability for study interventions. In addition, increased staff turnover during the COVID-19 pandemic required a variety of approaches to train and sustain training of new study interventionists. Particularly for multiyear studies, this training was often complicated by perpetual study and intervention staff turnover necessitating near-continuous, ongoing training and/or reinforcement of knowledge and practices among existing staff. The training for PC and ACP interventions requires establishment of confidence and competence, which usually evolves over time. Skill acquisition was assessed using a questionnaire and self-reported confidence. Many studies described challenges in balancing protocolized intervention delivery with variability among providers, particularly PC subspecialists, to adapt approaches to individual patients’ unique needs and circumstances.

Cultural and temporal changes in routine practice over time—All study teams described needing to adapt intervention approaches and content over time, particularly in response to varying cultures and circumstances. Nowhere was this more apparent than as studies adapted to the COVID-19 pandemic; most studies began enrollment before the pandemic and then mid-stream adopted pandemic-related changes or were even paused for enrollment and intervention visits. As studies also persisted over multiple years, some investigators noted challenges related to real-time cultural and pragmatic evolution in PC and ACP utilization and beliefs among patients, family members, other clinicians, and even health care systems.32 In some studies, such temporal and cultural changes contributed to unavoidable contamination between study arms.19

Limited Resources from Informational Technology (IT)—All the trials utilized telehealth approaches for either training and/or direct intervention delivery. Particularly among trials comparing telehealth with in-person approaches for PC or ACP, trial activities were regularly interrupted or complicated by scarce IT resources. For example, some sites and studies identified potential study participants with IT-based, rapid search of the EHR, while others lacked IT capabilities and instead relied on study coordinators manually reviewing patient, provider, and clinic lists. Teams deploying an intervention that required embedment within an EHR system described significant challenges and delays due to interacting with different IT teams and resources across health care systems. Many studies also described patients, family members, and clinics in under-resourced communities often lack access to necessary IT resources, rendering them unequipped to meet study deliverables.

DISCUSSION

Implementation fidelity monitoring is a systematic, complex, time-sensitive, and time intense process. It is a methodological strategy that contributes not only to ensuring quality interventions but also helps optimize interventions over time and can aid in understanding variations in intervention effectiveness. Enhancing the rigor and reproducibility of future PC and ACP clinical interventions will improve trial design and practice, and integration of evidence-based interventions into routine clinical care. Checklists ensuring intervention delivery and content can be time-consuming to develop but extremely helpful in practice, particularly in large and/or longer ACP and PC clinical trials.

The PCLN experiences and approaches outlined in this article can help inform the future of PC and ACP trials. Table 3 summarizes these recommendations based on the pragmatic experiences in monitoring and maintaining intervention fidelity. Notably, adaptability and flexibility in delivering interventions by providers in different contexts are essential to be considered upfront. However, clarity between the core elements for the specific populations and the degree to which adaptations are allowed such that they do not hamper the core elements and study integrity are fundamental. To optimize the uptake of evidence generated, greater attention to fidelity measurement and reporting will likely benefit future palliative care research.

TABLE 3 Potential Fidelity Checklist Components for Models of PC and ACP Pragmatic Clinical Trials

Fidelity component	Useful things to consider	
1. Intervention design	
 Content/focus of study visits (what is discussed).	ASCO and NCP domain-based PC checklists, standardized description of Serious Illness Care Program and guide	
 Interventionist, ie, the person delivering intervention	Subspecialist PC providers, Nurses, Other Medical Specialists, Community Health Workers, interprofessional primary care teams	
 Duration and frequency of visits, ie, time contact between the provider and patient/family	Time spent during each visit, total duration of PC time, timing between visits	
 Number of visits	Numbers can be tailored based on initial visit (prn) and intervention population	
2. Training of providers	
 Mode of training	Virtual, webinars, or in-person, time needed from trainers and trainees.	
 Refresher courses over time	Plans to train new providers as there is staff turnover over time, role-playing with corrective feedback.	
 Assessing knowledge or required skill set post training	Role-play/ do a test intervention, evaluating the competency of skill set acquired [eg, trainers evaluated subjectively, an audio recording of intervention visits (required proficiency >80%)]	
 Continued assessment of skills to deliver intervention over years.	Experiential learning, monthly calls for continued education, interdisciplinary team meetings to discuss difficult scenarios.	
3. Intervention delivery	
 Mode of delivery	Some studies inherently compare delivery modes and thus, fidelity requires strict adherence to the allocated delivery mode. Yet, when possible, flexibility to include both telehealth-based and in-person visits (depending on patient and provider needs).	
 Identify core and adaptable elements	Required documentation of core elements (eg, symptom assessment and goals of care discussions) and adaptations.	
 Cross-verification of checklist content with clinical documentation in the EHR	Manual review by the research team (virtually or in person) of randomly selected cases, or selected number of cases per site. Clinical note templates can help standardize content.	
 Mitigating contamination, adherence to the assigned group	Protocol manual delineating clarity of ways to avoid cross contamination with the comparative group. If contamination occurs, propose statistical methods to overcome it.	
4. Participant responsiveness	
 Optimal patient and caregiver understanding of the PC or ACP intervention	Standardized measures, participant satisfaction with care received.	
 Qualitative interviews from patients, caregivers, providers	Summative evaluation of intervention delivery and receipt.	
 Completion of ACP	Clinical documentation of ACP (eg, goals of care) and its use in care planning.	
5. Adaptations	
 Workflows at each clinic	The context and workflow design at each location could vary to facilitate implementation in multisite trials.	
 Variability among providers	As long as the core elements are addressed, interprovider variability is an inescapable element of PC trials.	
 Variability of individual patient needs	The key function of PC is to address individual patient needs, irrespective of how it is addressed.	
 Dosage	Frequency of visits and time spent during each consult may vary. However, a delineation of minimum number of visits or contacts must be accounted for.	
ASCO indicates American Society of Clinical Oncology; EHR, electronic health record.

Below are some key takeaways for intervention fidelity in palliative care research:The most effective palliative care interventions are unlikely to produce effects if not implemented with fidelity. Thus, fidelity assessment is a critical component of conducting large-scale PC and ACP trials.

Fidelity measurement is intervention-specific and needs to be tailored to each trial. Multiple methods can be simultaneously employed including periodic debriefing sessions with the interventionists to discuss individual challenges and scenarios.

Standardized training of nonpalliative care providers is commonly used for the initial training of interventionists, with refresher training offered periodically. A training program/ guide can serve as a reference for all interventionists.

Completing a checklist after each visit is a feasible and robust method to track and document what was done during the study visit. The same checklist can also be used to track the duration of intervention to assess “dosage.”

We recommend proactively identifying core elements of the intervention and integrating them into implementation practices and fidelity measures.

Allowing adaptations to noncore elements of the intervention can potentiate and facilitate recruitment, retention, and intervention delivery; documentation of these adaptations is strongly advised.

Supplementary Material

ACKNOWLEDGMENTS

All study team members deeply thank teams and individuals who have supported the Palliative Care Learning Network through this paper and these clinical trials, particularly the Westat team, including Shannon Reefer, and the Patient-Centered Outcomes Research Institute, particularly Carly Patterson and Neeraj Arora. The authors acknowledge the patients, caregivers, and other advisors who were integral partners in the studies, as well as the core research team members and staff at all study sites, particularly (a) Executive Committee members: Victor Navarro, Andrzej Kosinski, Tamar Taddei, Michael Volk, Kavitha Ramchandran, and Marie Bakitas, and all clinical centers and Duke Clinical Research Institute’s team members participating in the PAL LIVER study. (b) Core research team members: Keith S. Goldfeld, Allison M. Cuthel, Kaitlyn Van Allen, and Mara Flannery; Site Principal Investigators: Anar Shah, Kei Ouchi, Carolyn K. Holland, Chinwe Ogedegbe, Erin Zimny, Scott Dresden, Jason Bischof, Carter Neugarten, Richelle Cooper, Christopher Coyne, and Karen Jubanyik; co-Investigators: Tiny Varghese, Susan E. Cohen, Abraham A. Brody, Brenda Matti-Orozco, Joshua Lakin, Paige Barker, Jose Contreras, David Henkin, Erin Stevens, Melanie Smith, Elena Kuzin-Palmeri, Arum Kim, Joseph Lowy, Jennifer S. Scherer, Nancy E. Bael, Jeffrey Berger, Christopher Pietras, Kira Skavinski, and Dmitry Kozhevnikov for the EMPallA study. (c) The Meta-LARC ACP trial team. (d) The UC Health Care Planning study team members: Anne Walling, Juan Carlos Antonio Lopez, Douglas Bell, Kirsten Buen, Anna DePaolis-Dickey, Ron Hays, Victor Gonzalez, Javier Sanz, Katherine Santos, Chi-hong Tseng, Rebecca Sudore, Axel Hererra, Leah Karliner, Jonathan Lee, Brookelle Li, Kanan Patel, Christine Ritchie, Andrew Robinson, Gabriela Vanegas, Aiesha Volow, Lisa Gibbs, Maryam Rahimi, Jamie Anand, Aaron Chau, Valerie George, Rick Marshall, Eileen Sabino-Laughlin, Megan Whalen, Keeta Scholl, and Judy Thomas. (e) The REACH PC study site PIs: Jennifer Temel, Joseph Greer, James Tulsky, Toby Campbell, Leslie Blackhall, Stacy Fischer, Mike Rabow, Finly Zachariah, Eytan Szmuilowicz, Amelia Cullinan, Laura Shoemaker, Laura Hanson, Maria Bakitas, Maria Silveira, Tammie Quest, Sara Martin, Kostas Leventakos, Josephine Feliciano, Carl Grey, Sean O’Mahony, Lori Spoozak, and Jessica Bauman. (f) ACP and Five Wishes facilitators, Linda Briggs, Kenisha Bethea, and Emily Malone for the EQUAL ACP study.

This work was (partially) supported through the following Patient-Centered Outcomes Research Institute (PCORI) Awards: IHS-1609-36518-IC, PLC-1609-36306, PLC-1609-36381, PLC-1609-36714, PLC-1609-35995, PLC-1609-36277, PLC-1609-36291, and PLACER-2020C3-21247. All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the view of the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors, or the Methodology Committee. This work was also supported by the Fan Fox & Leslie R. Samuels Foundation Award 108995.

A.E.-J. serves as a consultant for Incyte Corporation, GSK, and Tuesday Health. The remaining authors declare no conflict of interest.

Supplemental Digital Content is available for this article. Direct URL citations are provided in the HTML and PDF versions of this article on the journal's website, www.lww-medicalcare.com.
==== Refs
REFERENCES

1 Craig P Dieppe P Macintyre S . Developing and evaluating complex interventions: the new Medical Research Council guidance. Brit Med J. 2008;337 :a1655.18824488
2 Palmer JA Parker VA Barre LR . Understanding implementation fidelity in a pragmatic randomized clinical trial in the nursing home setting: a mixed-methods examination. Trials. 2019;20 :1–10.30606236
3 Borrelli B . The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials. J Public Health Dent. 2011;71 (s1 ):S52–S63.
4 Carroll C Patterson M Wood S . A conceptual framework for implementation fidelity. Implementation Sci. 2007;2 :40.
5 Rixon L Baron J McGale N . Methods used to address fidelity of receipt in health intervention research: a citation analysis and systematic review. BMC Health Serv Res. 2016;16 :663.27863484
6 Haynes A Brennan S Redman S . Figuring out fidelity: a worked example of the methods used to identify, critique and revise the essential elements of a contextualised intervention in health policy agencies. Implement Sci. 2016;11 :23.26912211
7 Allen JD Shelton RC Emmons KM . Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination. Dissemination and Implementation Research in Health: Translating Science to Practice 2017: 267–284.
8 Schwartz S Carpenter KM . The right answer for the wrong question: consequences of type III error for public health research. Am J Publ Health. 1999;89 :1175–1180.
9 Dobson KS Singer AR . Definitional and practical issues in the assessment of treatment integrity. Clin Psychol Sci Pract. 2005;12 :384–387.
10 Weinfurt KP Hernandez AF Coronado GD . Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory. BMC Med Res Methodol. 2017;17 :1–10.28056835
11 Ingle MP Check D Slack DH . Use of theoretical frameworks in the development and testing of palliative care interventions. J Pain Symptom Manage. 2022;63 :e271–e280.34756957
12 Ang K Hepgul N Gao W . Strategies used in improving and assessing the level of reporting of implementation fidelity in randomised controlled trials of palliative care complex interventions: a systematic review. Palliat Med. 2018;32 :500–516.28691583
13 Ginsburg LR Hoben M Easterbrook A . Examining fidelity in the INFORM trial: a complex team-based behavioral intervention. Implement Sci. 2020;15 :1–11.31900167
14 Gearing RE El-Bassel N Ghesquiere A . Major ingredients of fidelity: a review and scientific guide to improving quality of intervention research implementation. Clin Psychol Rev. 2011;31 :79–88.21130938
15 Ferrell BR Twaddle ML Melnick A . National Consensus Project Clinical Practice Guidelines for quality palliative care guidelines, 4th ed. J Palliat Med. 2018;21 :1684–1689.30179523
16 Bakitas M Lyons KD Hegel MT . Effects of a palliative care intervention on clinical outcomes in patients with advanced cancer: the Project ENABLE II randomized controlled trial. Jama. 2009;302 :741–749.19690306
17 Temel JS Greer JA Muzikansky A . Early palliative care for patients with metastatic non-small-cell lung cancer. N Engl J Med. 2010;363 :733–742.20818875
18 Aslakson RA Chandrashekaran SV Rickerson E . A multicenter, randomized controlled trial of perioperative palliative care surrounding cancer surgery for patients and their family members (PERIOP-PC). J Palliat Med. 2019;22 :44–57.31486730
19 Aslakson RA Rickerson E Fahy B . Effect of perioperative palliative care on health-related quality of life among patients undergoing surgery for cancer: a randomized clinical trial. JAMA Netw Open. 2023;6 :e2314660.37256623
20 Verma M Kosinski AS Volk ML . Introducing palliative care within the treatment of end-stage liver disease: the study protocol of a cluster randomized controlled trial. J Palliat Med. 2019;22 (s1 ):S-34.
21 DeNofrio JC Verma M Kosinski AS . Palliative care always: hepatology—virtual primary palliative care training for hepatologists. Hepatol Commun. 2022;6 :920–930.34719137
22 Grudzen CR Schmucker AM Shim DJ EMPallA Outpatient Investigators. Development of an outpatient palliative care protocol to monitor fidelity in the emergency medicine palliative care access trial. J Palliat Med. 2019;22 (s1 ):S-66.
23 Chua IS Zachariah F Dale W . Early integrated telehealth versus in-person palliative care for patients with advanced lung cancer: a study protocol. J Palliat Med. 2019;22 (s1 ):S-7.
24 LeBlanc TW Roeland EJ El-Jawahri A . Early palliative care for patients with hematologic malignancies: is it really so difficult to achieve? Curr Hematol Malign Rep. 2017;12 :300–308.
25 Sudore RL Walling AM Gibbs L . Implementation challenges for a multisite advance care planning pragmatic trial: lessons learned. J Pain Symptom Manage. 2023;66 :e265–e273.37098388
26 Hamilton AB Finley EP . Reprint of: Qualitative methods in implementation research: an introduction. Psychiatry Res. 2020;283 :112629.31735374
27 Walling AM Sudore RL Bell D . Population-based pragmatic trial of advance care planning in primary care in the University of California Health System. J Palliat Med. 2019;22 (s1 ):72–81.31486723
28 Ejem DB Barrett N Rhodes RL . Reducing disparities in the quality of palliative care for older African Americans through improved advance care planning: study design and protocol. J Palliat Med. 2019;22 (s1 ):S-90.
29 Totten AM Fagnan LJ Dorr D . Protocol for a cluster randomized trial comparing team-based to clinician-focused implementation of advance care planning in primary care. J Palliat Med. 2019;22 (s1 ):S82–S89.
30 Izumi SS Caron D Guay-Bélanger S . Development and evaluation of serious illness conversation training for interprofessional primary care teams. J Palliat Med. 2023;26 :1198–1206.37040304
31 Verma S Hingwala J Low JT . Palliative clinical trials in advanced chronic liver disease: challenges and opportunities. J Hepatol. 2023;79 :1236–1253.37419393
32 Nguyen HQ McMullen C Haupt EC . Findings and lessons learnt from early termination of a pragmatic comparative effectiveness trial of video consultations in home-based palliative care. BMJ Support Palliat Care. 2022;12 (e3 ):e432–e440.
