
==== Front
BMJ Open
BMJ Open
bmjopen
bmjopen
BMJ Open
2044-6055
BMJ Publishing Group BMA House, Tavistock Square, London, WC1H 9JR

10.1136/bmjopen-2023-082372
bmjopen-2023-082372
Protocol
Health Policy
1703
1506
Alternative payment models in Dutch hospital care: what works, how, why and under what circumstances? Protocol for a realist evaluation study
http://orcid.org/0000-0002-9401-7983
Hendriks Celine Maria Rosanne 1hendriks@eshpm.eur.nl

Vugts Miel Antonius Petrus 2miel.vugts@rivm.nl

Eijkenaar Frank 1eijkenaar@eshpm.eur.nl

Struijs Jeroen Nathan 23j.n.struijs@lumc.nl

Cattel Daniëlle 1cattel@eshpm.eur.nl

1 Erasmus School of Health Policy & Management, Erasmus University Rotterdam, Rotterdam, Netherlands
2 Department of Quality of Care and Health Economics, National Institute for Public Health and the Environment, Bilthoven, Netherlands
3 Health Campus The Hague/ Department of Public Health and Primary Care, Leiden University Medical Center, Leiden, Netherlands
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

None declared.

Ms; hendriks@eshpm.eur.nl
2024
23 9 2024
14 9 e08237221 11 2023
04 9 2024
Copyright © Author(s) (or their employer(s)) 2024. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Abstract

Introduction

The predominant provider payment models in healthcare, particularly fee-for-service, hinder the delivery of high-value care and can encourage healthcare providers to prioritise the volume of care over the value of care. To address these issues, healthcare providers, payers and policymakers are increasingly experimenting with alternative payment models (APMs), such as shared savings (SS) and bundled payment (BP). Despite a growing body of literature on APMs, there is still limited insight into what works in developing and implementing successful APMs, as well as how, why and under what circumstances. This paper presents the protocol for a study that aims to (1) identify these circumstances and reveal the underlying mechanisms through which outcomes are achieved and (2) identify transferrable lessons for successful APMs in practice.

Methods and analysis

Drawing on realist evaluation principles, this study will employ an iterative three-step approach to elicit a programme theory that describes the relationship between context, mechanisms and outcomes of APMs. The first step involves a literature review to identify the initial programme theory. The second step entails empirical testing of this theory via a multiple case study design including seven SS and BP initiatives in Dutch hospital care. We will use various qualitative and quantitative methods, including interviews with involved stakeholders, document analysis and difference-in-differences analyses. In the final step, these data and the applicable formal theories will be combined to test and refine the (I)PT and address the research objectives.

Ethics and dissemination

Ethical approval has been granted by the Research Ethics Review Committee of Erasmus School of Health Policy and Management (Project ID ETH2122-0170). Where necessary, informed consent will be obtained from study participants. Among other means, study results will be disseminated through a publicly available manual for stakeholders (eg, healthcare providers and payers), publications in peer-reviewed scientific journals and (inter)national conference presentations.

Health Services
Health policy
Organisation of health services
Hospitals
Implementation Science
Netherlands Organization for Health Research and Development (ZonMw) 516008005
==== Body
pmcStrengths and limitations of this study

This study employs a rigorous realist evaluation approach to elicit, test and refine an (initial) programme theory explaining how alternative payment models (APMs) for healthcare providers can be successfully developed and implemented to achieve their intended outcomes under varying circumstances.

The involvement of relevant stakeholders in the design of the study provides confidence that the evaluation will generate meaningful findings for policy and practice, guiding effective payment reforms that could ultimately enhance healthcare value.

This study uses a mixed methods approach to enhance the validity of the findings and to foster the generation of in-depth insights.

The seven APM initiatives studied vary greatly in key contextual and outcome dimensions, including the medical conditions covered, enabling a comprehensive testing of the programme theory under development.

The focus on condition-specific bundled payment and shared savings models in the Dutch healthcare system may limit the generalisability of our findings to other APM types and other contexts.

Introduction

There is a growing consensus that the predominant provider payment models, particularly fee-for-service (FFS), contribute to healthcare systems performing suboptimally.13 Under FFS, healthcare providers are compensated for each service rendered, regardless of the quality of care provided.48 This payment model undermines healthcare system performance in two ways. First, FFS does not reward (and may even hinder) the delivery of high-value care.6 9 Because provider revenue depends on the number of services provided, providers that reduce unnecessary services, prevent illness and/or achieve lower complication rates are penalised in financial terms.158 10 In addition, FFS does not financially reward clinical excellence and collaboration between providers.138 11 Second, and relatedly, FFS may create perverse incentives that encourage providers to prioritise the volume of care over the value of care.12 69

Therefore, healthcare providers, payers and policymakers are experimenting with alternative payment models (APMs) that enable and incentivise providers to increase value by linking reimbursement to higher quality and/or lower medical spending.1 2 6 12 The introduction of APMs may be an effective strategy for improving value, as providers are often in the best position to identify potential areas for improvement.13 APMs can vary in design, depending on the specific aims. This paper focuses on APMs targeting specific medical conditions, which can be designed as either a shared savings (SS) or a bundled payment (BP) model. Under an SS model, payers retain the FFS architecture and continue to reimburse providers on an FFS basis. Afterward, total FFS spending is compared with a prospective spending benchmark. If spending is below the benchmark, providers receive a share of the savings. Providers may also bear a share of the losses if spending exceeds the benchmark. Providers’ share of savings/losses may depend on their performance on predefined quality measures.14 15 Under a BP model, the FFS architecture is abandoned, and a provider-led entity receives a fixed, predetermined payment intended to cover the expected costs of all the services associated with a clinically defined episode of care.16 17 A key difference with an SS model lies in the providers’ full financial responsibility for all spending covered by the BP model, while under an SS model this responsibility is only partial. Both models enable and incentivise providers to act cost-consciously, to coordinate care across the episode and to ensure high quality.18

SS and BP models are widely scrutinised and implemented in numerous countries, including the USA, the UK and the Netherlands.18 19 With the growing prevalence of these APMs, a growing body of knowledge has emerged regarding their empirical effects on care quality, utilisation and medical spending. Current evidence suggests that both SS and BP models can reduce spending growth, while improving or maintaining quality. However, the effects vary in size, and between specific APM designs and settings.1318 2022 An important limitation of this evidence is that studies have mainly focused on APMs within the US healthcare system.18 23 In addition, although an increasing number of studies provide insights into whether specific payment models ‘work’ in terms of reducing spending growth and/or improving quality,1820 21 2325 they often add little information on how, why and under what circumstances these models work. More research on the role of context is essential to advance our understanding of how causal influences are sensitive to varying circumstances.26

In addition to assessing quantitative effects, researchers have focused on analysing the complex process of developing and implementing APMs that may potentially hinder practical adoption. Efforts have been made to document the contextual factors that shape this process.9 27 28 Despite some valuable insights, these studies have not substantially contributed to our understanding of causal pathways, including the underlying generative mechanisms. For example, though divergent individual interests and goals within provider organisations are commonly cited as potential barriers to transitioning payment incentives from volume to value,9 27 it remains largely unclear which specific thought processes and interactions among stakeholders are triggered or blocked by these factors. A better understanding of these underlying processes may inform those involved in payment reforms, helping them reason and interact effectively to increase the likelihood of success. Such insights are crucial for APMs to reach their full potential of improving healthcare value.

This paper presents a protocol of a realist evaluation (RE) study of condition-specific SS and BP models, including case studies of seven initiatives in the setting of hospital care in the Netherlands (see Methods and analysis). The study aims to enhance our understanding of ‘what works’ in developing and implementing successful APMs, as well as how, why and under what circumstances they work. To this end, the study will draw on RE principles and elicit context-mechanism-outcome (CMO) configurations.26 Our primary objective is to develop a programme theory describing the circumstances under which condition-specific SS and BP models can be successfully developed and implemented, uncovering the mechanisms triggered or blocked by these circumstances. The secondary objective is to identify transferrable lessons for successful SS and BP models in practice.

The study is expected to generate valuable insights for providers, payers, policymakers and other stakeholders that may support the development and implementation of successful SS and BP models, ultimately contributing to improved healthcare value. Additionally, among those evaluating provider payment models, this study may lead to greater awareness of the context in which these models are developed and implemented, emphasising the need for comprehensive evaluation of this context to take account of the relevant causal pathways.

Methods and analysis

Realist evaluation

This study will apply an RE approach, which is a theory-driven evaluation approach well-suited for evaluating complex interventions.262931 RE seeks to provide an in-depth understanding of what works, for whom, under what circumstances, how and why? Rooted in scientific realism, this approach recognises the potential impact of unobservable factors like culture and institutions on intervention outcomes and acknowledges that individuals respond differently to interventions in varying contexts.26 31 32 To understand how and why interventions yield different outcomes in different populations and settings, RE examines causal pathways including underlying generative mechanisms.26 31

In RE, interventions are presumed to be underpinned by a programme theory that explains how change is enacted and outcomes are produced. This theory is explicated by developing CMO configurations, which describe how a contextual factor (C) activates or blocks a certain mechanism (M), resulting in a certain outcome (O). These configurations are the main structure of analysis in RE and can be elicited, tested and refined through an iterative research process.26 31 Table 1 summarises the RE concepts used in our study.

Table 1 Definitions of the different realist evaluation concepts applied in this study

Concept (references)	Definition	
Context9 56 59 60	Context refers to a set of pre-existing circumstances that trigger or block mechanisms when an intervention is introduced (in this study, the intervention refers to shared savings and bundled payment models for healthcare providers). The circumstances can be external or internal to organisations involved in the intervention and may include sociocultural, historical, economic, political and/or relational factors. In addition, the characteristics of actors involved in the intervention (ie, provider and payer organisations or their representatives) are part of the context. The design features of the intervention (such as the care services included and the contract duration) also fall under context. Finally, reform strategies are also considered context. Reform strategies include intended changes in resource allocation undertaken by the organisations involved in the payment reform with the aim of facilitating the development and implementation of the intervention and/or to make it work as intended. Examples of reform strategies are third-party mediation and investing in data infrastructure.	
Mechanisms53 60	Mechanisms are the generative forces that lead to outcomes. They describe the psychological, economic or social explanations of the behaviour of (groups of) individuals involved in the intervention and define how change is enacted (eg, increased trust due to more frequent interactions).	
Outcomes9 61 62	In this study, we distinguish between two interdependent outcome domains that together define the outcomes being evaluated.The first outcome domain refers to different stages in the process of payment reform and the extent to which these are successful. Successful refers to whether efforts undertaken by the stakeholders increase the probability of stakeholders reaching the next stage of payment reform and/or the intervention achieving the intended effects. We discern two main stages of payment reform: (1) development and (2) implementation. The development stage includes the process of stakeholders shaping the intervention and ultimately the signing of a payment contract. By the extent to which actor involvement in the development is absent or passive, this stage may also be seen as a process of becoming aware of the new intervention, forming an attitude and adopting it (and followed by the signing of a payment contract). The subsequent implementation stage includes the active and planned efforts undertaken by providers and payers to mainstream the intervention within their organisation. This involves a transition period during which those involved become increasingly skilled in and committed to the intervention. The implementation stage may be followed by a third stage: the continuation stage. In this stage, the providers and payers involved agree to extend the intervention beyond the initially agreed contract period. The intervention may be extended in its original form or changes may be made based on the lessons learnt. In this study, we consider the continuation stage part of implementation.The second outcome domain includes the (un)intended effects of the intervention on the quality of care, medical spending and/or the utilisation of care.	
CMO configurations26 31 60	CMO configurations are propositions describing the relationship between context, mechanisms and outcomes. In this study, CMO configurations describe the contexts that trigger or block mechanisms leading to certain outcomes.	
CMOcontext-mechanism-outcome

Study design

Following the framework for RE described by Pawson and Tilley,26 this study will be conducted in three steps (figure 1): (1) elicit the initial programme theory (IPT), (2) test the (initial) programme theory ((I)PT) using empirical data and (3) refine the (initial) programme theory. These steps are presented here as being sequential, while in practice PT development is iterative, informed by theory and interim findings to allow the gathering of additional data on emerging themes.33 RE is a method-neutral approach, resulting in variations in design and research methods applied in previous studies.29 31 34 35 Our study will use a literature review to elicit the IPT and a multiple case study to test and refine the (I)PT, followed by a synthesis of findings into a final PT.36 We will investigate seven SS and BP initiatives in Dutch hospital care using a combination of qualitative and quantitative methods. The study will be conducted in accordance with the RAMESES (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) II reporting standards for REs.31 37 Data collection for this study is expected to be completed by September 2024.

Figure 1 Realist evaluation research cycle with a three-step approach. Adapted from Pawson and Tilley.26

Step 1: elicit the initial programme theory

In step 1, we will develop the IPT that clarifies under what circumstances, how, and why SS and BP models can be successfully developed and implemented and contribute to the effects observed. This IPT will include different CMO configurations and will form the starting point for our empirical research.

We will build the IPT based on literature that is relevant to our research questions. This includes literature that provides an insight into any element of the IPT. Following RE principles,26 31 37 38 literature from a wide range of sources, including both empirical and conceptual and scientific and grey literature, will be searched and considered. Relevant literature may include systematic reviews on APMs and conceptual papers about the development and/or implementation of APMs. Additionally, relevant conceptual work on applicable (grand) theoretical approaches from various social science disciplines will be searched and studied. These approaches are particularly useful for understanding the underlying mechanisms driving APM outcomes and for explaining the observed patterns of contexts, mechanisms and outcomes at a more general level.38 This may include, among others, contract theory, behavioural economics, institutional theory and the theory of planned behaviour.

We will start by drawing up a list of potentially relevant documents known to our interdisciplinary research team. This will include scientific papers from the APM literature (not limited to SS or BP models) that were previously identified through systematic searches and research conducted by our team (eg,1819 23 25 27 3941). Furthermore, we will search Google Scholar to identify studies that have applied social science theories to study the development and implementation of APMs, as well as provider payment reform more broadly. To achieve this, we will create multiple search strings incorporating combinations of keywords related to health, payment and a social science theory (eg, theory of planned behaviour). The first 10 pages per search string will be screened by title and abstract for potential relevance. Documents will be included for analysis based on full-text assessment if they contribute to theory building, which is the case if they shed light on the relationships between context, mechanisms and/or outcomes within the IPT. Our experience with pilot searches and realist research indicates that the most relevant and contributory studies typically appear within the first 5–10 pages of search results. However, if important concepts or aspects have not yet reached saturation,42 we will extend our search beyond the initial 10 pages and/or adapt our search strategy to better address the missing data, as prescribed by good realist research practices.42 Literature searches are likely to be iterative because, as the formation of the IPT progresses, specific elements of the IPT may require further exploration of the literature, focusing on particular concepts or aspects in more detail.38 42 Searches will continue until the IPT is saturated, meaning that the list of CMO configurations is exhaustive and each element is thoroughly described.38 42

For each document identified, data indicative of a causal path will be extracted and analysed using a standardised extraction form (more details are provided in online supplemental appendix 1.1). Data extraction will be done by author pairs, with one author taking the lead to elicit CMO configurations and another checking for missing information and inconsistencies. Any discrepancies will be resolved through discussion. The IPT will be developed and refined in weekly meetings with our research team. Additionally, we will seek opinions from at least three external experts on provider payment reform to assess the list of included articles and the IPT, identifying any missing documents or elements.

At the time of writing, data collection for the IPT has started. Initial collection and searches have identified over 60 relevant documents yielding a list of over 400 data rows. These rows contain background information on the study type (including the APM type that the findings apply to, where applicable), rigour appraisals and findings with respect to context, mechanisms and outcomes. Initial steps have been undertaken to condense this information, involving the clustering of rows that exhibit substantial regularities in content, with a primary focus on assessing the similarity of contextual factors.43 This process will ultimately result in a set of CMO configurations of a middle-range level of abstraction.43 One example of a preliminary CMO configuration that has resulted from this process is: if providers participate actively in developing the APM, development and implementation are more likely to be successful because the design of the payment model is expected to be better aligned with providers’ professional values and intrinsic motivation.

Step 2: test the (initial) programme theory

In step 2, we will gather data to test the (I)PT using a multiple case study design. This design is well-suited for studying phenomena in their real-life context36 and is common practice in RE.34 44 45

Case selection

At the time of writing, case selection has been completed. We have included seven cases (see table 2), all of which are SS or BP initiatives in the Dutch hospital care setting. This selection is the result of a combination of convenience and purposive sampling.46 The criteria for case selection included: (1) stakeholders that were willing to cooperate in data collection (eg, representatives available for interviewing); (2) availability of data to assess the effects on quality, care utilisation and medical spending; and/or (3) variation on important elements of the IPT. The third criterion was deemed necessary to enable the IPT to be comprehensively tested.31 37 The cases vary on several context and outcome dimensions that are relevant to the interventions of interest, such as the condition or care covered by the SS or BP model, and the stage of payment reform (eg, still in the development stage or was implemented several years ago).

Table 2 General characteristics of selected study cases

	Condition or care type	Implementation year(s)	APM type	Included care services	Contract specifications	Number of insurers involved	Number and characteristics of providers involved	
1	Total joint arthroplasty	2018	Shared savings and losses model	All services related to the diagnosis, surgery and treatment of associated complications up to 1 year after surgery.	Retrospective annual reconciliation.

Costs associated with complications above a certain threshold are excluded.

No cap on volume of care if quality score meets certain criteria.

Multiyear contract.

	1	10 general hospitals, 4 specialised clinics and 6 top clinical hospitals located in urban and rural areas.	
2	Cataract	2018	Shared savings and losses model	All services related to the diagnosis, surgery and treatment of associated complications up to 2 years after surgery.	Retrospective annual reconciliation.

No cap on volume of care if quality score meets certain criteria.

Multiyear contract.

	1	1 general hospital, 3 specialised clinics and 4 top clinical hospitals located in urban and rural areas.	
3	Breast cancer	2019	Shared savings and losses model	All services related to the surgery (ie, lumpectomy or mastectomy) and treatment of associated complications up to 2 years after surgery. Services related to the provision of chemotherapy, radiotherapy and/or hormone therapy are not included.	Retrospective annual reconciliation.

Minimum savings and losses rate that must be met to qualify for shared savings or prevent shared losses.

No cap on volume of care if quality score meets certain criteria.

Multiyear contract.

	1	7 general hospitals and 2 top clinical hospitals located in urban and rural areas.	
4	Rheumatoid arthritis	2019	Shared savings model	All expensive medicines (ie, biological and targeted synthetic disease-modifying anti-rheumatic drugs) used in the treatment of rheumatoid arthritis over a period of 1 year.	Retrospective annual reconciliation.

Provider shares in savings if quality score meets certain criteria.

Multiyear contract.

	1	4 general hospitals, 1 specialised clinic and 2 top clinical hospitals located in urban and rural areas.	
5	Heart failure	2022	Shared savings model*	All services related to the diagnosis and treatment over a period of 1 year.	Retrospective annual reconciliation.

Costs above a certain threshold are truncated.

Minimum savings rate that must be met to qualify for shared savings.

Multiyear contract.

	1	1 top clinical hospital located in an urban area.	
6	Ischaemic stroke	2019	Shared savings and losses model	All services related to the diagnosis, treatment and rehabilitation of ischaemic stroke.	Retrospective annual reconciliation.

Gradual introduction of shared losses after the first year of implementation.

Shared savings and/or losses conditional on quality score and distributed among the providers involved based on a predefined distribution key.

Multiyear contract.

	1	1 academic hospital and 3 rehabilitation centres located in an urban area.	
7	Maternity care	2017 (n=6), 2019 (n=1) and 2021 (n=2)	Bundled payment	Bundle defined in four phases (ie, prenatal, natal, postnatal and maternity care assistance), covering all necessary care services in each phase according to the national guidelines.	Prospectively defined bundled fees for the defined modules within the different phases.

A provider-led entity (ie, integrated maternity care organisation) benefits from all savings and are liable for all losses.

No link between bundled fees and quality measures, although annual accountability reports regarding quality measures are part of the contract.

Some are annual contracts, while others are multiyear contracts.

	Multiple	9 integrated maternity care organisations, consisting of community midwife practices, 6 general hospitals, 3 top clinical hospitals and postpartum maternity care assistance organisations located in urban and rural areas.	
* Strictly speaking, this model entails a global payment for this specific condition with shared savings.

APMalternative payment model

Data collection and analysis

For each case, multiple data sources will be used and analysed. Quantitative data will be used to assess the effects on quality, utilisation and medical spending. Qualitative data will be used to understand how these effects occurred and ‘what works’ in developing and implementing successful SS and BP models, and how, why and under what circumstances these models work.

Qualitative methods

Qualitative data collection

Qualitative data collection will include document analysis and interviews. First, for each case, we will obtain the payment contract and other available documentation, like meeting minutes and reports substantiating the choices made during development. These documents will be searched for information on the SS or BP model itself (eg, contract terms, payment model specifications, reform strategies and timelines) as well as on the intended outcomes (eg, aims of the payment reform and outcome measures used).

Second, we will conduct semi-structured interviews with representatives from purposively sampled stakeholders involved in each case (henceforth referred to as respondents). Interviews are a suitable and widely used method for data collection to test and refine a realist PT.47 For each case, we will strive to maximise the variation observed in the context and the outcomes between sampled providers and insurers. We aim to interview respondents in various roles within the organisations sampled, including executive managers, medical specialists, healthcare purchasers and sellers and business controllers. Additionally, although patient involvement in the initiatives is often limited, we will interview representatives from patient organisations to learn about their potential roles in developing and implementing APMs. Finally, respondents working for providers who ultimately decided not to participate in the studied SS and BP initiatives will be interviewed to understand their reasons. Interview invitations will be sent via email by our research team. We aim to conduct approximately 65 interviews in total.

An interview guide will be developed, informed by the IPT and document data. This guide will cover three main domains: (1) development, (2) implementation and (3) (un)intended effects and will be tailored to the specific stakeholder group (ie, insurer, provider or patient organisation). The interview guides will be pilot-tested with stakeholders. During the interviews, a ‘realist interviewing style’ will be adopted,47 using general phrases like ‘what external factors’ and ‘what thoughts were on your mind?’ to inquire about context and mechanisms, respectively. We will also actively explain the concepts of interest to respondents to ensure a shared understanding of the terminology and purpose of the questions.34 47 Moreover, we will present parts of the IPT to jointly understand what happened throughout the stages of payment reform, allowing us to refine elements of the (I)PT.47 The interview guide may be modified during our RE based on emerging insights gained in previous interviews that were initially overlooked or unknown during the initial development of the guide.

We will conduct two rounds of interviews with different respondents in each round. After both rounds, we will refine the (I)PT as outlined in step 3. The first interview round will help us explore the richness of case data regarding various IPT elements and identify any potentially missing elements. Subsequently, the second round can focus on further developing emergent findings and testing and refining key elements.31 37

Before each interview, the respondent(s) will be informed of the study’s objectives and written informed consent will be obtained. Interviews will be approximately 60 min in duration, conducted in author pairs and will be audio-recorded and verbatim transcribed.

Qualitative data analysis

All documents and interview transcripts will be coded using Atlas.ti (V.9). According to the recently published principles for analysing qualitative data transparently within RE,48 a structure of codes will be created, to which theoretical memos about PT development will be attached to. Each CMO configuration in our IPT will receive a title and a corresponding code in Atlas.ti. Each code will link to a theoretical memo containing the CMO configuration as formulated in the IPT.48 If new information arises during coding that is not covered by existing codes, a new code will be created and linked to a new memo.

Coding will involve selecting relevant text fragments in the documents and transcripts and assigning them to applicable codes. Four members of the research team will conduct this coding, in pairs. To ensure consistency during coding, the four members will independently code several documents and transcripts, after which they will compare and discuss codes until coding practices are aligned. Next, the remaining documents and transcripts will be allocated to the two pairs for independent coding. The pairs will meet regularly to review and maintain coding consistency.

Table 3 illustrates a potential data structure resulting from this coding process.

Table 3 Example data structure resulting from the coding procedure

Code (in Atlas.ti)	Data source	Theoretical memo	
Co-creation	Interview 2	IPTIf providers participate in developing the SS or BP model (C), development and implementation are more likely to be successful (O) because the design of the payment model is expected to be better aligned with providers’ values and intrinsic motivation (M).NotesMay 24: Another mechanism identified: Increased ownership of problems and solutions among different stakeholders. Search for additional literature required.June 10: ‘Engagement’ might be a better term/code than participation.	
Interview 6	
Document 3	
BPbundled paymentIPTinitial programme theorySSshared savings

Quantitative methods

Quantitative data collection

Administrative data from participating insurers and providers will be used to assess the effects of the seven SS and BP initiatives on care quality, utilisation and medical spending. These data will include information on patient characteristics (eg, age, sex, comorbidities) and insurance claims (to be used for defining medical spending, utilisation and quality measures) for patients treated for the relevant condition and enrolled with the relevant insurer. We will collect data for periods both before and after implementing the SS and BP model and for patients treated by participating providers (the intervention group) and non-participating providers (the control group).

Quantitative data analysis

We will assess the effectiveness of the seven APM initiatives using a difference-in-differences (DiD) design,49 which is a quasi-experimental approach commonly used for impact evaluations of complex health interventions. We will use regression modelling to analyse trends in outcomes over time for patients treated by participating and non-participating providers. This approach enables researchers to account for trends in the outcomes that are unrelated to the introduction of the intervention, isolating the causal effect on outcomes.49 We will analyse the effects on various quality measures (eg, complication and readmission rates), care utilisation (eg, diagnostic tests and length of stay) and medical spending (in total and in subcategories), adjusting for provider fixed effects and patient characteristics.50

DiD analyses rely on the parallel trends assumption (PTA),49 which asserts that the trends in outcomes of the intervention and control group are similar before the intervention. If this is true, it is reasonable to assume that these trends would continue without the intervention, making the control group’s trend a valid counterfactual for the intervention group’s trend in the absence of the intervention. The PTA will be assessed by visually inspecting trends and conducting falsification tests for all outcomes in each case.49 To test the robustness of our results, we will perform sensitivity analyses, including using alternative specifications of control groups and outcomes (eg, yes/no truncation of spending variables).

Step 3: refine the (initial) programme theory

Bi-monthly team meetings will be organised to refine the preliminary CMO configurations through discussion. We will use coded text fragments as the primary inputs for refining the (I)PT. Theoretical memos will be used to capture all the refinements made to CMO configurations based on coded data. These memos will provide a detailed rationale for all refinements, explicitly capturing the full team’s reasoning in developing the PT. The (I)PT will initially be refined after coding the first round of interviews. We will adjust existing codes based on the preliminary findings and create new codes (and linked theoretical memos) if necessary. These adjusted codes and memos will be used for the second round of interviews, for which the same coding and refinement procedure will be followed. Additionally, we will (re)examine the relevant formal theories during theory refinement to better understand the observed CMO configurations and enrich the causal path descriptions. For example, when respondents mention ‘trust’ repeatedly as a trigger for certain behaviour, formal definitions and taxonomies of ‘trust’ will be explored from theoretical sources. In addition to the qualitative data, quantitative data will be used to refine the (I)PT. Various models for combining these data exist, which vary with respect to, for example, the timing of data collection (ie, concurrently or sequentially) and the approaches used for combining the data (eg, mixing results during interpretation or analysis).51 We will explore different models for mixing findings. Based on the research objectives, the timing of data collection and analyses and the resources available across cases, the appropriate model(s) will be selected (more details are provided in online supplemental appendix 1.2). Lastly, refined CMO configurations will be discussed with (groups of) experts in the field of APMs and related areas (eg, quality measurement, competition, shared decision-making and organisation studies). Experts will be asked to identify missing or inadequately refined elements in the PT. Approximately 10–15 experts will be consulted. Based on their inputs, coded data and formal theories will be reevaluated and additional literature will be explored to refine the PT.

Patient and public involvement

The input of representatives of various stakeholder organisations, including healthcare providers, health insurers, patient organisations and policymakers, has been collected and integrated into the design of this protocol. For example, the research questions and outcomes of interest have been collaboratively formulated with these representatives, drawing on their experience in developing and implementing the initiatives of interest to this study. These stakeholders will continue to be involved in the research, for instance in terms of contributing to the data collection and interpretation of results.

Ethics and dissemination

Ethics

This study will be conducted in accordance with the existing guidelines on good research practices and integrity (Netherlands Code of Conduct for Research Integrity, 2018).52 The study has received approval from the Research Ethics Review Committee of Erasmus School of Health Policy and Management (registration number ETH2122-0170; 14 December 2021) and data providers. A data management plan, overseen by a data steward from Erasmus University Rotterdam, ensures secure data collection and storage. Informed consent will be obtained for interviews, with alternative data collection methods considered if consent is declined. Any unnecessary collection of personal data will be minimised, individual-level data will be pseudonymised for analysis and any personally identifiable information will be removed before reporting study findings.

Dissemination

To ensure the adoption of our findings in practice, results will be disseminated through a hands-on manual for policy and practice describing the building blocks of SS and BP models, the steps required and the conditions for their successful development and implementation. This manual will be publicly accessible and distributed widely to relevant stakeholders, for example, providers, payers, patient associations and policymakers. Our findings will be presented at various (inter)national conferences and published in peer-reviewed open-access journals. We will present our (preliminary) findings, among other venues, at an annual conference organised by our research group, which will be accessible to all stakeholders. To reach a broader audience, we will produce three explanatory videos and an infographic elucidating our findings. These materials will be available on a dedicated website. Finally, findings will be integrated into undergraduate, graduate, postgraduate, doctoral and online education programmes provided by our institutions.

Discussion

This paper outlines the protocol for an RE based on a literature review and seven case studies of APMs in practice.

Traditionally, evaluations of complex (policy) interventions in healthcare have focused mainly on quantifying the causal effects on outcomes using (quasi-)experimental research designs. Although these designs are suitable for assessing the effectiveness of interventions, scholars increasingly emphasise the complex and dynamic nature of health systems and the healthcare organisations that shape intervention effects, which cannot be fully captured by these research designs. This limits drawing context-sensitive conclusions,29 34 53 which is an important limitation because effective, scalable and sustainable reform requires a deeper understanding of the causal mechanisms that underly (changes in) outcomes and their interaction with context. An increasingly applied approach in recent years for evaluating health policy interventions that aims to meet these requirements is RE.26 29 31 34 35 45 53 54 To our knowledge, our study is the first to use an RE approach to evaluate APMs in healthcare.

Our study aims to develop a broadly applicable PT providing insights into under what circumstances, how and why APMs for providers can be developed and implemented successfully. This PT is expected to offer valuable lessons for healthcare providers, payers, policymakers and researchers who are interested in (future) payment reforms and methods and strategies that contribute to the sustainability of healthcare systems. In particular, the PT will provide in-depth insights into the mechanisms driving (un)successful APM development and implementation, and the contextual circumstances that trigger or block these mechanisms. Regardless of our focus on condition-specific APMs, we expect our theory to apply to APMs beyond SS and BP models. For example, a CMO configuration explaining the prerequisites for the successful incorporation of quality measures into SS and BP models will be relevant to all APMs—in particular pay-for-performance schemes—in which the reimbursement of providers is linked explicitly to quality. The same applies to CMO configurations covering more general themes, such as how the degree of mutual trust influences collaboration. Finally, we expect our approach and findings to provide guidance to others evaluating APMs.

RE poses numerous challenges for researchers,29 53 55 three of which have already been encountered while designing this study and collecting data for the IPT. First, identifying generative mechanisms in the existing literature on APMs has been difficult because mechanisms are often not (explicitly) described. In addition, the description of mechanisms is complicated by a lack of shared conceptions among scholars regarding what exactly constitutes a mechanism, which means that various definitions are used.53 Similarly, differentiating between mechanisms and context can be difficult because some phenomena serve as both. For example, (emergence of) trust can act as a mechanism in explaining stakeholder responses in collaborative activities or as a contextual factor when considering the preconditions for initiating these activities.56 To address these issues, we will explore literature beyond APMs (eg, from the field of behavioural economics and organisation studies) and prioritise the uncovering of mechanisms during the interviews.

A second challenge is that some of our cases involve APM models that were introduced years ago and have since then undergone changes. For example, the objectives and focus of the SS model for knee and hip replacement surgery and cataract surgery have been significantly revised. Furthermore, within cases the design of the payment contracts varies between providers. Documenting these changes and variations and understanding their impact on the outcomes studied will be challenging but essential. Therefore, we will pay close attention to contractual details and changes in the interventions studied over time and maintain close contact with the providers and insurers that have been involved in these changes.

A final challenge relates to RE being a resource-intensive approach that demands substantial methodological and content expertise from the researchers involved.53 55 57 These issues have been considered when estimating the capacity required to carry out this study and the recruitment of personnel.

To enhance the rigour of this study, the RE quality standards for realist synthesis and realist evaluation (eg, RAMESES II) will be followed during data collection, analyses and reporting.31 37 38 We will develop topic lists and coding schemes to facilitate and standardise data collection and analyses. During and after interviews, team members will verify the respondents’ responses for clarity, and if needed, schedule follow-up interviews. Quantitative data analysis will include sensitivity checks to confirm the robustness of DiD estimates against analytical choices made. We expect that leveraging various data sources, theories and methods will enhance the validity of our findings and the same applies for involving researchers with diverse backgrounds and perspectives (ie, medical, health economics, health policy, public health and organisational sciences) in the study. Lastly, refined CMO configurations will be discussed with experts from various fields, including APMs.

Although we anticipate that our findings will offer generalisable insights for the development and implementation of SS and BP models, as well as APMs more broadly, we acknowledge that the generalisability of our findings may be limited by the fact that all initiatives were developed and implemented within the Dutch context and primarily pertain to hospital care services. For instance, findings may differ between countries due to variations in payer type (ie, public vs private), cultural factors and providers’ experience with bearing financial accountability.58 A second limitation stems from the potential scope of effects associated with the implementation of APMs, which may extend beyond what our data sources or analytical focus can (fully) capture. Examples of such effects include provider cost-shifting (ie, the shift to services and care settings outside the bundle, as a strategy to limit spending),24 as well as broader implications such as possible changes in patient cost-sharing schemes resulting from the implementation of APMs.1 It is important to bear this in mind when interpreting our findings.

In summary, despite a growing body of literature on APMs in healthcare, important questions remain with regard to under what circumstances, how and why these models can improve value in healthcare. Our study seeks to develop a broadly applicable PT that provides answers to these questions. This PT is expected to offer valuable lessons for healthcare providers, payers, policymakers and researchers interested in payment reforms, as well as methods and strategies that could improve the sustainability of healthcare systems.

supplementary material

10.1136/bmjopen-2023-082372 online supplemental file 1

Acknowledgements

We are grateful to the Netherlands Organization for Health Research and Development (ZonMw) for their funding. In addition, we acknowledge those representatives of healthcare providers, health insurers, in particular Menzis Health Insurance Company, and patient organisations that commented on drafts of the funding proposal and made valuable suggestions for improving our study design. We thank Anna Volkova for her contributions in drafting the data management plan. We also want to thank Chandeni Gajadien, Peter Dohmen and Nèwel Salet for contributing to the data collection for the initial programme theory thus far.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.

Review Process File
23 09 2024

Funding: This study was funded by a research grant from the Netherlands Organization for Health Research and Development (ZonMw) under grant number 516008005. The funding organisation was not involved in the study design, nor will it be involved in data collection, analyses, interpretation or manuscript writing.

Prepub: Prepublication history and additional supplemental material for this paper are available online. To view these files, please visit the journal online (https://doi.org/10.1136/bmjopen-2023-082372).

Provenance and peer review: Not commissioned; externally peer reviewed.

Patient consent for publication: Not applicable.

Patient and public involvement: Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.
==== Refs
References

1 HCPLAN Alternative Payment Model (APM) Framework Available https://hcp-lan.org/workproducts/apm-refresh-whitepaper-final.pdf Accessed 8-Dec-2022
2 World Economic Forum The Moment of Truth for Healthcare Spending: How Payment Models Can Transform Healthcare Systems Available https://www3.weforum.org/docs/WEF_The_Moment_of_Truth_for_Healthcare_Spending_2023.pdf Accessed 2-Apr-2024
3 OECD Health Policy Studies Better Ways to Pay for Health Care Available https://www.oecd.org/publications/better-ways-to-pay-for-health-care-9789264258211-en.htm Accessed 2-Apr-2024
4 Frakt AB Mayes R Beyond Capitation: How New Payment Experiments Seek To Find The ‘Sweet Spot’ In Amount Of Risk Providers And Payers Bear Health Aff (Millwood) 2012 31 1951 8 10.1377/hlthaff.2012.0344 22949443
5 McGuire TG The Oxford Handbook of Health Economics 2011
6 Miller HD From Volume To Value: Better Ways To Pay For Health Care Health Aff (Millwood) 2009 28 1418 28 10.1377/hlthaff.28.5.1418 19738259
7 Jegers M Kesteloot K De Graeve D et al A typology for provider payment systems in health care Health Policy 2002 60 255 73 10.1016/S0168-8510(01)00216-0 11965334
8 Ellis RP Martins B Miller MM Provider payment methods and incentives Heggenhougen HK Quah S International encyclopedia of public health 2017
9 Conrad DA Vaughn M Grembowski D et al Implementing Value-Based Payment Reform: A Conceptual Framework and Case Examples Med Care Res Rev 2016 73 437 57 10.1177/1077558715615774 26545852
10 McGuire TG Physician agency Culyer A Newhouse J Handbook of Health Economics Amsterdam Elsevier Science 2000 461 536
11 Robinson JC Theory and practice in the design of physician payment incentives Milbank Q 2001 79 149 77 10.1111/1468-0009.00202 11439463
12 Porter ME Kaplan RS How to Pay for Health Care Harv Bus Rev 2016 94 88 98 27526565
13 Baicker K Chernew ME Alternative Alternative Payment Models JAMA Intern Med 2017 177 222 3 10.1001/jamainternmed.2016.8280 28055046
14 NEJM Catalyst What are bundled payments? Available https://catalyst.nejm.org/doi/full/10.1056/CAT.18.0247 Accessed 8-Dec-2022
15 Bailit M Hughes C Key design elements of shared-savings payment arrangements Issue Brief (Commonw Fund) 2011 20 1 16 21847849
16 Ridgely MS de Vries D Bozic KJ et al Bundled Payment Fails To Gain A Foothold In California: The Experience Of The IHA Bundled Payment Demonstration Health Aff (Millwood) 2014 33 1345 52 10.1377/hlthaff.2014.0114 25092835
17 Blumenthal D Squires D The Promise and Pitfalls of Bundled Payments. Commonwealth Fund Available https://www.commonwealthfund.org/blog/2016/promise-and-pitfalls-bundled-payments Accessed 13-Dec-2022
18 Struijs J de Vries E Baan C et al Bundled-Payment Models Around the World: How They Work and What Their Impact Has Been. The Commonwealth Fund Available https://www.commonwealthfund.org/sites/default/files/2020-04/Struijs_bundled_payment_models_around_world_ib.pdf Accessed 8-Dec-2022
19 Cattel D Eijkenaar F Ahaus K et al Bundelbekostiging in de zorg is mogelijk, ondanks belemmeringen ESB 2021 106 86 9
20 Reindersma T Sülz S Ahaus K et al The Effect of Network-Level Payment Models on Care Network Performance: A Scoping Review of the Empirical Literature Int J Integr Care 2022 22 3 10.5334/ijic.6002
21 Agarwal R Liao JM Gupta A et al The Impact Of Bundled Payment On Health Care Spending, Utilization, And Quality: A Systematic Review Health Aff (Millwood) 2020 39 50 7 10.1377/hlthaff.2019.00784 31905061
22 Milad MA Murray RC Navathe AS et al Value-Based Payment Models In The Commercial Insurance Sector: A Systematic Review Health Aff (Millwood) 2022 41 540 8 10.1377/hlthaff.2021.01020 35377757
23 Cattel D Eijkenaar F Value-Based Provider Payment Initiatives Combining Global Payments With Explicit Quality Incentives: A Systematic Review Med Care Res Rev 2020 77 511 37 10.1177/1077558719856775 31216945
24 Yee CA Pizer SD Frakt A Medicare’s Bundled Payment Initiatives for Hospital-Initiated Episodes: Evidence and Evolution Milbank Q 2020 98 908 74 10.1111/1468-0009.12465 32820837
25 Eijkenaar F Emmert M Scheppach M et al Effects of pay for performance in health care: a systematic review of systematic reviews Health Policy 2013 110 115 30 10.1016/j.healthpol.2013.01.008 23380190
26 Pawson R Tilley N Realistic Evaluation SAGE Publications 1997
27 Steenhuis S Struijs J Koolman X et al Unraveling the Complexity in the Design and Implementation of Bundled Payments: A Scoping Review of Key Elements From a Payer’s Perspective Milbank Q 2020 98 197 222 10.1111/1468-0009.12438 31909852
28 Conrad DA Grembowski D Hernandez SE et al Emerging lessons from regional and state innovation in value-based payment reform: balancing collaboration and disruptive innovation Milbank Q 2014 92 568 623 10.1111/1468-0009.12078 25199900
29 Goicolea I Vives-Cases C San Sebastian M et al How do primary health care teams learn to integrate intimate partner violence (IPV) management? A realist evaluation protocol Implement Sci 2013 8 36 10.1186/1748-5908-8-36 23522404
30 Skivington K Matthews L Simpson SA et al A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance BMJ 2021 374 n2061 10.1136/bmj.n2061 34593508
31 Wong G Westhorp G Manzano A et al RAMESES II reporting standards for realist evaluations BMC Med 2016 14 96 10.1186/s12916-016-0643-1 27342217
32 Public Health England A brief introduction to realist evaluation Available https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1004663/Brief_introduction_to_realist_evaluation.pdf Accessed 18-Nov-2022
33 Pawson R Digging for Nuggets: How ‘Bad’ Research Can Yield ‘Good’ Evidence Int J Soc Res Methodol 2006 9 127 42 10.1080/13645570600595314
34 McHugh S Tracey ML Riordan F et al Evaluating the implementation of a national clinical programme for diabetes to standardise and improve services: a realist evaluation protocol Implement Sci 2016 11 107 10.1186/s13012-016-0464-9 27464711
35 Dalkin SM Jones D Lhussier M et al Understanding integrated care pathways in palliative care using realist evaluation: a mixed methods study protocol BMJ Open 2012 2 e001533 10.1136/bmjopen-2012-001533
36 Yin R Case Study Research: Design and Methods 5th ed Thousand Oaks, CA SAGE Publications 2014
37 Wong G Westhorp G Greenhalgh J et al Quality and reporting standards, resources, training materials and information for realist evaluation: the RAMESES II project Health Serv Deliv Res 2017 5 1 108 10.3310/hsdr05280
38 Wong G Greenhalgh T Westhorp G et al RAMESES publication standards: realist syntheses BMC Med 2013 11 21 10.1186/1741-7015-11-21 23360677
39 Cattel D Eijkenaar F Schut FT Value-based provider payment: towards a theoretically preferred design Health Econ Policy Law 2020 15 94 112 10.1017/S1744133118000397 30259825
40 Eijkenaar F Pay for performance in health care: an international overview of initiatives Med Care Res Rev 2012 69 251 76 10.1177/1077558711432891 22311954
41 Eijkenaar F Key issuses in the design of pay for performance programs Eur J Health Econ 2013 14 117 31 10.1007/s10198-011-0347-6.48 21882009
42 Wong G Westhorp G Pawson R et al Realist synthesis. Rameses training materials Available https://www.ramesesproject.org/media/Realist_reviews_training_materials.pdf Accessed 30-Aug-2024
43 Jagosh J Macaulay AC Pluye P et al Uncovering the benefits of participatory research: implications of a realist review for health research and practice Milbank Q 2012 90 311 46 10.1111/j.1468-0009.2012.00665.x 22709390
44 Greenhalgh T Humphrey C Hughes J et al How do you modernize a health service? A realist evaluation of whole-scale transformation in london Milbank Q 2009 87 391 416 10.1111/j.1468-0009.2009.00562.x 19523123
45 Tremblay D Touati N Roberge D et al Conditions for production of interdisciplinary teamwork outcomes in oncology teams: protocol for a realist evaluation Implement Sci 2014 9 76 10.1186/1748-5908-9-76 24938443
46 Teddlie C Yu F Mixed Methods Sampling: A Typology With Examples J Mix Methods Res 2007 1 77 100 10.1177/1558689819844638
47 Manzano A The craft of interviewing in realist evaluation Evaluation (Lond) 2016 22 342 60 10.1177/1356389016638615
48 Dalkin S Forster N Hodgson P et al Using computer assisted qualitative data analysis software (CAQDAS; NVivo) to assist in the complex process of realist theory generation, refinement and testing Int J Soc Res Methodol 2021 24 123 34 10.1080/13645579.2020.1803528
49 Dimick JB Ryan AM Methods for evaluating changes in health care policy: the difference-in-differences approach JAMA 2014 312 2401 2 10.1001/jama.2014.16153 25490331
50 Zeldow B Hatfield LA Confounding and regression adjustment in difference-in-differences studies Health Serv Res 2021 56 932 41 10.1111/1475-6773.13666 33978956
51 Creswell JW Plano Clark VL Designing and conducting mixed methods research Creswell JW Plano Clark VL Choosing a mixed methods design Thousand Oaks, CA SAGE Publications 2007 58 88
52 Nederlandse gedragscode wetenschappelijke integriteit 2018 Available https://easy.dans.knaw.nl/ui/datasets/id/easy-dataset:110600/tab/2 Accessed 8-Dec-2022
53 Marchal B van Belle S van Olmen J et al Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research Evaluation (Lond) 2012 18 192 212 10.1177/1356389012442444
54 Smeets RGM Hertroijs DFL Mukumbang FC et al First Things First: How to Elicit the Initial Program Theory for a Realist Evaluation of Complex Integrated Care Programs Milbank Q 2022 100 151 89 10.1111/1468-0009.12543 34812527
55 Marchal B Dedzo M Kegels G A realist evaluation of the management of a well-performing regional hospital in Ghana BMC Health Serv Res 2010 10 24 10.1186/1472-6963-10-24 20100330
56 Greenhalgh J Manzano A Understanding ‘context’ in realist evaluation and synthesis Int J Soc Res Methodol 2022 25 583 95 10.1080/13645579.2021.1918484
57 Wand T White K Patching J Applying a realist(ic) framework to the evaluation of a new model of emergency department based mental health nursing practice Nurs Inq 2010 17 231 9 10.1111/j.1440-1800.2010.00488.x 20712661
58 Simmons C Pot M Lorenz-Dant K et al Disentangling the impact of alternative payment models and associated service delivery models on quality of chronic care: A scoping review Health Policy 2024 143 S0168-8510(24)00044-7 10.1016/j.healthpol.2024.105034
59 Damschroder LJ Aron DC Keith RE et al Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science Implement Sci 2009 4 50 10.1186/1748-5908-4-50 19664226
60 Steenkamer B Drewes H Putters K et al Reorganizing and integrating public health, health care, social care and wider public services: a theory-based framework for collaborative adaptive health networks to achieve the triple aim J Health Serv Res Policy 2020 25 187 201 10.1177/1355819620907359 32178546
61 Miller H How to Create an Alternative Payment Model Designing Value-Based Payments That Support Affordable, High-Quality Healthcare Services Available https://chqpr.org/downloads/How_to_Create_an_Alternative_Payment_Model_ExecSumm.pdf Accessed 19-Nov-2022
62 Greenhalgh T Robert G Macfarlane F et al Diffusion of innovations in service organizations: systematic review and recommendations Milbank Q 2004 82 581 629 10.1111/j.0887-378X.2004.00325.x 15595944
