
==== Front
Pharmacoeconomics
Pharmacoeconomics
Pharmacoeconomics
1170-7690
1179-2027
Springer International Publishing Cham

39085565
1416
10.1007/s40273-024-01416-5
Original Research Article
Unravelling Elements of Value of Healthcare and Assessing their Importance Using Evidence from Two Discrete-Choice Experiments in England
http://orcid.org/0000-0002-3160-7097
Gongora-Salazar Pamela pamelago@iadb.org

12
http://orcid.org/0000-0003-2418-2091
Perera Rafael 3
http://orcid.org/0000-0003-2233-6544
Rivero-Arias Oliver 24
http://orcid.org/0000-0002-4662-8915
Tsiachristas Apostolos 35
1 https://ror.org/02gjn4306 grid.431756.2 0000 0004 1936 9502 Social Protection and Health Division, Inter-American Development Bank, Washington, DC USA
2 https://ror.org/052gg0110 grid.4991.5 0000 0004 1936 8948 Nuffield Department of Population Health, Health Economics Research Centre (HERC), University of Oxford, Oxford, UK
3 https://ror.org/052gg0110 grid.4991.5 0000 0004 1936 8948 Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, UK
4 https://ror.org/052gg0110 grid.4991.5 0000 0004 1936 8948 Nuffield Department of Population Health, National Perinatal Epidemiology Unit (NPEU), University of Oxford, Oxford, England UK
5 https://ror.org/052gg0110 grid.4991.5 0000 0004 1936 8948 Department of Psychiatry, University of Oxford, Oxford, UK
31 7 2024
31 7 2024
2024
42 10 11451159
7 7 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/.
Background

Health systems are moving towards value-based care, implementing new care models that allegedly aim beyond patient outcomes. Therefore, a policy and academic debate is underway regarding the definition of value in healthcare, the inclusion of costs in value metrics, and the importance of each value element. This study aimed to define healthcare value elements and assess their relative importance (RI) to the public in England.

Method

Using data from 26 semi-structured interviews and a literature review, and applying decision-theory axioms, we selected a comprehensive and applicable set of value-based elements. Their RI was determined using two discrete choice experiments (DCEs) based on Bayesian D-efficient DCE designs, with one DCE incorporating healthcare costs expressed as income tax rise. Respondent preferences were analysed using mixed logit models.

Results

Six value elements were identified: additional life-years, health-related quality of life, patient experience, target population size, equity, and cost. The DCE surveys were completed by 402 participants. All utility coefficients had the expected signs and were statistically significant (p < 0.05). Additional life-years (25.3%; 95% confidence interval [CI] 22.5–28.6%) and patient experience (25.2%; 95% CI 21.6–28.9%) received the highest RI, followed by target population size (22.4%; 95% CI 19.1–25.6%) and quality of life (17.6%; 95% CI 15.0–20.3%). Equity had the lowest RI (9.6%; 95% CI 6.4–12.1%), decreasing by 8.8 percentage points with cost inclusion. A similar reduction was observed in the RI of quality of life when cost was included.

Conclusion

The public prioritizes value elements not captured by conventional metrics, such as quality-adjusted life-years. Although cost inclusion did not alter the preference ranking, its inclusion in the value metric warrants careful consideration.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40273-024-01416-5.

http://dx.doi.org/10.13039/501100023233 National Institute for Health and Care Research Applied Research Collaboration Oxford and Thames Valley issue-copyright-statement© Springer Nature Switzerland AG 2024
==== Body
pmcKey Points for Decision Makers

The general public in England assigns high importance to elements of value that are not captured by conventional value metrics, such as quality-adjusted life-year (QALY).	
Due to its disproportionate impact on other value elements, cost should be included in value metrics with caution, especially when linear additive models are employed.	
Local healthcare commissioners in England could use the relative weights derived in this study to systematically assess new care models.	

Introduction

In response to increasing demand for healthcare, rising care costs, and tighter fiscal policies, health systems are being redesigned towards value-based care [1, 2]. This is driven by new models of care that focus on a holistic and coordinated approach of organizing healthcare services around population needs, and aim to improve population health, patient experience, and efficiency [3]. Most of these models are emerging as bottom-up innovations and are prioritized by local decision makers based on a multi-composite perception of value [4, 5]. In England, such models of care include the provider collaboratives for acute and mental health services and their predecessors [6, 7]. They can therefore range from condition-specific interventions with integrated elements, such as the Integrated Diabetes Care Programme, to broader initiatives such as the Aging Well Programme, which addresses the needs of people with multiple long-term conditions [6]. Therefore, there is an urgent need among national and local decision makers in England and elsewhere to determine the elements of value of healthcare and their relative importance (RI) to the public, integrating societal values into the decision-making process [2, 8]. This tendency stems from the public’s expectation that local commissioners make decisions on behalf of their communities, and that the public’s ‘voice’ should be incorporated in the priority-setting process, reciprocating their contribution through taxes [9].

Although a consensus on value-based healthcare (VBHC) definition remains elusive, it could be described as the equitable, sustainable, and transparent allocation of resources for enhanced outcomes and patient experiences, transcending traditional metrics such as quality-adjusted life-year (QALY) [8, 10]. The application of the VBHC concept in (local) healthcare priority setting is however still unclear [11]. Previous studies have attempted to define a comprehensive set of elements of VBHC [12–14], but its operationalization is challenging. Data are not always available, it is difficult to define a widely accepted scale of measurement (e.g., value of hope), and there is potential overlap between some of the sets of elements (e.g., enjoyment of life and psychological well-being). Currently, priority setting is informed, at its best, by several fragmented ad hoc analyses of cost effectiveness, equity, budget impact, etc., and revisited based on numerous, and frequently arbitrary, performance indicators [15].

Attempts to define VBHC are further complicated by the ongoing debate about whether the cost of healthcare should be a component of value (reflecting forgone value or opportunity cost) that could counterweigh other elements of value. Proponents argue that cost reduction is one of the objectives of new care models [16–18] and essential in decision making [13, 19]. Reflecting these principles, multiple value frameworks in the context of priority setting, such as the EU-funded Horizon2020 project SELFIE (Sustainable intEgrated chronic care modeLs for multimorbidity: delivery, FInancing, and performancE), have explicitly included cost as one of their elements [20, 21]. On the other hand, opponents contend that cost is not a value element, and its inclusion would be theoretically wrong [22–24]. This debate might be eased if cost inclusion would not change the ranking of other elements of value in terms of importance. Preliminary stated-preference studies have concluded that the integration of cost as a healthcare value element does not alter the structure of preference for other value elements [25, 26]. However, the reliability of these findings can be questioned because of the potential irrelevance of healthcare costs to respondents that do not pay directly for healthcare services (e.g., in the English National Health Service [NHS]) [26]. Furthermore, focusing these studies on specific patient groups limits their broader applicability [26, 27], overlooking the general public’s funding role through taxes or insurance.

The aim of this study was to define the elements of VBHC and assess their RI to the public in England. In addition, the study explored the impact of including cost in a multi-composite value metric on the RI of other elements of value.

Methods

Decision Context and Perspective of Value

In England, local healthcare commissioners are responsible for purchasing most of the hospital and community NHS services in their local areas. With the 2022 Health and Care Act, local commissioners became part of one of the 42 Integrated Care Systems (ICSs) across the country [28]. Each ICS aims to improve outcomes in population health and healthcare; tackle inequalities in outcomes, experience and access; and enhance productivity and value for money [29]. To achieve these goals, ICS commissioners are expected to work in collaboration with health and social care providers and the local authorities to promote the implementation of the new care models [28].

As part of this process, local commissioners have to routinely assess the performance of the different care models, and allocate tight healthcare budgets to the most valuable ones. This prioritization process is ideally based on ICS’ goals and requires a clear definition of what constitutes VBHC. In addition, commissioners are expected to act on behalf of the public as they are the taxpayer's base and the ones that bear the cost of healthcare decisions [9]. It is therefore desirable to define elements of VBHC in light of the aims that ICSs pursue and to assess their RI from the perspective of the public.

Selection and Definition of the Value Elements

We followed a stepwise approach to select relevant and measurable elements of VBHC. In the first step, we created a full list of potential candidate value elements. To do this, we conducted 26 semi-structured interviews between April and August 2021, asking local decision makers in South East England to indicate factors that are or should be considered when prioritizing health interventions, including new models of care. Information about the methodology used to conduct the interviews and analyse information collected is described in detail elsewhere [15]. The draft list of candidate value elements was supplemented with the findings of a systematic literature review of empirical studies that had applied multi-criteria decision analysis (MCDA) to inform prioritization of healthcare interventions [30]. We focused on MCDA studies because this analysis requires the explicit and systematic generation of a multi-composite value metric and assigns RI to the value elements. Furthermore, MCDA has been widely applied to guide decisions in healthcare [30, 31]. The full list included 28 potential value elements (7 from the semi-structured interviews and 21 from the systematic literature review).

In the second selection step, two researchers (PG, AT) assessed each of the candidate elements against theoretical properties of multi-composite value metrics founded in decision theory [32]. These properties included: (1) completeness—all relevant factors to the decision context were considered; (2) non-redundancy—unimportant or irrelevant to the decision context value elements were excluded; (3) non-overlap—minimize overlap between elements of value to avoid double counting; and (4) operationality—elements of value can be objectively assessed based on data that local decision makers have available to systematically assess and prioritize care models. Figure 1 summarizes the selection process of the elements of VBHC. The list of all potential elements and the corresponding assessment is illustrated in electronic supplementary material (ESM) File A. The six elements of VBHC selected were:Final or intermediate health outcomes.

Quality of life and well-being considerations.

Quality of care, patient experience or features of the process of care delivery.

Cost.

Equity.

Size of the target population.

Fig. 1 Flowchart to select value elements

Design of a Discrete Choice Experiment

To elicit the RI of the six elements of value, we designed two discrete choice experiments (DCEs) following best practice guidance for conducting such studies [33–37]. DCEs are theoretically sound, relatively easy to administer, and widely used to quantify public preferences in healthcare priority-setting [38–40].

Definition of Attributes and Levels

For the definition of the six elements of value (henceforth, ‘attributes’) and their corresponding levels, we first reviewed previous stated-preference studies in healthcare priority-setting to identify all possible options. In addition, we reviewed definitions from policy documents and surveys conducted by NHS England. Based on both sources, we elaborated on a preliminary definition for each attribute and selection of levels, taking into account formative research on best practice guidance [33–37, 41]. Afterwards, we presented the attribute definitions and corresponding levels to the Patient and Public Involvement (PPI) group from the National Institute for Health and Care Research (NIHR) Oxford and Thames Valley Applied Research Collaboration (OTV-ARC) in June 2022. We discussed each of the attributes and refined the wording based on the PPI’s feedback, as part of the pretesting stage [42]. ESM File B summarizes the rationale behind the definition of each of the attributes, and ESM File C includes a detailed explanation of the attribute-level selection process.

The attribute names, definitions and levels used in our descriptive system are presented Table 1. These attributes describe the value elements that should guide the local commissioning of models of care. As indicated in the introduction, new care models involve multiple teams or health professionals that target groups, settings, or levels rather than, or in addition to, individual patients. Examples include the ‘Aging Well Programme’, the ‘Integrated Diabetes Care Programme’, or ‘Early Intervention in Psychosis Services’. Table 1 Attributes and levels

Value-based healthcare elements	Attribute name	Definition	Levels	
Final or intermediate health outcomes	Additional years of life (YoL)a	The number of additional years of life that each patient is expected to have with the care programme	0.5 years of extra life

1 year of extra life

3 years of extra life

	
Quality-of-life and well-being considerations	Quality of life improvements (QoL)	Improvements in patients’ mobility, self-care, usual activities, pain, discomfort, anxiety, and/or depression, due to the care programme	20 points improvement

40 points improvement

60 points improvement

	
Quality of care, patient experience or features of the process of care delivery	Patient experience

(Exp)

	How patients, their families and carers experience health and social care. For example, ease of access, quality of communication, etc.	Poor

Fair

Good

	
Size of the target population	Size of target population

(Size)

	The number of people who benefit directly from the care programme per 100,000 citizens	50/100,000 citizens

5000/100,000 citizens

10,000/100,000 citizens

	
Equity	Target population

(Equ)

	Proportion of the target population that comes from disadvantaged or vulnerable backgrounds; for example, people from low-income households	25% disadvantaged

50% disadvantaged

75% disadvantaged

	
Cost	Additional budget needed

(AddTax)

	Additional income tax per year that every taxpayer needs to pay to cover the costs of the care programme	£20 additional tax/year

£40 additional tax/year

£60 additional tax/year

	
aThe decision not to set a baseline for the 'years of life' attribute (i.e. the potential age of the patient) was made in line with the aim of the study to define and assess preferences for a set of value elements that could be applied to different models of care, regardless of the age or condition of the target population (e.g. people with multimorbidity, children, the elderly). Although some discrete choice experiments have set clear baselines for longevity attributes, the literature suggests that preferences may differ depending on the baseline, with the general public often favouring gains in years of life for younger individuals over the elderly [43, 44]. While one option would have been to set an age for patients benefiting from care programmes that varied according to the age of the respondent, this approach was not considered at the design stage of the experiment.

All attributes were expected to have a positive relation with utility except for the ‘Additional Budget Needed’. The combination of the two first attributes, i.e. additional years of life and quality-of-life improvements, allowed the estimation of QALY measure. As the RI of the QALY compared with other value elements is unclear, we created a QALY gain variable using the collected experimental data (see Sect. 2.5).

Construction of Choice Tasks

An unlabelled paired comparison elicitation format was used in this experiment. Each paired comparison included two hypothetical care programmes and the profiles for each programme, describing the attributes with their specific levels. For each choice task, participants were asked to choose between two ‘Care programmes’, with no opt-out option, as local healthcare commissioners do not have such option in the real world. Participants were asked to imagine themselves as local healthcare commissioners that have to decide which care programme to prioritize for funding, given a limited budget.

To increase a participant’s attention and reduce task complexity, we used graphics and colour coding [45, 46]. A colour-blindness simulator (www.color-blindness.com) was employed to ensure accessibility of the visual elements. The choice task was discussed in a second meeting with the NIHR OTV-ARC PPI group, and tested at the Oxford Biomedical Research Centre (BRC) Open Day in July 2022 with members of the public. For the latter, copies with different potential choice tasks were printed, and members of the public were asked to complete the choice task and give their opinion (see ESM File F). Members of the PPI group confirmed that all icons/graphics were user-friendly, enabling respondents to thoughtfully engage with the different attributes while making their choices [47].

To explore the impact of including cost in a multi-composite value metric, we randomly assigned participants across two DCE versions: one excluding the cost attribute (i.e., DCE-NoAddTax) and the other including the cost attribute (i.e., DCE-AddTax). Both DCEs were identical in all other respects. An example of a choice task, with the cost attribute included, is presented in Fig. 2.Fig. 2 Example of choice task

Experimental Design

Two Bayesian D-efficient factorial designs were created using Ngene (www.choice-metrics.com) [48]. The 'NoAddTax' design included the first five attributes, each with three levels, while the 'AddTax' design also included the attribute ‘Additional budget required’. The two experimental designs were created using results from a pilot study with 40 respondents from the public. For each design, we used 28 rows divided into two blocks, to which participants were randomly allocated, resulting in 14 choice tasks in each block. A similar number of choice tasks has been used in previous similar studies [26, 40, 44, 49]. To reduce task complexity and improve choice consistency, we used attribute-level overlap, with two attributes at the same level in each choice task [45]. To this purpose, partial profile designs were used by creating candidate sets in Stata [50]. We examined the independence of preferences between attributes as this could affect the ranking of preferences and its consequent application in healthcare priority setting [24]. Although in the pilot study interaction terms were not included, we employed a model averaging approach to make possible the estimation of interaction terms [51].

Both designs had a low D-error and a relatively even distribution of movements between levels for each attribute. Details about the results of the pilot survey and the experimental design are presented in ESM Files D and E.

Survey and Data Collection

The survey, conducted online, comprised eight sections, beginning with a welcoming landing page.

Section 2 featured five screening questions, followed by survey information and informed consent in Sect. 3. Section 4 detailed the attributes and levels, including warm-up tasks for attribute familiarization. This led to two practice questions, the latter being a dominant choice task (Sect. 5), and then to the main 14 choice tasks (Sect. 6), described using simple, neutral language to ensure scenario clarity and encourage honest responses [52, 53]. Subsequent to the choice tasks, Section 7 posed two debriefing questions about survey difficulty, with the final section gathering data on respondents' education, employment, and health. The survey instrument used can be found in ESM File G.

The survey indicated an estimated 15–20 min for completion to manage participant expectations and encourage thoroughness. Based on the time reported by colleagues who tested the instrument, the pilots conducted and experience from similar DCEs [49], we anticipated a realistic minimum engagement time of around 4 min, with an expected average engagement time of approximately 15 min (i.e., the mean completion time, based on pilot studies).

We randomized the order tasks to control for learning curves and fatigue [54]. When collecting the data, we used three quality checks and disqualified respondents who (1) spent less than one-third of the median time to complete the survey; (2) chose the same alternative across the 14 choice tasks (i.e., those that always picked programme A or always picked programme B); and (3) had failed the dominant test, presented as the second practice choice task in the survey.

The survey, targeting adult members of the English public, was conducted online by Dynata, a market research firm responsible for both creation and data collection. To represent the general English population accurately, we employed national census-balanced quotas and targeted recruitment strategies across regions, sex, age, and ethnicity. Dynata ensured respondent quality and representativeness by targeting panel invitations and enforcing strict quality checks based on our quota sampling strategy.

To estimate the minimum sample size requirements, we used data from the pilot study and followed the parametric approach proposed by de Bekker-Grob and colleagues [55]. For the 'AddTax' design, we needed a minimum sample size of 166 respondents with a statistical power of 0.8 at a confidence level of 95%. For the 'NoAddTax' design, the minimum required sample size was 120 respondents. Based on these results, we aimed to collect 200 responses for each design. To account for potential left-right bias, for each of the blocks we created ‘mirror blocks’ by reversing the order in which profiles for ‘Care Programme A’ and ‘Care Programme B’ were displayed [56]. The 400 sample of respondents was therefore randomized across eight blocks, with 50 respondents per block (i.e., the four original blocks plus the four ‘mirror blocks’).

The data of the main survey were collected between 1 November and 1 December 2022.

Data Analysis

Characteristics of the respondents were summarized using descriptive statistics. To model participants’ choices for each of the designs, we assumed a random utility model under which the two alternative care programmes (A and B) are characterized by a utility function with a deterministic and a random component [38]. For each respondent n, the utility function Uni of alternative i is a random variable based on attributes that influence individual’s behaviour (i.e., deterministic or systematic component), Vni, and a stochastic disturbance term εni(i.e., random component). The later measures the deviation from the modelled utility for alternative i and respondent n (Eqs. 1a and 1b):1a DCE-NoAddTax:UniNoAddTax=VniNoAddTax+εniNoAddTaxwherei=A,B

1b DCE-AddTax:UniAddTax=VniAddTax+εniAddTaxwherei=A,B

The deterministic part of the utility, Vni, is typically assumed to have an additive structure defined by the attributes of the alternatives and the corresponding estimated parameter β, as follows (Eqs. 2a and 2b):2a VniNoAddTax=∑k=15βnXkNoAddTaxXkni=δ10+β11YoL+β12QoL+β13Exp+β14Size+β15Eq

2b VniAddTax=∑k=16βnXkAddTaxXkni=δ20+β21YoL+β22QoL+β23Exp+β24Size+β25Equ+β26AddTax

With δ10 and δ20 defining the alternative-specific constant (ASC), indicating the propensity of participants to choose A over B, and expected to be not significant in our model due to  the mirror blocking used in the experimental design. The estimated parameters β11:15andβ21:26 capture the marginal sensitivity to changes in the attribute levels. All attributes were categorical and dummy coded.

To account for random variation across respondents, we estimated mixed multinomial logit models (MMNL), with all parameters set as random and normally distributed [36, 57, 58]. For the simulation of the choice probabilities we use 1000 Modified Latin Hypercube Sampling (MLHS) draws per individual [59].

To determine the RI of the attributes, and identify whether the preference ranking changes when the monetary attribute ‘Additional Tax’ is included, we calculated RI scores using Eq. 3 [26, 60]:3 RI=Maxpwuk-Minpwuk∑kmaxpwuk-Minpwuk∗100

where pwuk corresponds to the part-worth utility (the coefficients) for the attribute k. To obtain a 95% confidence interval (CI) around the RI scores, we used a bootstrapping procedure with 1000 replications [26].

To estimate the RI of ‘QALY gain’ compared with the other elements of value, we combined the survival attribute with the health-related quality-of-life attribute. Since 'QoL' and 'YoL' have three levels each, the 'QALY gains' variable consisted of seven values ranging from 0.1 (i.e., 0.5 additional years of life and 20 points improvement in 'QoL') to 1.8 (i.e., 3 additional years of life and 60 points improvement in 'QoL'). An illustration of the combination of the years of life and quality-of-life attributes to generate QALYs is presented in Fig. 3.Fig. 3 Illustration of generating QALY gains from two attributes. QALY quality of life

To further explore the impact of the ‘Additional Tax’ attribute on people’s preferences, we estimated marginal rates of substitutions (MRS) and compared differences between the two DCE subsamples (i.e., DCE-NoAddTax vs. DCE-AddTax). To this end, ‘YoL’ and ‘QoL’ were treated as continuous.

To test the robustness of the models employed and the quality of the data, both DCEs (i.e. DCE-NoAddTax and DCE-AddTax) were re-estimated excluding respondents who completed the choice tasks in <10 min (i.e., median completion time).

To determine statistically significant coefficients and standard deviations (SDs), we used a significance level of 5%. All data analyses were performed in Apollo software [61–63].

Results

Respondent Characteristics

The survey was completed by 402 respondents via an online panel (201 in each of the DCEs). The median completion time in the DCE subsample with five attributes (i.e. DCE-NoAddTax) was 10.2 min (mean 14.3, SD 13.2), and 11.4 min (mean 14.6, SD 14.0) in the subsample with six attributes (i.e. DCE-AddTax). Around 20% found the survey difficult or very difficult (NoAddTax: 18.4%; AddTax: 19.9%). Table 2 summarizes the sociodemographic characteristics of the respondents. The total sample was representative of the English population in terms of sex, age, region and ethnicity. The two DCE subsamples were fairly similar, with people from the 'NoAddTax' DCE reporting few more comorbidities and a slightly lower EQ-5D-5L index or visual analogue scale (VAS)-based quality-of-life score; however, none of these differences was statistically significant (p < 0.05). Table 2 Sociodemographic characteristics of the respondents

	NoAddTax-DCE attribute [n = 201]	AddTax-DCE attribute [n = 201]	Total sample	National statisticsa (%)	
Sex					
 Male	95 (47.3)	95 (47.3)	190 (47.3)	49.0	
 Female	105 (52.2)	105 (52.2)	210 (52.2)	51.0	
 Other	1 (0.5)	1 (0.5)	2 (0.5)		
Age, years					
 18–24	23 (11.4)	18 (9.0)	41 (10.2)	10.6	
 25–34	31 (15.4)	35 (17.4)	66 (16.4)	17.1	
 35–44	34 (16.9)	33 (16.4)	67 (16.7)	16.2	
 45–54	40 (19.9)	35 (17.4)	75 (18.7)	16.9	
 55–64	33 (16.4)	30 (14.9)	63 (15.7)	15.7	
 65+	40 (19.9)	50 (24.9)	90 (22.4)	23.5	
Mean (SD)	47.41 (16.08)	48.38 (16.31)			
Median (IQR)	49 (34–60)	49 (34–64)			
Region					
 South East	31 (15.4)	34 (16.9)	65 (16.2)	16.3	
 South West	21 (10.5)	21 (10.5)	42 (10.4)	10.2	
 London	30 (14.9)	33 (16.4)	63 (15.7)	15.6	
 East of England	27 (13.4)	18 (9.0)	45 (11.2)	11.1	
 West Midlands	20 (10.0)	23 (11.4)	43 (10.7)	10.5	
 East Midlands	13 (6.5)	20 (10.0)	33 (8.2)	8.7	
 Yorkshire and the Humber	21 (10.5)	19 (9.5)	40 (10.0)	9.8	
 North West	27 (13.4)	25 (12.4)	52 (12.9)	13.0	
 North East	11 (5.5)	8 (4.0)	19 (4.7)	4.8	
Ethnic group					
 White	171 (85.1)	170 (84.6)	341 (84.8)	84.2	
 Mixed/multiple ethnic groups	3 (1.5)	3 (1.5)	6 (1.5)	1.9	
 Asian or Asian British	15 (7.5)	19 (9.5)	34 (8.5)	8.3	
 Black, Black British, Caribbean or African	10 (4.5)	7 (3.5)	17 (4.2)	3.7	
 Other ethnic group	2 (1.0)	2 (1.0)	4 (1.0)	1.9	
Education					
 Primary education	1 (0.5)	1 (0.5)			
 Secondary education	56 (27.9)	83 (41.3)			
 Short-cycle tertiary education	39 (19.4)	16 (8.0)			
 Bachelor’s or equivalent level	69 (34.3)	67 (33.3)			
 Postgraduate education	35 (17.4)	32 (15.9)			
 I prefer not to answer	1 (0.5)	2 (1.0)			
Health problems					
 At least one	146 (72.6)	129 (64.2)			
 Two or more	83 (41.3)	63 (31.3)			
 In the family	114 (56.7)	93 (46.3)			
 Carer for a family member	36 (17.9)	33 (16.2)			
Utility (EQ-5D-5L)b					
 Mean (SD)	0.85 (0.20)	0.87 (0.19)			
 Median (IQR)	0.92 (0.82–1)	0.92 (0.83–1)			
Utility (EuroQol VAS)b					
 Mean (SD)	71.4 (20.0)	73.9 (19.0)			
 Median (IQR)	75 (60–87)	79 (69–86)			
Data are expressed as n (%) unless otherwise specified

DCE discrete choice experiment, VAS visual analogue scale, SD standard deviation, IQR interquartile range

aWe used interlocked, nationally representative quotas on age, sex and region [64], and non-interlocked quotas on ethnicity [65]

bWe computed the EQ-5D-5L utilities in Stata using tariffs from the public in England [50, 66]. For both questions, the EQ-5D-5L and the Quality of Life–VAS, we used the official instruments [67]

Choice Analysis

Results from the random parameter panel mixed logit models are presented in Table 3, with all variables treated as categorical. Part-worth utilities are summarized in ESM File H. Results from the root likelihood test indicated a good fit of the choice model on the respondents’ choices. In both choice experiments (DCE-NoAddTax and DCE-AddTax), the models were robust even when respondents who completed the survey in <10 min were excluded (104 responses under DCE-NoAddTax, and 128 responses under DCE-AddTax) [see ESM File I]. When the 'AddTax' attribute was not included, all coefficients had the expected sign and were statistically significant. The derived SD of most of the random coefficients was highly significant (p < 0.05), indicating the existence of heterogeneity across respondents around the mean parameter estimate (see Table 3). Table 3 Parameter coefficients from the mixed logit model

Variable	MMNL DCE – NoAddTax	MMNL DCE – AddTax	
Coefficient (SE)	CI	t-ratio	SD (SE)	t-ratio	Coefficient (SE)	CI	t-ratio	SD (SE)	t-ratio	
ASC	0.094 (0.054)	− 0.010 to 0.199	1.766	0.190 (0.139)	1.330	.– 0.003 (0.058)	– 0.114 to 0.109	– 0.049	0.236 (0.137)	1.754	
Additional years of life—YoL											
 1 year of extra life	0.711 (0.085)	0.537–0.885	7.990	– 0.025 (0.396)	– 0.106	0.731 (0.137)	0.466–0.995	5.411	– 0.194 (0.310)	– 0.678	
 3 years of extra life	1.891 (0.174)	1.520–2.263	9.979	1.100 (0.158)	6.542	2.288 (0.265)	1.791–2.785	9.023	1.330 (0.218)	6.058	
Quality-of-life improvements—QoL											
 40 points improvement	0.690 (0.134)	0.401– 0.980	4.681	0.015 (0.207)	0.265	0.742 (0.160)	0.4330–1.051	4.706	0.008 (0.176)	0.133	
 60 points improvement	1.313 (0.14)	1.007–1.619	8.416	.– 0.482 (0.224)	–1.784	1.096 (0.185)	0.686–1.505	5.241	– 0.603 (0.172)	–3.659	
Patient experience—Exp											
 Fair	1.610 (0.137)	1.275–1.945	9.419	0.615 (0.198)	2.832	1.403 (0.218)	0.998–1.808	6.790	– 0.911 (0.161)	–5.576	
 Good	1.880 (0.192)	1.455–2.305	8.665	.–1.147 (0.209)	–4.686	2.02 (0.206)	1.590–2.450	9.213	1.430 (0.174)	7.885	
Size of target population—Size											
 5000/100,000 citizens	1.068 (0.140)	0.753–1.384	6.637	.– 0.335 (0.315)	–1.243	1.204 (0.207)	0.866–1.542	6.982	– 0.033 (0.161)	– 0.932	
 10,000/100,000 citizens	1.672 (0.151)	1.309–2.035	9.040	.–1.159 (0.145)	–7.424	1.879 (0.232)	1.386–2.372	7.471	–1.207 (0.136)	–8.576	
Target population—Equ											
 50% disadvantaged	0.768 (0.139)	0.495–1.040	5.520	.– 0.565 (0.243)	–2.373	0.311 (0.140)	0.025– 0.598	2.129	– 0.115 (0.316)	– 0.535	
 75% disadvantaged	0.714 (0.111)	0.468– 0.961	5.674	.– 0.959 (0.127)	–6.633	0.256 (0.222)	– 0.136 to 0.649	1.282	0.329 (0.265)	1.450	
Additional budget needed—AddTax											
 £40 additional tax/year						– 0.737 (0.144)	–1.005 to – 0.468	–5.381	0.047 (0.231)	0.564	
 £60 additional tax/year						–1.491 (0.214)	–1.923 to –1.058	–6.754	1.265 (0.143)	8.319	
Log likelihood (final)	− 1587.49	–1611.79	
AIC	3218.98	3275.59	
BIC	3349.71	3.43E+03	
ASC alternative‐specific constant, SE standard error, AIC Akaike Information Criterion, BIC Bayesian Information Criterion, MMNL mixed multinomial logit model, DCE discrete choice experiment, CI confidence interval, SD standard deviation, RLH root-log likelihood

We compute the probability of an RLH value being ≤ 1/k, with k denoting the number of alternatives per choice tasks [68]. In all models the Pr (RLH ≤ 0.5) is close to zero. In the MMNL DCE–AddTax model, the respondent with the worst average fit across observations had an RLH of 0.368. In both models, < 10% of the sample failed the test. RI scores (i.e., criteria weights) did not change (no statistically significant difference) when these respondents were removed from the models

According to the RI mean scores (Fig. 4a), people in England assigned the highest values to 'YoL' (25.3%; 95% CI 22.5–28.6%), 'Exp' (25.2%; 95% CI 21.6–28.9%) and 'Size' (22.4%; 95% CI 19.1–25.6%), followed by 'QoL' (17.6%; 95% CI 15.0–20.3%). The 'Equ' attribute was the least important in our descriptive system (9.6%; 95% CI 6.4–12.1%). When the 'AddTax' attribute was present, ‘Equ’ is no longer statistically significant (2.8%; 95% CI −2.1% to 6.4%) and there is an equal-sized trade-off between the importance of ‘AddTax’ and the importance of 'QoL' and 'Equ'. 'YoL' (25.3; 95% CI 22.1–29.3%), 'Exp' (22.4%; 95% CI 18.6–26.6%) and ‘Size’ (20.8%; 95% CI 18.1–23.8%) remain as the most important attributes, followed by 'AddTax' (16.5%; 95% CI 13.7–18.8%) and 'QoL' (12.1%; 95% CI 9.3–14.6%).Fig. 4 Comparison of RI scores (a) RI scores with error bars (95% CI); (b) RI scores with QALY gain variable derived from DCE data and error bars (95% CI). RI relative importance, CI confidence interval, DCE discrete choice experiment, QALY quality-adjusted life-year. *Error bars (95% confidence interval)

When we use the 'QALY gains' variable in the DCE-NoAddTax model (ESM File J), all parameter estimates have the expected signs and are statistically significant. According to the RI scores (Fig. 4b), 'QALY gains' (36.1%; 95% CI 32.0–40.8%) and 'Exp' (26.6%; 95% CI 22.7–31.1%) were valued more by people, followed by 'Size' (26.4%; 95% CI 22.5–30.3%) and 'Equ' (10.9%; 95% CI 7.5–14.2%). If the 'AddTax' attribute is present, 'QALY gains' (32.4%; 95% CI 27.9–39.2%), 'Size' (26.2%; 95% CI 22.0–32.5%) and 'Exp' (25.1%; 95% CI 19.9–33.5%) remain as the most preferred elements to the general public in England, but the 'Equ' parameter is not statistically different to zero. As previously, having to pay additional income taxes for a care programme seems to contribute to patients’ choice (14.3%; 95% CI 10.4–19.4%), although not as much as the first three value elements.

ESM File K details the MRS estimates derived from mixed logit models treating 'Yol', 'QoL', and 'AddTax' as continuous variables. Statistically significant differences in MRS estimates expressed in terms of 'YoL' are observed between the two DCE subsamples (i.e., NoAddTax vs. AddTax) across all value attributes except for 'Size' and 'Equ'. For example, when the 'AddTax' attribute is present, individuals seem willing to trade nearly 1 year of life for a 1-point improvement in quality of life (MRS 0.86; 95% CI 0.85–0.88), whereas without 'AddTax', the willingness drops to merely 0.05 years for quality-of-life improvements (MRS 0.05; 95% CI 0.049–0.05). When 'QoL' serves as the denominator, MRS estimates significantly differ for all attributes.

Finally, in regard to the preference independence principle, we found that some of the interaction terms were statistically significant (e.g., 'Size' and 'QoL'). We therefore ran mixed logit models allowing for correlation between the random parameters [69], and results were similar those in Table 3; no significant improvement in the model fit was found. When calculating the corresponding RI scores, results led to the same attribute ordering. ESM Files L and M summarize the results of models with the interactions between attributes, and results of the correlated mixed logit.

Discussion

The findings of this study suggest that life-years and health-related quality of life are important elements of VBHC, therefore providing reassurance that the use of QALYs, as a bi-composite measure of value, in prioritization decisions in healthcare is meaningful and relevant [10]. However, our results suggest that the perceived value of healthcare is broader than a QALY as it extends to patient experience, and covers distributional aspects regarding the size of the benefitted population and its socioeconomic vulnerability. This broadened perspective aligns with several international and national trends in defining and assessing VBHC. The UK government, in particular, has incorporated patient experience and equity at the core of ICS priorities [29]. Previous literature has also shown that people value benefits of healthcare other than health benefits and are even willing to trade health gains for better care experiences [10, 14].

The six elements of value identified in this study align with the four pillars of VBHC proposed by the Expert Panel of the European Commission [70]. First, ‘Final or intermediate health outcomes’ and ‘Quality of life and well-being considerations’ align with personal value (pillar 1) as they address individual patients' goals. ‘Quality of care, patient experience or features of the process of care delivery’ relates to ‘societal value’ (pillar 2) as high-quality care and positive experiences may enhance social participation and connectedness. ‘Equity’ and ‘Size of the target population’ align with allocative value (pillar 3) by addressing social disparities and promoting equitable benefit distribution across all patient groups. Finally, ‘Cost’ ties into technical value (pillar 4) through the efficient use of resources. These six value elements also encompass the NHS England definition of VBHC: “the equitable, sustainable and transparent use of the available resources to achieve better outcomes and experiences for every person” [8].

Our findings indicate patient experience as the second most important value element, aligning with studies acknowledging its importance, although none exclusively focused on a 'Patient experience' attribute [14, 71, 72]. In a DCE study conducted for eight European countries, for instance, ‘continuity of care’ and ‘person-centeredness’ are used as measures of patient experience. According to results from the UK, respondents do not place these two attributes at the top of the preference ranking, but the sum of their RI scores suggests that people value patient experience more than physical functioning [14].

The relatively high value assigned to ‘Patient experience’ may reflect challenges in the process of care delivery (e.g., long waiting lists, lack of continuity of care). The coronavirus disease 2019 (COVID-19) pandemic might have played a role by increasing the number of patients waiting for treatment as well as people’s demand for better provider interaction [73]. Another explanation may be the rising numbers of people living with multiple conditions (more than one in four of the adult population in England) or with a long-standing health problem (37.7% in the UK). Healthcare needs for these patients are typically more intricate, requiring more complex and coordinated healthcare than healthier patients [74, 75]. One could argue that the use of smiley/sad faces as the graphic for the patient experience attribute might have induced respondents to rely on heuristics (e.g., salience bias). However, similar graphics have been used successfully in other choice experiments [76], and our PPI group confirmed that the other icons were also accessible and allowed respondents to engage thoughtfully with the various attributes when making their choices. Regardless of the underlying reasons, the high RI of patient experience highlights the need for measuring patient experience in the NHS and providing these data to decision makers. However, patient experience is a broad and complex concept that refers to the entire care delivery journey, and as such, encompasses many dimensions [77]. It is therefore crucial to agree on a set of universal patient-reported experience measures and standardized data collection for effective care model evaluation and monitoring. Ongoing initiatives to routinely collect data on continuity and coordination of care, responsiveness to patient concerns, the opportunity of care, and professional/patient communication should be strengthened [78].

‘Quality-of-life improvements’ were valued less than ‘Additional years of life’, adding to the voices that question the use of QALYs in certain decision-making contexts. QALYs overlook individual preference variations and distributional concerns. Hence, a care programme that leads to a 0.8 QALY gain is seen as equal in value to another care programme that also results in 0.8 QALY gain. Our findings suggest that care models that increase life expectancy at lower levels of quality of life may generate more value to the population than care programmes that achieve the same QALY gain, but with higher improvement in quality of life and less gains in life expectancy. This is in line with studies showing that, from a societal perspective, some QALY gains due to improvements in longevity are more valuable than QALY gains associated with improvements in health-related quality of life [79, 80].

Regarding equity, our study indicates it ranks lowest in value, especially when healthcare cost was factored into the equation, aligning with previous research that shows a preference for efficiency or effectiveness over equity [81, 82]. The relatively low value assigned to equity contrasts with the high value given to the size of the target population, suggesting that the public favours an allocation of healthcare benefits across more people than to favour the most vulnerable populations. At first glance, this does not conform to John Rawls’ difference justice principle, under which social and economic inequalities must be arranged so that they work to the greatest benefit of the less fortunate [83]; however, this result may also reflect a society that recognizes the NHS as a universal health system under which all residents are assured access to healthcare. As such, people place a higher value on universal access to healthcare, and therefore on care programmes that offer high population coverage.

Regarding the healthcare costs, our results suggest that people care about the additional healthcare costs that a care programme entails. Although this attribute did not receive the highest value and its inclusion did not alter preferences ordering, it reduced the importance attained to ‘Quality of life improvements’ and ‘Target population’ attributes, almost by the same proportion, and had a ‘crowding-out’ effect on the ‘Equity’ attribute. Differences on the MRS between the two DCE subsamples also confirmed the impact of healthcare costs on people’s preferences. These results might be partially explained by the cost-of-living crisis currently being experienced in England [84].

Our results suggest that a decision framework is incomplete when the preferences for the monetary attribute are ignored. Nevertheless, its inclusion in the value metric reduces the relevance of other elements of value, with some more affected by others, and it can also be argued that cost is not a value element in itself [22–24]. Both approaches may be relevant and useful to decision makers. On the one hand, the relative weights derived in this study can be used to calculate a cost-per-value ratio, with the denominator based on multi-attribute benefit scores (i.e. using the results when costs were excluded). An MCDA could be used for this purpose, together with the use of routinely collected data to obtain performance scores [30, 31]. However, as the elements of value have been broadly defined, decision makers would need to decide on the most appropriate performance indicators to operationalize each attribute (MCDA criteria), and, to avoid potential bias or inconsistency, it would be crucial to justify the choice of these indicators, taking into account the specific health condition or type of health intervention being assessed [85]. This approach would make the efficiency of the care programmes under evaluation more explicit. On the other hand, the relative weights of the six elements of value can be used to obtain a single composite measure for all different alternatives, making the contribution of each element more explicit [10]. Under this approach, decision makers could also develop a ranking of several competing alternative care programmes to inform budget allocation decisions [19].

Strengths and Limitations

The main strength of this study is the decomposition of cost from value, expressed in terms of additional income taxes. This is the first choice experiment using a split-sample design that expresses costs in terms of income tax increase, adding to the literature in the field. Previous studies have used an out-of-pocket expenditure or changes in insurance premiums-based definition. In principle, this suggests that the type of payment vehicle used to define the monetary attribute does not seem to affect choice preferences in health [26]. The second strength of this study is the use of interviews, literature and decision theory to identify a comprehensive yet applicable set of value-based elements. The suggested elements of value not only encompass the NHS England definition of VBHC but can be operationalized on the basis of routinely collected data. Local healthcare commissioners in England could use the relative weights that we derived in this study to systematically assess new models of care using an MCDA approach. In particular, an additive MCDA model, easy to communicate to decision makers, can be employed to assess new care models, as the elements of value identified in this study comply with the axiomatic properties of decision theory [85]. This adds to efforts to incorporate public values into the local priority-setting and resource-allocation process [86]. Another strength lies in the rigorous methodology followed to conduct the DCEs, which included two pilot studies, involvement from PPI representatives, and the use of quality checks to increase internal validity.

This study has also several limitations. First, the semi-structured interviews were conducted with managers and commissioners, and did not cover carers, clinicians or patients. Second, we did not elicit preferences of different stakeholder groups for the elements of value. However, the sample was representative of the English general population and EQ-5D-5L health states are derived from the general population's preferences through choice experiments. In addition, a latent class analysis of a similar DCE concluded that the preferences of public, patients, payers, and healthcare providers for value elements of integrated care were not statistical significantly different [14]. Third, as respondents received a small payment for participating, this payment might bias the sample if individuals participate solely for financial gain rather than interest in the survey topic. However, evidence suggests that the effect of incentives on response quality can vary depending on the type of survey and the amount of incentive given [86-88], and we used multiple quality checks when collecting the data. Another limitation lies in the review's focus on MCDA in healthcare, rather than on a broader value-based care literature. This may have resulted in a selective sample of value elements towards those highlighted in MCDA literature, rather than a more targeted value-based care perspective. However, although not exhaustive, using both interviews and a systematic literature review helped create a comprehensive list of potential VBHC elements. A last limitation concerns the relative weight estimated for ‘QALY gains’ as it was a construction from the stated preference dataset. Future studies could explore the extent to which preferences for QALY gains differ when its elicitation is derived from a choice experiment that directly includes QALY gains into the descriptive system.

Conclusions

Although preferences for maximizing population-level health gains remain highly relevant, our results highlight the importance of elements of value that are not included in traditional health economic evaluations, as they are not captured by conventional value metrics, such as QALY. The relatively high importance assigned to patient experience stresses the need for collating data on this dimension and adopting economic evaluation approaches that account for the multiple elements of VBHC. Healthcare decision makers should also be cautious when including cost in a multi-composite value metric as it may reduce the RI of other elements of value, such as equity.

Supplementary Information

Below is the link to the electronic supplementary material.Supplementary file1 (DOCX 1619 KB)

Acknowledgements

The authors thank the respondents who took part in this study, as well as members from the Patient and Public Involvement group from the NIHR Oxford and Thames Valley Applied Research Collaboration for their feedback on the survey instrument. They would also like to thank Dr John Buckell and Professor Marcel Jonker for their advice when conducting the DCE experiment and analysing the results. The authors are also grateful for the support from the NIHR Applied Research Collaboration Oxford and Thames Valley.

Declarations

Funding

This study was funded by the Nuffield Department of Population Health of the University of Oxford as part of the corresponding author’s doctorate scholarship, and by the NIHR Oxford and Thames Valley Applied Research Collaboration. AT also acknowledges support from the Oxford Biomedical Research Centre.

Conflicts of interest

Pamela Gongora-Salazar, Rafael Perera, Oliver Rivero-Arias, and Apostolos Tsiachristas declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethics approval

Ethics approval to conduct this study was obtained from the Medical Sciences Inter-Divisional Research Ethics Committee (IDREC) at the University of Oxford (R74765/RE003).

Informed consent and Consent for publication

All respondents provided informed consent before entering the study.

Data availability

The dataset for this study is available from the corresponding author on reasonable request.

Code availability

The code for estimating RI using R is available upon request to the study authors.

Author Contributions

PG, AT and OR designed the study and identified attributes and levels, with inputs from RP. PG created the experimental design, guided by OR. PG coordinated the fieldwork and performed the analysis. PG and AT interpreted the results, with input from OR. AT and RP secured funding. PG drafted the manuscript, with critical review by AT, OR and RP.

Oliver Rivero-Arias and Apostolos Tsiachristas are co-senior authors on this work.
==== Refs
References

1. World Health Assembly World Health Assembly (69): Framework on integrated, people-centred health services: report by the Secretariat 2016 Geneva World Health Organization
World Health Assembly. World Health Assembly (69): Framework on integrated, people-centred health services: report by the Secretariat. Geneva: World Health Organization; 2016.
2. Gray M. Value based healthcare. BMJ. 2017;356.
3. World Health Organization. Integrated care models: an overview. World Health Organization Regional Office for Europe; 2016. p. 31-31.
4. Zanotto BS Value-based healthcare initiatives in practice: a systematic review J Healthc Manag 2021 66 5 340 365 34192716
Zanotto BS, et al. Value-based healthcare initiatives in practice: a systematic review. J Healthc Manag. 2021;66(5):340–65.34192716
5. Thorstensen-Woll C, Bottery S. Integrated care systems and social care: the opportunities and challenges. 2021 December 8, 2021 January 21. Available at: https://www.kingsfund.org.uk/publications/integrated-care-systems-and-social-care.
6. NHS England, Five-Year Forward View. 2014.
7. Lewis RQ Integrated Care in England - what can we Learn from a Decade of National Pilot Programmes? Int J Integr Care 2021 21 4 5 10.5334/ijic.5631 34754281
Lewis RQ, et al. Integrated Care in England - what can we Learn from a Decade of National Pilot Programmes? Int J Integr Care. 2021;21(4):5.34754281 10.5334/ijic.5631
8. Hurst L Defining Value-based Healthcare in the NHS Centre Evid-Based Med Report 2019 04 1 13
Hurst L, et al. Defining Value-based Healthcare in the NHS. Centre Evid-Based Med Report. 2019;04:1–13.
9. Broqvist M Garpenby P To accept, or not to accept, that is the question: citizen reactions to rationing Health Expect 2014 17 1 82 92 10.1111/j.1369-7625.2011.00734.x 22032636
Broqvist M, Garpenby P. To accept, or not to accept, that is the question: citizen reactions to rationing. Health Expect. 2014;17(1):82–92.22032636 10.1111/j.1369-7625.2011.00734.x
10. Spencer A The QALY at 50: one story many voices Soc Sci Med 2022 296 114653 10.1016/j.socscimed.2021.114653 35184921
Spencer A, et al. The QALY at 50: one story many voices. Soc Sci Med. 2022;296: 114653.35184921 10.1016/j.socscimed.2021.114653
11. Cossio-Gil Y The roadmap for implementing value-based healthcare in European University Hospitals—Consensus Report and Recommendations Value in Health 2022 25 7 1148 1156 10.1016/j.jval.2021.11.1355 35779941
Cossio-Gil Y, et al. The roadmap for implementing value-based healthcare in European University Hospitals—Consensus Report and Recommendations. Value in Health. 2022;25(7):1148–56.35779941 10.1016/j.jval.2021.11.1355
12. Lakdawalla DN Defining elements of value in health care—a health economics approach: an ISPOR Special Task Force Report [3] Value in Health 2018 21 2 131 139 10.1016/j.jval.2017.12.007 29477390
Lakdawalla DN, et al. Defining elements of value in health care—a health economics approach: an ISPOR Special Task Force Report [3]. Value in Health. 2018;21(2):131–9.29477390 10.1016/j.jval.2017.12.007
13. Leijten FRM The SELFIE framework for integrated care for multi-morbidity: Development and description Health Policy 2018 122 1 12 22 10.1016/j.healthpol.2017.06.002 28668222
Leijten FRM, et al. The SELFIE framework for integrated care for multi-morbidity: Development and description. Health Policy. 2018;122(1):12–22.28668222 10.1016/j.healthpol.2017.06.002
14. Rutten-van Mölken M Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries BMJ Open 2020 10 10 e037547 10.1136/bmjopen-2020-037547 33039997
Rutten-van Mölken M, et al. Comparing patients’ and other stakeholders’ preferences for outcomes of integrated care for multimorbidity: a discrete choice experiment in eight European countries. BMJ Open. 2020;10(10): e037547.33039997 10.1136/bmjopen-2020-037547
15. Gongora-Salazar P, Glogowska M, Fitzpatrick R, Perera R, Tsiachristas A. Commissioning [Integrated] care in England: an analysis of the current decision context. Int J Integr Care. 2022;22(4):3,1-16. 10.5334/ijic.6693
16. Berwick DM Nolan TW Whittington J The triple aim: Care, health, and cost Health Aff 2008 27 3 759 769 10.1377/hlthaff.27.3.759
Berwick DM, Nolan TW, Whittington J. The triple aim: Care, health, and cost. Health Aff. 2008;27(3):759–69.10.1377/hlthaff.27.3.759
17. Shaw S, Rosen R, Rumbold B. What is integrated care?, In: An overview of integrated care in the NHS. 2011: London.
18. Stokes J Checkland K Kristensen SR Integrated care: theory to practice J Health Serv Res Policy 2016 21 4 282 285 10.1177/1355819616660581 27473860
Stokes J, Checkland K, Kristensen SR. Integrated care: theory to practice. J Health Serv Res Policy. 2016;21(4):282–5.27473860 10.1177/1355819616660581
19. van den Bogaart EHA Economic Evaluation of new models of care: does the decision change between cost-utility analysis and multi-criteria decision analysis? Value in Health 2021 24 6 795 803 10.1016/j.jval.2021.01.014 34119077
van den Bogaart EHA, et al. Economic Evaluation of new models of care: does the decision change between cost-utility analysis and multi-criteria decision analysis? Value in Health. 2021;24(6):795–803.34119077 10.1016/j.jval.2021.01.014
20. Rutten-van Mölken M Strengthening the evidence-base of integrated care for people with multi-morbidity in Europe using Multi-Criteria Decision Analysis (MCDA) BMC Health Serv Res 2018 18 1 576 10.1186/s12913-018-3367-4 30041653
Rutten-van Mölken M, et al. Strengthening the evidence-base of integrated care for people with multi-morbidity in Europe using Multi-Criteria Decision Analysis (MCDA). BMC Health Serv Res. 2018;18(1):576.30041653 10.1186/s12913-018-3367-4
21. Zhang M What is value in health and healthcare? A systematic literature review of value assessment frameworks Value Health 2022 25 2 302 317 10.1016/j.jval.2021.07.005 35094803
Zhang M, et al. What is value in health and healthcare? A systematic literature review of value assessment frameworks. Value Health. 2022;25(2):302–17.35094803 10.1016/j.jval.2021.07.005
22. Claxton K Three questions to ask when examining MCDA Value Outcomes Spotlight 2015 1 1 18 20
Claxton K. Three questions to ask when examining MCDA. Value Outcomes Spotlight. 2015;1(1):18–20.
23. Sculpher M Claxton K Pearson SD Developing a value framework: the need to reflect the opportunity costs of funding decisions Value Health 2017 20 2 234 239 10.1016/j.jval.2016.11.021 28237201
Sculpher M, Claxton K, Pearson SD. Developing a value framework: the need to reflect the opportunity costs of funding decisions. Value Health. 2017;20(2):234–9.28237201 10.1016/j.jval.2016.11.021
24. Marsh KD The use of MCDA in HTA: great potential, but more effort needed Value Health 2018 21 4 394 397 10.1016/j.jval.2017.10.001 29680094
Marsh KD, et al. The use of MCDA in HTA: great potential, but more effort needed. Value Health. 2018;21(4):394–7.29680094 10.1016/j.jval.2017.10.001
25. Sever I Verbič M Klarić Sever E Cost attribute in health care DCEs: Just adding another attribute or a trigger of change in the stated preferences? J Choice Model 2019 32 100135 10.1016/j.jocm.2018.03.005
Sever I, Verbič M, Klarić Sever E. Cost attribute in health care DCEs: Just adding another attribute or a trigger of change in the stated preferences? J Choice Model. 2019;32: 100135.10.1016/j.jocm.2018.03.005
26. Genie MG Ryan M Krucien N To pay or not to pay? Cost information processing in the valuation of publicly funded healthcare Soc Sci Med 2021 276 113822 10.1016/j.socscimed.2021.113822 33752103
Genie MG, Ryan M, Krucien N. To pay or not to pay? Cost information processing in the valuation of publicly funded healthcare. Soc Sci Med. 2021;276: 113822.33752103 10.1016/j.socscimed.2021.113822
27. Bryan S Magnetic resonance imaging for the investigation of knee injuries: an investigation of preferences Health Econ 1998 7 7 595 603 10.1002/(SICI)1099-1050(1998110)7:7<595::AID-HEC381>3.0.CO;2-E 9845253
Bryan S, et al. Magnetic resonance imaging for the investigation of knee injuries: an investigation of preferences. Health Econ. 1998;7(7):595–603.9845253 10.1002/(SICI)1099-1050(1998110)7:7<595::AID-HEC381>3.0.CO;2-E
28. UK Government, Health and Care Act 2022, in Chapter 31. 2022.
29. NHS England and NHS Improvement, Building strong integrated care systems everywhere. ICS implementation guidance on working with people and communities. 2021: London.
30. Gongora-Salazar P, et al. The use of multicriteria decision analysis to support decision making in healthcare: an updated systematic literature review. Value in Health. 2022.
31. Department of Health and Social Care. The Prioritisation Framework: making the most of your budget. 2018 [cited 2021 Dec 7]. Available at: https://www.gov.uk/government/publications/the-prioritisation-framework-making-the-most-of-your-budget.
32. Dodgson J, et al. Multi-criteria analysis: a manual. 2009: London.
33. Bliemer M, Rose J. Designing and conducting stated choice experiments. In: Hess, S., Daly, A. editors. Handbook of choice modelling, Second edition, forthcoming. 2022.
34. Bridges JFP Conjoint analysis applications in health—a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force Value in Health 2011 14 4 403 413 10.1016/j.jval.2010.11.013 21669364
Bridges JFP, et al. Conjoint analysis applications in health—a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value in Health. 2011;14(4):403–13.21669364 10.1016/j.jval.2010.11.013
35. Green J Thorogood N Qualitative methods for health research 2018 Berlin SAGE Publications Ltd 440 440
Green J, Thorogood N. Qualitative methods for health research. Berlin: SAGE Publications Ltd; 2018. p. 440–440.
36. Hauber AB Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force Value in Health 2016 19 4 300 315 10.1016/j.jval.2016.04.004 27325321
Hauber AB, et al. Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force. Value in Health. 2016;19(4):300–15.27325321 10.1016/j.jval.2016.04.004
37. Johnson R Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force Value in Health 2013 16 1 3 13 10.1016/j.jval.2012.08.2223 23337210
Johnson R, et al. Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force. Value in Health. 2013;16(1):3–13.23337210 10.1016/j.jval.2012.08.2223
38. McFadden D Editor ZP Conditional logit analysis of qualitative choice behavior Frontiers in econometrics 1974 New York Academic Press 105 142
McFadden D. Conditional logit analysis of qualitative choice behavior. In: Editor ZP, editor. Frontiers in econometrics. New York: Academic Press; 1974. p. 105–42.
39. Lancsar E Fiebig DG Hole AR Discrete Choice Experiments: A Guide to Model Specification Estimation and Software. PharmacoEconomics 2017 35 7 697 716 10.1007/s40273-017-0506-4 28374325
Lancsar E, Fiebig DG, Hole AR. Discrete Choice Experiments: A Guide to Model Specification. Estimation and Software PharmacoEconomics. 2017;35(7):697–716.28374325 10.1007/s40273-017-0506-4
40. Soekhai V Discrete choice experiments in health economics: past, present and future Pharmacoeconomics 2019 37 2 201 226 10.1007/s40273-018-0734-2 30392040
Soekhai V, et al. Discrete choice experiments in health economics: past, present and future. Pharmacoeconomics. 2019;37(2):201–26.30392040 10.1007/s40273-018-0734-2
41. Rivero-Arias O Using stated-preferences methods to develop a summary metric to determine successful treatment of children with a surgical condition: a study protocol BMJ Open 2022 12 6 e062833 10.1136/bmjopen-2022-062833 35680263
Rivero-Arias O, et al. Using stated-preferences methods to develop a summary metric to determine successful treatment of children with a surgical condition: a study protocol. BMJ Open. 2022;12(6): e062833.35680263 10.1136/bmjopen-2022-062833
42. Campoamor NB Pretesting discrete-choice experiments: a guide for researchers The Patient Patient-Cent Outcomes Res 2024 17 2 109 120 10.1007/s40271-024-00672-z
Campoamor NB, et al. Pretesting discrete-choice experiments: a guide for researchers. The Patient Patient-Cent Outcomes Res. 2024;17(2):109–20.10.1007/s40271-024-00672-z
43. Whitty JA A systematic review of stated preference studies reporting public preferences for healthcare priority setting The Patient Patient-Cent Outcomes Res 2014 7 4 365 386 10.1007/s40271-014-0063-2
Whitty JA, et al. A systematic review of stated preference studies reporting public preferences for healthcare priority setting. The Patient Patient-Cent Outcomes Res. 2014;7(4):365–86.10.1007/s40271-014-0063-2
44. Gu Y Attributes and weights in health care priority setting: a systematic review of what counts and to what extent Soc Sci Med 2015 146 41 52 10.1016/j.socscimed.2015.10.005 26498059
Gu Y, et al. Attributes and weights in health care priority setting: a systematic review of what counts and to what extent. Soc Sci Med. 2015;146:41–52.26498059 10.1016/j.socscimed.2015.10.005
45. Jonker MF Attribute level overlap (and color coding) can reduce task complexity, improve choice consistency, and decrease the dropout rate in discrete choice experiments Health Econ 2019 28 3 350 363 10.1002/hec.3846 30565338
Jonker MF, et al. Attribute level overlap (and color coding) can reduce task complexity, improve choice consistency, and decrease the dropout rate in discrete choice experiments. Health Econ. 2019;28(3):350–63.30565338 10.1002/hec.3846
46. Vass CM A picture is worth a thousand words: the role of survey training materials in stated-preference studies The Patient Patient-Cent Outcomes Res 2020 13 2 163 173 10.1007/s40271-019-00391-w
Vass CM, et al. A picture is worth a thousand words: the role of survey training materials in stated-preference studies. The Patient Patient-Cent Outcomes Res. 2020;13(2):163–73.10.1007/s40271-019-00391-w
47. Mühlbacher AC, de Bekker-Grob EW, Rivero-Arias O, et al. How to Present a decision object in health preference research: attributes and levels, the decision model, and the descriptive framework. Patient (2024). 10.1007/s40271-024-00673-y
48. ChoiceMetrics, Ngene 1.3 User manual and reference guide. The cutting edge in experimental design. 2021.
49. Morrell L What Aspects of Illness Influence Public Preferences for Healthcare Priority Setting? A Discrete Choice Experiment in the UK Pharmacoeconomics 2021 39 12 1443 1454 10.1007/s40273-021-01067-w 34409564
Morrell L, et al. What Aspects of Illness Influence Public Preferences for Healthcare Priority Setting? A Discrete Choice Experiment in the UK. Pharmacoeconomics. 2021;39(12):1443–54.34409564 10.1007/s40273-021-01067-w
50. StataCorp, Stata Statistical Software: Release 17, T.S.L. College Station, Editor. 2021.
51. Rose, J.M., R. Scarpa, and M.C.J. Bliemer, Incorporating model uncertainty into the generation of efficient stated choice experiments: A model averaging approach Institute of transport and logistics studies, 2009. Working Paper ITLS-WP-09-08
52. Buckell J Sindelar JL The impact of flavors, health risks, secondhand smoke and prices on young adults' cigarette and e-cigarette choices: a discrete choice experiment Addiction (Abingdon, England) 2019 114 8 1427 1435 10.1111/add.14610 30866132
Buckell J, Sindelar JL. The impact of flavors, health risks, secondhand smoke and prices on young adults’ cigarette and e-cigarette choices: a discrete choice experiment. Addiction (Abingdon, England). 2019;114(8):1427–35.30866132 10.1111/add.14610
53. Hurley J Mentzakis E Walli-Attaei M Inequality aversion in income, health, and income-related health J Health Econ 2020 70 102276 10.1016/j.jhealeco.2019.102276 31955864
Hurley J, Mentzakis E, Walli-Attaei M. Inequality aversion in income, health, and income-related health. J Health Econ. 2020;70: 102276.31955864 10.1016/j.jhealeco.2019.102276
54. Reed Johnson F Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force Value Health 2013 16 1 3 13 10.1016/j.jval.2012.08.2223 23337210
Reed Johnson F, et al. Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force. Value Health. 2013;16(1):3–13.23337210 10.1016/j.jval.2012.08.2223
55. De Bekker-Grob EW Sample size requirements for discrete-choice experiments in healthcare: a practical guide Patient 2015 8 5 373 384 10.1007/s40271-015-0118-z 25726010
De Bekker-Grob EW, et al. Sample size requirements for discrete-choice experiments in healthcare: a practical guide. Patient. 2015;8(5):373–84.25726010 10.1007/s40271-015-0118-z
56. Mott DJ Valuing EQ-5D-Y-3L health states using a discrete choice experiment: do adult and adolescent preferences differ? Med Decis Making 2021 41 5 584 596 10.1177/0272989X21999607 33733920
Mott DJ, et al. Valuing EQ-5D-Y-3L health states using a discrete choice experiment: do adult and adolescent preferences differ? Med Decis Making. 2021;41(5):584–96.33733920 10.1177/0272989X21999607
57. Train, K., Discrete Choice Methods with Simulation. Discrete Choice Methods with Simulation, 2009. 2nd Edition:1–388.
58. Hess S Conditional parameter estimates from Mixed Logit models: distributional assumptions and a free software tool J Choice Model 2010 3 2 134 152 10.1016/S1755-5345(13)70039-3
Hess S. Conditional parameter estimates from Mixed Logit models: distributional assumptions and a free software tool. J Choice Model. 2010;3(2):134–52.10.1016/S1755-5345(13)70039-3
59. Hess S Train KE Polak JW On the use of a Modified Latin Hypercube Sampling (MLHS) method in the estimation of a Mixed Logit Model for vehicle choice Transport Res Part B Methodol 2006 40 2 147 163 10.1016/j.trb.2004.10.005
Hess S, Train KE, Polak JW. On the use of a Modified Latin Hypercube Sampling (MLHS) method in the estimation of a Mixed Logit Model for vehicle choice. Transport Res Part B Methodol. 2006;40(2):147–63.10.1016/j.trb.2004.10.005
60. Gonzalez J A Guide to Measuring and Interpreting Attribute Importance The Patient Patient-Cent Outcomes Res 2019 12 1 9
Gonzalez J. A Guide to Measuring and Interpreting Attribute Importance. The Patient Patient-Cent Outcomes Res. 2019;12:1–9.
61. Hess S Palma D Apollo: a flexible, powerful and customisable freeware package for choice model estimation and application J Choice Model 2019 32 100170 10.1016/j.jocm.2019.100170
Hess S, Palma D. Apollo: a flexible, powerful and customisable freeware package for choice model estimation and application. J Choice Model. 2019;32: 100170.10.1016/j.jocm.2019.100170
62. Hess, S. and D. Palma, Apollo version 0.01. 2020.
63. RStudio Team, RStudio: Integrated Development for R. RStudio, PBC, Editor. 2020: Boston, MA.
64. ONS, Estimates of the population for the UK, England and Wales, Scotland and Northern Ireland, O.o.N. Satatistics, Editor. 2021.
65. ONS, Population estimates by ethnic group, England and Wales, O.f.N. Statistics, Editor. 2021.
66. Devlin NJ Valuing health-related quality of life: An EQ-5D-5L value set for England Health Econ 2018 27 1 7 22 10.1002/hec.3564 28833869
Devlin NJ, et al. Valuing health-related quality of life: An EQ-5D-5L value set for England. Health Econ. 2018;27(1):7–22.28833869 10.1002/hec.3564
67. EuroQoL Group. EuroQoL, http://www.euroqol.org/. 2022 24 June, 2022].
68. Jonker MF Roudijk B Maas M The sensitivity and specificity of repeated and dominant choice tasks in discrete choice experiments Value Health 2022 25 8 1381 1389 10.1016/j.jval.2022.01.015 35527163
Jonker MF, Roudijk B, Maas M. The sensitivity and specificity of repeated and dominant choice tasks in discrete choice experiments. Value Health. 2022;25(8):1381–9.35527163 10.1016/j.jval.2022.01.015
69. Hensher DA Rose JM Greene WH Applied Choice Analysis 2015 2 Cambridge Cambridge University Press
Hensher DA, Rose JM, Greene WH. Applied Choice Analysis. 2nd ed. Cambridge: Cambridge University Press; 2015.
70. European, C., H. Directorate-General for, and S. Food, Defining value in ‘value-based healthcare’ – Opinion by the Expert Panel on effective ways of investing in Health (EXPH). 2019: Publications Office.
71. Nicolet, A., et al. Patient and Public Preferences for Coordinated Care in Switzerland: Development of a Discrete Choice Experiment. The Patient - Patient-Centered Outcomes Research, 2022.
72. Hoedemakers, M., et al. Heterogeneity in preferences for outcomes of integrated care for persons with multiple chronic diseases: a latent class analysis of a discrete choice experiment. Quality of Life Research, 2022.
73. Department of Health and Social Care, Direct and Indirect Health Impacts of COVID-19 in England. 2021.
74. ONS, People with long-term health conditions, UK: January to December 2019. 2020, Office for National Statistics.
75. NIHR Evidence. Multiple long-term conditions (multimorbidity): making sense of the evidence. 2021 Mar 30, 2021 Mar 3]. Available at: https://evidence.nihr.ac.uk/collection/making-sense-of-the-evidence-multiple-long-term-conditions-multimorbidity/.
76. Rivero-Arias O Defining treatment success in children with surgical conditions Arch Dis Child 2024 109 5 377 386 10.1136/archdischild-2023-326156 38135491
Rivero-Arias O, et al. Defining treatment success in children with surgical conditions. Arch Dis Child. 2024;109(5):377–86.38135491 10.1136/archdischild-2023-326156
77. Ryan M Valuing patients' experiences of healthcare processes: towards broader applications of existing methods Soc Sci Med 2014 106 194 203 10.1016/j.socscimed.2014.01.013 24568844
Ryan M, et al. Valuing patients’ experiences of healthcare processes: towards broader applications of existing methods. Soc Sci Med. 2014;106:194–203.24568844 10.1016/j.socscimed.2014.01.013
78. NHS England. Patient experience [Internet]. Available at: https://www.england.nhs.uk/gp/patient-experience/. Accessed 3 Mar 2023
79. van de Wetering EJ Are some QALYs more equal than others? Eur J Health Econ 2016 17 2 117 127 10.1007/s10198-014-0657-6 25479937
van de Wetering EJ, et al. Are some QALYs more equal than others? Eur J Health Econ. 2016;17(2):117–27.25479937 10.1007/s10198-014-0657-6
80. Lancsar E The relative value of different QALY types J Health Econ 2020 70 102303 10.1016/j.jhealeco.2020.102303 32061405
Lancsar E, et al. The relative value of different QALY types. J Health Econ. 2020;70: 102303.32061405 10.1016/j.jhealeco.2020.102303
81. Paolucci F Equity and efficiency preferences of health policy makers in China—a stated preference analysis Health Policy Plan 2015 30 8 1059 1066 10.1093/heapol/czu123 25500745
Paolucci F, et al. Equity and efficiency preferences of health policy makers in China—a stated preference analysis. Health Policy Plan. 2015;30(8):1059–66.25500745 10.1093/heapol/czu123
82. Mentzakis E Equity and efficiency priorities within the Spanish health system: a discrete choice experiment eliciting stakeholders preferences Health Policy Technol 2019 8 1 30 41 10.1016/j.hlpt.2019.01.003
Mentzakis E, et al. Equity and efficiency priorities within the Spanish health system: a discrete choice experiment eliciting stakeholders preferences. Health Policy Technol. 2019;8(1):30–41.10.1016/j.hlpt.2019.01.003
83. Rawls J A Theory of Justice 1999 Oxford Oxford University Press
Rawls J. A Theory of Justice. Oxford: Oxford University Press; 1999.
84. Harari D, et al. Rising cost of living in the UK, in Commons Library Research Briefing. UK Parliament; 2023.
85. Marsh K Multiple criteria decision analysis for health care decision making-emerging good practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force Value in Health. 2016 19 2 125 137 10.1016/j.jval.2015.12.016 27021745
Marsh K, et al. Multiple criteria decision analysis for health care decision making-emerging good practices: Report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value in Health. 2016;19(2):125–37.27021745 10.1016/j.jval.2015.12.016
86. Gustavsson E, Lindblom L. Justification of principles for healthcare priority setting: the relevance and roles of empirical studies exploring public values J Med Ethics. Published Online First: 22 February 2023. 10.1136/jme-2022-108702
