
==== Front
J Comp Policy Anal
J Comp Policy Anal
Journal of Comparative Policy Analysis
1387-6988
1572-5448
Routledge

10.1080/13876988.2024.2362712
2362712
Version of Record
Research Article
Research Article
Comparative Measurement of Citizens’ Perceptions of Local Policies: A 3MC (Multinational, Multicultural, Multiregional) Approach
M. A. Yerkes et al.
Citizen’s perceptions of local policies
https://orcid.org/0000-0002-5480-4878
Yerkes Mara A. *
https://orcid.org/0000-0001-8160-556X
Meitinger Katharina **
https://orcid.org/0000-0002-3845-9867
van Hedel Karen *
* Department of Interdisciplinary Social Science, Utrecht University , Utrecht, The Netherlands
** Department of Methods and Statistics, Utrecht University , Utrecht, The Netherlands
Correspondence Address: Mara A. Yerkes, Department of Interdisciplinary Social Science, Utrecht University, Utrecht, The Netherlands. Email: M.A.Yerkes@uu.nl
12 7 2024
2024
12 7 2024
26 5 448465
Integra16 9 2024
Integra16 9 2024
11 7 2023
27 5 2024
© 2024 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group.
2024
The Author(s)
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. The terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent.

Abstract

Comparative social policy research continues to face considerable methodological challenges. Scholars particularly struggle to capture sub-national policy variation in comparatively reliable ways, including subjective perceptions of local policies. Local-level variation increases the need for conceptual and methodological attention to comparability across differing national, cultural, and linguistic settings. The article outlines a conceptually grounded approach relying on the 3MC (multinational, multicultural, multiregional) method to measure and compare individual perceptions of local policies cross-nationally. Often applied in cross-national survey research, a 3MC approach can help address methodological challenges inherent in comparative policy research and improve cross-national studies of local policy differences.

Keywords

3MC approach
comparative social policy
local policy
subjective policy perceptions
survey research
social sciences
European Research Council 10.13039/100010663 771290 This work was supported by the European Research Council under the European Union’s Horizon 2020 Research Innovation programme [Grant agreement No 771290].
==== Body
pmcIntroduction

For comparative policy scholars, significant methodological challenges arise when trying to maintain comparability in cross-national, cross-cultural, and linguistically different research settings. Conceptually, crucial cross-country differences exist in the policy instruments used to address social issues (Capano and Engeli 2022) as well as the levels at which these instruments are implemented (Ciccia and Javornik 2019). Empirically, an absence of comparable data at any level can limit cross-national comparisons (e.g. Clasen et al. 2016). Even when comparable data exist, establishing similarities between cases can be difficult (e.g. Locke and Thelen 1998; Stegmueller 2011; Peters and Fontaine 2020), often limiting the conclusions that can be drawn from comparative policy research (Geva-May et al. 2018).

These challenges of achieving comparability become even more complex when social policies and services are offered at lower levels of government, which produces additional inter- and intra-country variation across cities and municipalities (Ciccia and Javornik 2019; Satyro and Cunha 2019). Conceptually, this variation can lead to commensurate variation in what policy is understood to be (e.g. Lidström 1998; Steyvers 2019). Empirically, data availability becomes more problematic as sub-national data are often unavailable (Clasen et al. 2016), let alone comparable (Locke and Thelen 1998; Wolman 2008; Ciccia and Javornik 2019; Steyvers 2019; Peters and Fontaine 2020). We seek to advance comparative policy research by focusing on the local level, integrating insights from the social science comparative survey literature on issues of comparability (i.e. equivalence; see e.g. Locke and Thelen 1998; Stegmueller 2011). Our aim is to show how a 3MC (multinational, multicultural, multiregional) survey approach (see e.g. Johnson et al. 2019) can help address several comparative methodological challenges when measuring and comparing citizens’ subjective views of local policies and services. This explicitly comparative approach requires the collection of data from multiple populations with a design that deliberately recognizes cultural and linguistic diversity across regions. We further aim to contribute to comparative social policy research by opening up the “black box” (e.g. Damian et al. 2022) of the multiple steps taken to achieve this local-level comparability across countries.

Comparative Social Policy Research and the Turn towards the Local Level

The comparative method is “a way of thinking” aimed at drawing lessons, often through a comparison of macro-social elements (e.g. social policies) or causal relationships between policies and micro-level outcomes (Geva-May et al. 2018; Ferragina and Deeming 2023). Despite long-standing attention to local policy issues (see Lidström 1998), comparative social policy research at the sub-national level remains limited (see, for example, Lidström 1998; see also Kazepov 2010; Steyvers 2019), in part given the additional conceptual and methodological challenges outlined above (Wolman 2008; Ciccia and Javornik 2019). Compounding these challenges, greater attention is needed for the fact that individuals are not equally able to find, engage with, and use local policies as a resource (Yerkes et al. 2019; also Simonse et al. 2023). Indeed, Geva-May and colleagues (2018, p. 32) suggest in their study of 20 years of comparative policy research in this journal that more attention regarding how policies are subjectively perceived “is critical to an understanding of policy success and failure”.

Additional challenges arise in systematically comparing local policies and subjective perceptions of these policies (Lidström 1998; Wolman 2008; Steyvers 2019), alongside the limited availability of comparable data. A key conceptual issue is how to define local policy. Studies vary widely in their inclusion and exclusion of policy characteristics dependent upon the scope (i.e. breadth) and depth of the comparison (Lidström 1998; Wolman 2008; Steyvers 2019; Peters and Fontaine 2020; Kazepov et al. 2022). Often a delicate balance must be found between comparative scope and depth sufficient to adequately capture national, regional, or local governance structures, subsidiarity principles, and relationships between these levels, for example. Studying how citizens engage with or view these local policies comparatively requires even greater conceptual clarity to ensure the construct of “local policy” is similarly interpreted across localities, cultures, and languages. Scholars may also face methodological choices regarding explanatory factors in local policy uptake, demand or supply at multiple levels. Consequently, the contextualization of comparative research becomes even more salient (Locke and Thelen 1998; Hantrais 1999; Minas and Overbye 2010; Kazepov et al. 2022).

We suggest that a 3MC approach, widely applied in many comparative survey projects like the European Social Survey (ESS), can address two key methodological concerns when comparing local policies (Wolman 2008; Steyvers 2019; Peters and Fontaine 2020): construct bias (i.e. ensuring that what we are measuring exists in each country and is understood similarly across cultures; van de Vijver and Poortinga 1997), and item bias (i.e. the content of each survey item is equivalently understood across units of comparisons; He and van de Vijver 2012). A 3MC survey approach emphasizes and strives for a high level of comparability of data across countries, regions, and cultures (Johnson et al. 2019).

We applied a 3MC approach as part of a larger, cross-national, mixed methods research project, ERC-CAPABLE, which included graphical policy analysis, interviews, focus groups, and a cross-national survey. The CAPABLE project investigates gender inequality in work–life balance, with a focus on the role of community. Community was conceptualized in terms of local relationships (e.g. families, friends), locality (place, space, and scale; Yerkes et al. 2020), and local policies and services (e.g. child and elder care provision, schools), which we focus on here. Also central to the project is providing comparative insights into the interplay between national and local-level policies. The case localities and countries in this part of the project included Amsterdam and Nijmegen (the Netherlands), Ljubljana and Maribor (Slovenia), Barcelona and Pamplona (Spain), and London and Leeds (United Kingdom). The Netherlands, Slovenia, Spain, and the UK provide a rich comparison given shared decentralization trends, with nuanced cross-country variation (Martinelli et al. 2017; Jansen et al. 2021). Within this comparison, we acknowledge the geographical differences amongst our case countries, with significant territorial and policy differences in the UK and Spain, where, for example, the territorially organized system is complex and multi-level (Gallego and Subirats 2012; also Deusdad et al. 2017). Against this background, the varied distribution of responsibilities between national/federal, regional, and local governments can shape citizens’ perceptions of local policies. In this article, we detail the steps taken within a 3MC approach to develop comparable survey questions aimed at capturing individuals’ capabilities for work–life balance (e.g. Hobson 2014). That is, the real freedoms individuals have to live valued lives (e.g. Sen 1992) in relation to their perceived access to and use of local policies and services (hereafter PAULS).

Improving Comparative Survey Methods at the Local Level: The 3MC Approach

The 3MC approach and comparative social policy research have a shared aim of achieving equivalence (i.e. comparability) given the complexity of cross-national data (Locke and Thelen 1998; Harkness et al. 2010; Stegmueller 2011; Clasen et al. 2016; Ciccia and Javornik 2019; Steyvers 2019). Achieving equivalence in cross-national survey research means that “the range of interpretations associated with the evaluated items varies acceptably between cultural or language groups, given the survey measurement objectives” (Willis 2015, p. 363; emphasis added). Equivalence becomes even more salient when considering potential greater variation at the local level. We suggest that the 3MC approach, developed in the field of social science survey methods (Harkness et al. 2010; Johnson et al. 2019), can help address several conceptual and methodological problems encountered in comparative social policy research at the local level.

The 3MC approach focuses on achieving high levels of comparability by: (1) relying on an explicitly comparative design; (2) collecting data from multiple populations; and (3) deliberately designing survey instruments and other survey aspects to compare diverse populations (Harkness et al. 2010; Pennel et al. 2017). In short, 3MC survey research design proactively accounts for the fact that a study is researching the experiences and behaviours of respondents in multiple countries with different cultural backgrounds, whose first language may not be the same as the source questionnaire (Gray and Blake 2015). It therefore requires the explicit recognition of such differences in the early stages of research. Although comparative research designs underscore this need, much comparative research provides only limited acknowledgement of these differences in practice, particularly in cross-national survey research (Behr et al. 2021).

Equivalence problems can easily arise in cross-national research (Stegmueller 2011; Johnson et al. 2019) related to multiple forms of bias, that is, “nuisance factors that jeopardize the validity of instruments applied in different cultures” (He and van de Vijver 2012, p. 3). Three types of bias can threaten measure comparability: construct, method, and item bias. Construct bias occurs when the measured construct differs across cultures or might not exist at all in one or more of the countries being examined (van de Vijver and Poortinga 1997). For example, survey measures may be incomparable if respondents apply different interpretations or have different levels of underlying knowledge. Varying interpretation of questions can occur due to (poor) translation but can also be due to differing cultural backgrounds or variation in respondents’ use of answer scales (Johnson et al. 2019). Conceptualization also becomes paramount: a lack of construct overlap (i.e. differing conceptualizations) between the source and target questionnaire (i.e. translated versions) can increase construct bias if a construct in one or more countries fails to capture all relevant aspects as developed in the source questionnaire (Schoua-Glusberg and Villar 2014; Padilla et al. 2019). Method bias can appear due to distorting effects through differences in sampling, structural features of the instrument (e.g. differences in familiarity with stimulus material such as pictures or response formats such as multiple choice questions), administration processes (e.g. differences in the mode of data collection), or response styles (He and van de Vijver 2012). Lastly, item bias can appear due to poor item translation, ambiguous source items, or the inapplicability of item contents or connotations associated with the item wording in a given country context (He and van de Vijver 2012). Construct bias threatens conceptual or theoretical validity whereas method bias and item bias potentially lower measurement validity. The greater the bias, the larger the measurement error (i.e. the difference between the observed value and the “true” value) (Biemer et al. 2013). The goal of a thorough 3MC design is to minimize error and maximize equivalence (Smith 2019).

Despite potential barriers to applying a 3MC approach in practice (e.g. it requires a good deal of time, methodological expertise, and potentially greater financial investment), applying a 3MC approach in comparative research designs can help address existing data limitations (Ciccia and Javornik 2019; Steyvers 2019). It can do so by guiding the development of comparable sub-national survey measures across diverse populations, supported by pretesting for reducing item bias, while giving significant attention to linguistic differences through more extensive and ultimately accurate translation procedures (Willis 2015; Harkness et al. 2016; Behr 2017; Pennel et al. 2017). It also helps to increase transparency in comparative research by shedding light on the steps taken to achieve comparability through extensive documentation (Damian et al. 2019). Given space limitations, we focus on the steps taken to reduce construct and item bias in our study. We note, however, that significant attention was also given to reducing method bias (e.g. by applying the same sampling procedures, giving attention to potential cross-country differences in response styles when developing answer scales, etc.).

From Concept to Measurement: Comparatively Studying Local Policies and Services

Several steps were taken in the preparation and implementation of data collection (September 2019–September 2021) aimed at conceptualizing and measuring citizens’ perceived access to and use of local policies and services (PAULS) in our application of a 3MC approach. We offer full transparency (see e.g. Daniels et al. 2019), providing examples at each stage (from research design through data collection), which included interviews with local policymakers and organizations, qualitative insights from local citizens, the translation of an online (CAWI) survey questionnaire fielded in the autumn of 2021, integrated with multiple rounds of cross-cultural national and local expert review. We particularly focus on the comparative challenges and the lessons to be learned in conceptualizing and operationalizing citizens’ subjective view of local policies using 3MC principles. Given project timing, we discuss the impact of the COVID-19 pandemic where relevant.

Defining Local Policies: Conceptual and Empirical Issues

A first key challenge to address when incorporating local policies into comparative policy research is to conceptualize local policies, with multiple dimensions to include or exclude (e.g. Lidström 1998; Steyvers 2019). Conceptualization challenges can increase in larger-scale research projects that rely on a mixed methods design because of the need to investigate potentially differing perceptions of local policy (e.g. citizens, policymakers, and/or local stakeholders). Whereas a definition of local policy referring to subsidiarity principles or territorial dimensions may resonate with policymakers, citizens’ perceptions are likely to differ substantially from this governmental perspective. The policy domain under consideration also matters, with varying constellations of actors and responsibilities within and across domains (e.g. Kazepov 2010).

In our project, we faced the challenge of conceptualizing local policies potentially relevant to individuals’ capabilities for work–life balance. On paper, our research design envisioned assessing the availability, accessibility, and design of local work–life policies and resources by distributing 15–20 questionnaires in each city to academics, local stakeholders (e.g. municipality/city government), and local work–family service providers in each country. In practice, obtaining a relevant overview of policies relevant to work–life balance at the local level was not feasible given differences in geographical scale and territorial politics but also linguistic ambiguity about the term local policies, as noted by national and local cultural experts. Following the involvement of a comparative methods expert, we decided greater conceptual clarity was needed as well as a narrowing of the empirical scope of the envisioned research design to reduce the potential for construct bias.

The empirical scope was initially narrowed by planning for three focus groups in each city, as part of the development of a cross-national survey (discussed further below): young workers without care responsibilities, working parents, and working carers (consistently providing help or care for family members or loved ones).1 Moreover, to reduce variation across cities of diverse sizes, we decided to target specific neighbourhoods in each city, comparing two neighbourhoods within a given city district. Further discussions were held with national and local academic experts to determine the most representative district in each city and, within those districts, which neighbourhoods to focus on.

To gain greater conceptual clarity, we also adjusted the research design to interview a selection of local stakeholders (policymakers and civil society organizations representing the interests of the planned focus groups; n = 108 across all municipalities) rather than rely on questionnaires as originally planned. The information provided by the policy interviews and concurrent discussions with local and national experts would be used to develop questions for the focus groups and it also provided two lessons.

First, despite careful research design, capturing the meaning of local “policy” in practice remained difficult given cross-country and linguistic differences. To illustrate, during the initial interviews with policymakers in the municipality of Amsterdam, an open question was asked around the type of “resources” offered by the municipality to employed residents. The term resources was used purposely by the researchers to spark discussion around policymakers’ terminology. Although the focus of the study was not on how policymakers define local policies themselves, their descriptions matter for understanding how policies and services are implemented and provided at sub-national levels, and thus for how citizens might perceive and access these policies and services. Policymakers explained that they viewed “policy” as the overarching structure developed by the municipality. Resources, services, and facilities were what municipalities offered to residents as a result of policy. Discussions that followed with local and national experts in the other countries revealed similar linguistic and conceptual issues. For example, in Spain, local experts questioned whether equipamientos públicos were included in the definition of services, revealing linguistic nuances. In Spain, “services and programmes” differ from “public facilities” and the latter is not always provided by the national government. These discussions in Spain reflected the complex financing structure of policies and services across governmental levels (i.e. local (Ayuntamiento), autonomous community level (Administraciones autonómicas), and the national level (Administración General del Estado). Consequently, further discussions with the research team and experts led to us, in the remaining interviews, asking policymakers and stakeholders to reflect on the kinds of resources, services, and facilities offered to each group of workers, as well as what was needed for these groups.

Second, the interviews with policymakers and consultations with experts revealed that the initial narrowing of the empirical scope was reasoned too much from a comparative research design perspective, with insufficient attention to local realities in practice – realities that also differ across countries. From a design approach, we wanted to limit the amount of local variation, as in a most similar systems approach (Peters and Fontaine 2020). Yet interviews with local policymakers and stakeholders in countries with relatively small municipalities, like Slovenia and the Netherlands, suggested that the decision to focus on specific neighbourhoods needed to be reversed. As respondents in these countries pointed out, local policies and services either targeted an entire municipality, or particular areas in the city, which made comparisons difficult. In countries with larger cities and municipalities, however, like Spain and the UK, national experts pointed out difficulties in not focusing comparative efforts on particular neighbourhoods. In short, defining local policies in comparative research requires even greater attention to equivalence issues as well as contextualization (through research planning, interviews with stakeholders and national and local experts) than in national-level comparative studies. This contextualization is needed at an early stage of research in order to understand the geographical and political/policy scope when comparing local policies (Locke and Thelen 1998), as well as in the analysis stage, as acknowledged in the literature (Steyvers 2019).

Gaining Conceptual Clarity: Perceptions of Policies in Local Contexts

A second key challenge in comparative research on local policies is accounting for potential differences in subjective perceptions of policies and services at the local level, in our case citizens’ perceptions. Following our contextualization of varying understandings of local policies through the policy interviews and cross-cultural expert consultations, we aimed to further increase conceptual clarity for the survey using comparative focus groups, thereby reducing potential item bias. Indeed, the original research design included using comparative focus groups (i.e. moderated small group discussion used to demonstrate shared experiences; Campanelli 2008, p. 197) to get a better idea of citizens’ perceived access to and use of local policies and services (PAULS) in reconciling work–life. Focus groups are useful for informing survey questionnaire design and the subsequent interpretation of quantitative results (Krueger and Casey 2014; Caspar et al. 2016). The intention was to conduct 4–6 focus groups in each case country (2–3 in each city). Recruitment took place city-wide (rather than focused in particular areas) in early 2020 given an initial low response to recruitment efforts. In the rapidly changing environment of the COVID-19 pandemic in spring 2020, however, we had to cancel all planned face-to-face focus groups and switch to online, asynchronous focus groups (Daniels et al. 2019; Ramo et al. 2019) held on Microsoft Teams due to institutional restrictions and the need to meet General Data Protection Regulation (GDPR) privacy guidelines. Although online focus groups were not an ideal replacement for face-to-face focus groups, national lockdown restrictions left us little methodological choice. Moreover, asynchronous focus groups can lead to more detailed and reflective responses from respondents given the time lag between the researcher and the respondents’ online presence (Barbour and Morgan 2017). Ultimately, however, Teams proved unworkable for the focus groups, leading to the open questions being asked of respondents in the form of an online survey on the platform Qualtrics. Consequently, participants were not able to see each other’s answers or respond to them, as envisioned in the comparative methodology.2

Despite the problems encountered, from a comparative perspective the following open-ended question provided additional conceptual insights on citizens’ PAULS: “What local services and institutions, facilities or infrastructure would you need to divide your time and energy between employment and non-work activities? These services include, e.g., municipal, community or private services and institutions.” This question was purposely left open to allow for respondents’ perceptions of what they understood services, institutions, facilities, or infrastructures to be.

We received qualitative insights from 64 respondents across the four countries at this stage. Data were simultaneously analysed in Excel to compare the answers cross-nationally. Although not representative, qualitative answers to this and other open-ended questions help to illustrate how the local policy context (e.g. the availability and perceived necessity of certain policies or services) shapes citizens’ subjective perceptions and ultimately, potentially contributes to cross-national conceptual differences. For example, the only reference to the need for elder care services came from a respondent in Spain, where the availability of elder care services is generally more limited (e.g. Blanca Deusdad et al. 2017). Other examples include respondents saying they needed internet services, medical services, and libraries (all mentioned in Spain and Slovenia, not elsewhere). Nuanced differences were noted between countries. For example, although respondents in each country mentioned the need for childcare services, a Slovenian respondent referred specifically to the availability of “a functioning public kindergarten system” that was “of the utmost importance” to them (emphasis added). Two respondents in Spain and the Netherlands talked about the need for extracurricular activities for children, country contexts in which early school closing times and insufficient or expensive after-school care options create difficulties in reconciling work and care.

Triangulating the insights from the focus groups and cultural experts at national and local levels (which provided insights into the validity of the interview data), we were able to cluster focus group answers into broad groups of services and facilities. Cumulatively, we distinguished eight categories of local policies and services and grouped the remaining services under a ninth category of “other services” (see Table 1). These included sports activities, leisure activities, cultural activities (including libraries), childcare, formal elderly care, work and income, public transport, public health, and other services. This conceptualization was tested and further refined at later stages, detailed below.Table 1. Overview of local policies and services for Spain, the Netherlands, Slovenia, and the United Kingdom

 	 	Answer provided in:	
Category	Local policy or service	ES	NL	SI	UK	
1. Sports activities	Gym/sport centres

	x	x	x	 	
Yoga and Pilates studios

	 	 	 	x	
Family-inclusive sport facilities

	 	 	 	x	
2. Leisure activities	Green spaces

	 	 	 	x	
Nature reserves

	 	x	 	 	
Water (pond, river, canal)

	 	x	 	 	
Parks and roads

	 	x	x	x	
Family-inclusive parks

	 	 	 	x	
Beach/pool

	 	 	x	x	
Playgrounds/

	 	 	 	x	
Indoor playgrounds

	x	 	 	 	
Public spaces for recreation

	 	 	x	 	
Restaurants

	 	x	 	x	
Shops

	x	 	 	 	
Saunas

	 	 	x	 	
Other hospitality services (“horeca”,

	 	x	 	 	
“café”, “terras”)

	 	 	 	 	
3. Cultural activities	Theatre and the cinema

	 	x	 	x	
Cultural and community

	x	 	x	 	
centres/activities

	 	 	 	 	
Civic centre

	x	 	 	 	
Neighbourhood associations

	x	 	 	 	
Volunteering programme at the local

	 	 	 	x	
hospital

	 	 	 	 	
Course/training centres

	 	x	 	 	
Shared place for

	 	x	 	 	
studying/working/crafting

	 	 	 	 	
A (creative) workspace nearby

	 	x	 	 	
4. Childcare	Local childcare/

	 	x	x	 	
childcare, nursery

	 	x	 	x	
Universal access to childcare services/

	 	 	x	 	
good quality childcare

	 	 	 	 	
Childcare at their workplace and

	 	 	x	 	
organized by employer

	 	 	 	 	
Childcare occasionally at home

	 	 	x	 	
Extracurricular activities for the kids/

	x	 	 	 	
after school services/care	 	x	 	 	
School/day care closer to home

	x	 	 	 	
Playgroups

	 	 	 	x	
5. Formal elderly care	Assistance facilities for the elderly

	x	 	 	 	
More access to elderly care services

	 	 	x	 	
6. Work and income	Job closer to home/less commuting

	 	 	x	 	
Adequate furniture to work from home

	x	 	 	 	
Opportunity to work from home

	 	x	 	 	
7. Public transport	Public transportation

	x	x	x	x	
More efficient and faster public

	 	 	x	 	
transportation

	 	 	 	 	
Good public transport connection to be

	 	x	 	 	
able to travel fast from work to home

	 	 	 	 	
and the other way

	 	 	 	 	
8. Public health	Rehabilitation centres

	x	 	 	 	
Private and communitarian services

	 	 	 	x	
Healthcare facilities [local]

	 	 	x	 	
Medical services closer to home

	x	 	 	 	
9. Other services	Internet

	x	 	 	x	
Phone coverage

	x	 	 	 	
More online services to avoid

	 	 	x	 	
bureaucracy

	 	 	 	 	
Easy connection from school to work [by

	 	x	 	 	
bike or walking]

	 	 	 	 	
Other	Not aware of any, or did not use them

	 	 	 	x	
 	Don’t understand the question

	 	 	x	 	
 	None

	x	 	 	 	
 	Don’t need any local services/facilities

	 	x	 	 	
Note: x indicates the theme was mentioned by at least one focus group participant in the respective country

It is important to note that a small number of respondents reported no use of local services or had difficulty answering the open-ended question. For example, one respondent from the UK said: “I’m not sure I understand this question. Activities outside of work are walking the dog, and keeping fit. I do not have children or anyone to care for.” These responses pointed to the need for greater clarification in what local services are, a point we return to in the Discussion section. Additionally, the pandemic context meant that some respondents found it difficult to reflect on which services they needed or used. As one UK respondent noted: “Due to coronavirus I am working from home and dedicating most of my time to out of work activities within the home also. The only potential local services and facilities I use for outside of work activities is local parks and canalways for exercise.”

Operationalizing Local Policies

The policy interviews, cross-cultural expert consultations, and qualitative insights from local citizens were helpful for conceptualizing citizens’ PAULS. To operationalize the concept of PAULS for use in the cross-national survey questionnaire (e.g. to ensure questions would work as intended) (Campanelli 2008), as part of the 3MC approach, multiple rounds of consultation took place within the research team together with multiple rounds of external expert review. We wanted to go beyond existing survey measures capturing the use of local policies and services to measure what individuals value, what they think is possible (subjective opportunities) as well as what they manage to do in practice (objective opportunities; Brummel et al. 2023). In other words, individuals may differ in how they see and engage with local policies as well as their actual use of these policies (and to what effect). Building on work from Abma et al. (2016), we created multiple three-pronged capability measures tapping into (1) the importance of a given resource or activity (i.e. how much one values it; see Kurowska 2018); (2) perceptions of one’s opportunities to access these services or partake in given activities; and (3) the extent to which individuals actually manage to use these services or undertake a given activity. A three-pronged measure of local policies and services was one of the multiple capability measures developed. One category identified in the qualitative insights from local citizens, namely work and income, was excluded following expert review. We note that the order of the three-pronged measure for local policies and services was rearranged, first asking about the respondents’ use of local policies and services before asking about the importance they place on these local policies and services, and the opportunities they feel they have to use them. This decision was taken to avoid respondents overestimating the use of services based on what they feel is important to them.

The final test of this measure was through cross-cultural cognitive pre-testing of survey items (i.e. administering the questionnaire and asking interview respondents for additional verbal information about their responses to gauge understanding of the survey items and answer categories; Beatty 2004; Fitzgerald et al. 2009). Cognitive pre-testing is a crucial step in ensuring comparability (Fitzgerald et al. 2009; Willis 2015; Menold et al. 2023). When administered during the questionnaire development stage, issues of lack of equivalence can be revealed and avoided early on (Behr et al. 2016). Cross-cultural cognitive interviewing, when cognitive interviews are held in at least two cultural settings, helps to assess the cross-cultural equivalence of survey questions (Willis 2015) and can be used to identify poor source question design, translation problems, and the cultural portability of questions (Fitzgerald et al. 2011). Web probing, in combination with the cross-cultural expert interviews we relied on throughout the development of the survey, has been shown to offer the most consistent improvements to reliability and measurement invariance in cross-national survey measures (Menold et al. 2023).

Although the English-language questionnaire was only to be fielded in the UK, we held cognitive interviews in both the UK and the US to provide sufficient English-speaker input on the English questionnaire. Additional cognitive interviews were also held in the Netherlands using one of the translated questionnaires. National experts were then consulted again in all countries to discuss the cultural portability of the capability measures. A total of 11 cognitive interviews were held online, given ongoing pandemic restrictions in November 2020 (nine in English) and March 2021 (two in Dutch), with men and women living with and without a partner, with and without co-resident minor children, across all educational levels.

Using probing techniques, interview respondents were asked to read and respond to the questions out loud and were then probed about why they provided certain answers and/or how they interpreted and understood certain words and/or terms. We note two findings from these cognitive interviews that were crucial to our measurement of citizens’ PAULS. First, the capability questions on local policies and services were understandable but not ideal to test during the COVID-19 pandemic (given the closure of services in different settings, which varied across countries). Second, respondents felt that the second and third parts of the three-pronged measure needed to be shown on the same page in order to distinguish between subjective and objective opportunity questions. These adjustments were made before fielding the survey.

The Comparative Devil Is in the Details: Maintaining Comparability in Translation

A methodological discussion of the challenges in researching local policies and services in comparative perspective would not be complete without recognition of linguistic differences. The importance of translation is often underestimated in comparative policy research, in terms of both linguistic differences as well as how policy concepts are embedded in varying cultures and institutional structures (Béland and Petersen 2014). Much cross-national research continues to rely on back translations to detect translation errors (Behr 2017; Pennel et al. 2017). However, back translations have been shown to miss translation mistakes and can lead to misleading results (Harkness et al. 2010). As part of the 3MC approach to increase equivalence, rather than rely on back translations we combined a pre-translation translatability assessment with a team-based TRAPD translation approach: Translation, Review, Adjudication, Pretesting, and Documentation, to capture linguistic and cultural differences and improve equivalence (Harkness 2003; Harkness et al. 2010).

“Translation” involves creating a translatability assessment (i.e. a draft translation of a pre-final questionnaire), with the goal of identifying potential translation issues at an early stage and resolving them before the actual translation process starts (Behr et al. 2016). The questionnaire is ideally translated in a team approach (Harkness et al. 2010), whereby two translators create a parallel translation. This approach reduces the risk that the final translation is too strongly influenced by the translation style of a single translator, accounts for potential regional or local linguistic differences, and avoids translation mistakes as well as differences in the interpretation of key concepts (Behr et al. 2016). Parallel translations also have the advantage of providing translation options, revealing translation mistakes, and triggering a discussion about which version of a translation is better or whether it might be useful to combine elements of both translations in the final version. Such aspects are crucial in comparative policy research, where core concepts can often be vague and differ substantially across cultural and institutional settings (Béland and Petersen 2014), even locally.

At the review stage, a third person (the “reviewer”), becomes involved in the process and discusses the two draft translations together with the translators. The resulting review is a reconciled version of the parallel translations. At this adjudication stage, any final decisions regarding the translation are made and any issues that could not be solved during the review stage are fixed. For example, further consultation from methods or substantive experts can be necessary to clarify the meaning of a question or a specific key term. In a best-case scenario, the final translation of any questionnaire should be pretested (e.g. with cognitive interviews). Lastly, this translation process should be well documented (Harkness et al. 2010; Behr et al. 2016; Pennel et al. 2017).

Our application was a slightly modified TRAPD translation method, relying on two independent translators and a cultural expert as adjudicator (i.e. reviewer), but with additional consultation between translators, adjudicators, and the research team. This translation process revealed, for example, difficulty in translating the capability measures from English to Spanish. The subtle differences between subjective and objective opportunities evident in the English-source questionnaire came across in Spanish as questioning the respondents’ answers. The translation of the three-pronged measure on local policies and services highlighted language difficulties as well. In Spanish, discussion took place between the translators and adjudicators regarding variation between public or municipal services (Servicios municipals/Servicios públicos). In Dutch, the adjudication led to the inclusion of the word “diensten” as well as its synonym, amenities (voorzieningen), a more familiar term for “services”. In short, the initial three-pronged measures were deemed difficult to understand and translate in several contexts given linguistic differences and cultural variation in how translations of English terms would work in practice.

Discussion and Conclusion

Achieving comparability remains a methodological and empirical challenge in comparative social policy research, particularly at the sub-national level (Lidström 1998; Locke and Thelen 1998; Wolman 2008; Ciccia and Javornik 2019; Steyvers 2019). When researching subjective perceptions of policies, additional methodological challenges arise when researchers rely on cross-national survey data (Harkness et al. 2010). We outlined key challenges particular to cross-country comparative research of citizens’ perceived access to and use of local policies (PAULS) to show how the application of a 3MC approach may provide some distinct advantages. In doing so, we also aimed to increase transparency in comparative research by outlining the steps taken to improve equivalence (Daniels et al. 2019), which included gaining insights from cross-cultural experts, qualitative insights from local citizens, undertaking multiple stages of expert review, undertaking cognitive interviews, and the use of TRAPD translation procedures.

Our paper underscores the potential of the 3MC approach to improve equivalence (Willis 2015; Menold et al. 2023) and the comparability of policy studies more broadly, and particularly those assessing citizens’ subjective views of policies at the local level. By recognizing and accounting for cross-country differences, as well as the variation in respondents’ cultural and linguistic backgrounds (Gray and Blake 2015), the 3MC approach can help researchers achieve high levels of comparability among diverse populations (Harkness et al. 2010; Pennel et al. 2017). First, it provides guidance on how to develop measures and design questionnaires to achieve a high level of comparability (Harkness et al. 2016), crucial in comparative policy research at any level. Second, the 3MC approach underlines the importance of pre-testing to reduce item bias before data are collected (Willis 2015). Attention to item bias is particularly salient when considering subjective perceptions of policy, given the need to ensure survey items are equivalently understood across individuals. Third, the 3MC approach, and its TRAPD translation procedure in particular, provide more accurate translations and avoid errors more common in back translation methods (Behr 2017; Pennel et al. 2017). Poor translation can increase construct and item bias, with respondents consequently applying different interpretations of a poorly translated construct when answering the survey questions. In comparative social policy research, greater attention is needed regarding such linguistic and cultural differences (Béland and Petersen 2014).

We note two key implications in our application of the 3MC approach. Our study suggests that the use of cultural experts (academics and local policymakers and stakeholders) and insights from local citizens are a useful and necessary conceptual resource for defining and refining measures aiming at capturing citizens’ subjective perceptions of policies, which can help further research in this area (Geva-May et al. 2018). Additionally, our study suggests that important linguistic nuance can be gained (Béland and Petersen 2014), for example incorporating terms like resources, facilities, and services to correctly capture local variations in citizens’ PAULS. Although beyond the scope of this paper, in another paper in development, we empirically investigate the extent to which our application of the 3MC approach improved the equivalence of our survey data in relation to measurement invariance (i.e. cross-national comparability). A recent study of the effectiveness of piloting methods in improving equivalence suggests in any case that the use of cross-cultural experts, in combination with web probing and cognitive interviews, as in our study, can indeed increase data reliability and improve measurement invariance (Menold et al. 2023). It must be noted that these implications may reflect the specific combination of countries studied here. Future research in citizens’ PAULS in a broader set of countries, for example with greater variation in decentralization trends, would be useful in this regard.

Although the 3MC approach provides clear advantages and improves comparability, some comparative policy scholars may see such an approach as burdensome. Primarily, the approach requires a significant amount of time, which is often in short supply in research projects. It also requires sufficient methodological expertise within a research team and might be seen as overly costly, particularly in relation to translation. These issues, however, should be acknowledged and accepted at a very early stage in the research process, ideally at the design stage. This early acknowledgement would allow for the sufficient allocation of time and resources for the multiple procedures needed to improve comparability. Without such procedures, comparative researchers risk gathering and analysing cross-national data containing measurement errors (Behr et al. 2021).

We faced additional advantages and disadvantages related to the COVID-19 pandemic, a context which could affect comparative policy research in the future. In some situations, the pandemic and eventual delays caused by lockdowns can be an advantage. Delays provide researchers with more time, which in our case gave the research team additional time to develop the survey measures and to reflect on the translation of constructs. Moreover, as discussed above, several elements of the 3MC toolkit remain unaffected by pandemic contexts. For example, expert review, cultural experts, the translatability assessment, and the TRAPD approach are easily feasible in an online environment. As this study highlights, however, the availability of local policies and services can be severely impacted by the pandemic, which can result in construct bias if respondents in varying contexts interpret constructs differently. In our case, measuring citizens’ PAULS required careful consideration of national and local pandemic conditions. Conducting cross-national research at any time, and particularly during a pandemic, requires sufficient local and national expertise to make researchers aware of potential cultural and institutional differences that can detract from comparability. Despite the potential temporal and financial challenges of applying the 3MC approach to comparative social policy research, we suggest that this approach offers a promising way forward in improving comparability, particularly for researchers interested in subjective policy perceptions at the local level.

Acknowledgements

The authors would like to thank Hannelore van Hoffen, Roos Knopper and the rest of the ERC CAPABLE team for their research assistance as well as Jana Javornik, the participants of the 2021 European Social Policy Analysis network (ESPAnet) session on comparative methodology, and two anonymous reviewers for their comments and feedback on earlier versions of the manuscript. The research presented in this article was made possible through funding from the European Research Council (ERC) under the European Union’s Horizon2020 Research Innovation programme (Grant agreement No 771290).

Notes

Mara A. Yerkes is Professor of Comparative Social Policy in relation to Social Inequalities at Utrecht University, the Netherlands and Research Associate at the Center for Social Development in Africa at the University of Johannesburg, South Africa. Her research centres on comparative social policy, social inequalities related to work, care, communities and families, and their interplays.

Katharina Meitinger is an Assistant Professor at the Department of Methods & Statistics at Utrecht University. Her research focuses on comparability issues of cross-national survey data, the interplay of cognition and visual design of surveys, and novel approaches of data collection.

Karen van Hedel worked as a postdoctoral researcher on the ERC project ‘Enhancing Capabilities? Rethinking Work-life Policies and their Impact from a New Perspective’ (ERC CAPABLE) at Utrecht University. Her research interests include social disparities, population health and the interplay between work and life.

Disclosure Statement

No potential conflict of interest was reported by the authors.

1. In addition to these groups, a country-specific group was to be chosen in each country to provide flexibility in addressing national specificities. These included the self-employed in the Netherlands (with or without care responsibilities), the “sandwich generation” in Slovenia (employees with children and informal care responsibilities), lone parents in Spain, and individuals with an ethnically diverse background in the UK.

2. We also initially adjusted the broader research proposal, intending to now conduct the focus groups 3 times online over a period of 18 months. In this manner, we hoped to account for the potential influence of the pandemic and the expected waning of the pandemic. As we noted in the amended research proposal, the topic list for the first set of focus groups would centre on questions relating to work, family, and community activities, referring to work-life policies and resources available before the COVID-19 pandemic and perceptions of these resources during the pandemic. Later measurements would be used to assess perceptions of work-life policies and resources once COVID-19 measures had been relaxed. However, following the problems encountered with the proposed online focus groups, only one round of data collection took place.
==== Refs
References

Abma, F. I., Brouwer, S., de Vries, H. J., Arends, I., Robroek, S. J., Cuijpers, M. P., van der Wilt, G. J., Bültmann, U. and van der Klink, J. J., 2016, The capability set for work: Development and validation of a new questionnaire. Scandinavian Journal of Work, Environment & Health, 42 (1 ), pp. 34–42. doi:10.5271/sjweh.3532.
Barbour, R. S. and Morgan, D. L., 2017, A New Era in Focus Group Research (London: Palgrave Macmillan).
Beatty, P. C., 2004, The dynamics of cognitive interviewing, in: S. Presser M. P. Couper, J. T. Lessler, E. Martin, J. Martin, J. M. Rothgeb and E. Singer (Eds) Methods for Testing and Evaluating Survey Questionnaires (Hoboken, NJ: Wiley), pp. 45–66.
Behr, D., 2017, Assessing the use of back translation: The shortcomings of back translation as a quality testing method. International Journal of Social Research Methodology, 20 (6 ), pp. 573–584. doi:10.1080/13645579.2016.1252188.
Behr, D., Braun, M., and Dorer, B., 2016, Measurement instruments in cross-national surveys. GESIS Survey Guidelines. Mannheim. doi:10.15465/gesis-sg.
Behr, D., Burnett, J., Fitzgerald, R., Granda, P., Luz Guerrero, L., Gyuzalyan, H., Johnson, T., Kim, J., Mneimneh, Z., Moynihan, P., Robbins, M., Schoua-Glusberg, A., Sha, M., Smith, T. W., Stoop, I., Tomescu-Dubrow, I., Zavala-Rojas, D., and Zechmeister, E. J., 2021, AAPOR/WAPOR Task Force Report on Quality in Comparative Surveys (Ann Arbor, MI: American Association for Public Opinion Research and World Association for Public Opinion Research). Available at https://wapor.org/resources/aapor-wapor-task-force-report-on-quality-in-comparative-surveys/.
Béland, D. and Petersen, K., 2014, Analysing Social Policy Concepts and Language: Comparative and Transnational Perspectives (Bristol: Bristol University Press).
Biemer, P. P., Groves, R. M., Lyberg, L. E., Mathiowetz, N. A. and Sudman, S. (Eds), 2013, Measurement Errors in Surveys, Vol. 548 (Hoboken, NJ: John Wiley & Sons).
Brummel, A., Jansen, E., Yerkes, M. A., and Javornik, J., 2023, Potential for social innovation in social work: applying the capability approach, in: J. P. Wilken, A. Parpan-Blaser, S. Prosser, S. van der Pas, E. Jansen (Eds) Social Work and Social Innovation Emerging Trends and Challenges for Practice, Policy and Education in Europe (Bristol: Policy Press).
Campanelli, P., 2008, Testing survey questions, in: E. D. de Leeuw, J. Hox and D. Dillman (Eds) International Handbook of Survey Methodology, 1st ed. (Abingdon, Oxford: Routledge), pp. 176–200.
Capano, G. and Engeli, I., 2022, Using instrument typologies in comparative research: Conceptual and methodological trade-offs. Journal of Comparative Policy Analysis: Research and Practice, 24 (2 ), pp. 99–116. doi:10.1080/13876988.2020.1871297.
Caspar, R., Peytcheva, E., Yan, T., Lee, S., Liu, M., and Hu, M., 2016, Questionnaire Design. Guidelines for Best Practice in Cross-Cultural Surveys (Ann Arbor, MI: Survey Research Center, Institute for Social Research, University of Michigan).
Ciccia, R. and Javornik, J., 2019, Methodological challenges for comparative welfare state research: Capturing intra-country variation in cross-national analyses. Journal of Comparative Policy Analysis: Research and Practice, 21 (1 ), pp. 1–8. doi:10.1080/13876988.2018.1551598.
Clasen, J., Clegg, D., and Goerne, A., 2016, Comparative social policy analysis and active labour market policy: Putting quality before quantity. Journal of Social Policy, 45 (1 ), pp. 21–38. doi:10.1017/S0047279415000434.
Damian, E., Meuleman, B., and van Oorschot, W., 2022, Transparency and replication in cross-national survey research: Identification of problems and possible solutions. Sociological Methods & Research, 51 (2 ), pp. 499–526. doi:10.1177/0049124119882452.
Daniels, N., Gillen, P., Casson, K. and Wilson, I., 2019, STEER: Factors to consider when designing online focus groups using audiovisual technology in health research. International Journal of Qualitative Methods, 18 , pp. 1–11. doi:10.1177/1609406919885786.
Deusdad, B., Javornik, J., Giralt, R. M. and Marbán-Flores, R., 2017, Care in the wake of the financial crisis: Gender implications in Spain and the United Kingdom, in: F. Martinelli, A. Anttonen and M. Mätzke (Eds) Social Services Disrupted. Changes, Challenges and Policy Implications for Europe in Times of Austerity (Cheltenham, UK ; Northampton, MA: Edward Elgar), pp. 176–198. doi:10.4337/9781786432117.00017.
Deusdad, B., Lev, S., Pace, C. and Vella, S., 2017, Care for older people in three mediterranean countries: discourses, policies and realities of de-institutionalisation, in: F. Martinelli, A. Anttonen and M. Mätzke (Eds) Social Services Disrupted. Changes, Challenges and Policy Implications for Europe in Times of Austerity (Cheltenham, UK and Northampton, USA: Edward Elgar), pp. 259–280.
Ferragina, E. and Deeming, C., 2023, Comparative mainstreaming? Mapping the uses of the comparative method in social policy, sociology and political science since the 1970s. Journal of European Social Policy, 33 (1 ), pp. 132–147. doi:10.1177/09589287221128438.
Fitzgerald, R., Widdop, S., Gray, M. and Collins, D., 2009, Working paper series testing for equivalence using cross-national cognitive interviewing.
Fitzgerald, R., Widdop, S., Gray, M. and Collins, D., 2011, Identifying sources of error in cross-national questionnaires: Application of an error source typology to cognitive interview data. Journal of Official Statistics, 27 (4 ), pp. 569–599.
Gallego, R. and Subirats, J., 2012, Spanish and regional welfare systems: Policy innovation and multi-level governance. Regional and Federal Studies, 22 (3 ), pp. 269–288. doi:10.1080/13597566.2012.688271.
Geva-May, I., Hoffman, D. C., and Muhleisen, J., 2018, Twenty years of comparative policy analysis: A survey of the field and a discussion of topics and methods. Journal of Comparative Policy Analysis: Research and Practice, 20 (1 ), pp. 18–35. doi:10.1080/13876988.2017.1405618.
Gray, M. and Blake, M., 2015, Cross-national, cross-cultural and multilingual cognitive interviewing, in D. Collins (Ed.) Cognitive Interviewing Practice (London; Thousand Oaks, Calif: SAGE Publications Ltd), pp. 220–242. doi:10.4135/9781473910102.N11.
Hantrais, L., 1999, Contextualization in cross-national comparative. International Journal of Social Research Methodology, 2 (2 ), pp. 93–108. doi:10.1080/136455799295078.
Harkness, J. A., 2016, Questionnaire Design (Ann Arbor, MI).
Harkness, J. A., Braun, M., Edwards, B., Johnson, T. P., Lyberg, L., Mohler, P. P., Pennell, B. E. and Smith, T. W., 2010, Comparative survey methodology, in: J. A. Harkness, M. Braun, B. Edwards, T. P. Johnson, L. Lyberg, P. P. Mohler, B. E. Pennell and T. W. Smith (Eds) Survey Methods in Multicultural, Multinational, and Multiregional Contexts (Hoboken, NJ: Wiley), pp. 3–16.
Harkness, J. A., 2003, Questionnaire translation, in: A. H. Janet, F. J. R. van de Vijver and P. P. Mohler, (Eds) Cross-Cultural Survey Methods (Hoboken, NJ: Wiley), pp. 35–56. Available at https://research.tilburguniversity.edu/en/publications/questionnaire-design-in-comparative-research (Accessed 25 September 2021)
Harkness, J. A., Villar, A., and, and Edwards, B., 2010, Translation, adaptation, & design, in: J. A. Harkness (Ed) Survey Methods in Multicultural, Multinational, and Multiregional Contexts (Hoboken, NJ: Wiley), pp. 33–58.
He, J. and van de Vijver, F., 2012, Bias and equivalence in cross-cultural research. Online Readings in Psychology and Culture, 2 (2 ), pp. 1–19. doi:10.9707/2307-0919.1111.
Hobson, B., 2014, Worklife Balance. The Agency and Capabilities Gap (Oxford: Oxford University Press).
Jansen, E., Javornik, J., Brummel, A. and Yerkes, M. A., 2021, Central-local tensions in the decentralization of social policies: Street-level bureaucrats and social practices in the Netherlands. Social Policy & Administration, 55 (7 ), pp. 1262–1275. doi:10.1111/spol.12722.
Johnson, T. P., Pennell, B. E., Stoop, I. A. E. and Dorer, B., 2019, The promise and challenge of 3MC research, in: T. P. Johnson, B. E. Pennell, I. A. E. Stoop and B. Dorer (Eds) Advances in Comparative Survey Methods: Multinational, Multiregional and Multicultural Contexts (3MC) (Hoboken, NJ: John Wiley & Sons), pp. 3–12.
Kazepov, Y., Barberis, E., Cucca, R. and Mocca, E., 2022, Handbook on Urban Social Policies: International Perspectives on Multilevel Governance and Local Welfare (Y. Kazepov, E. Barberis, R. Cucca and E. Mocca Eds, Cheltenham: Edward Elgar).
Kazepov, Y., 2010, Rescaling Social Policies: Towards Multilevel Governance in Europe (Y. Kazepov Ed, Abingdon: Routledge).
Krueger, R. A. and Casey, M. A., 2014, Focus Groups. A Practical Guide for Applied Research (London; Thousand Oaks, Calif: Sage).
Kurowska, A., 2018, (De)familialization and (De)genderization - competing or complementary perspectives in comparative policy analysis? Social Policy & Administration, 52 (1 ), pp. 29–49. doi:10.1111/spol.12272.
Lidström, A., 1998, The comparative study of local government systems—a research agenda. Journal of Comparative Policy Analysis: Research and Practice, 1 (1 ), pp. 97–115. doi:10.1080/13876989808412617.
Locke, R. and Thelen, K., 1998, Problems of equivalence in comparative politics: Apples and oranges, again. APSA-CP, 9 (1 ), pp. 9–12.
Martinelli, F., Anttonen, A. A., and Mätzke, M. (Eds), 2017, Social Services Disrupted: Changes, Challenges and Policy Implications for Europe in Times of Austerity (Cheltenham, UK ; Northampton, MA: Edward Elgar).
Menold, N., Hadler, P., and Neuert, C., 2023, Improving cross-cultural comparability of measures on gender and age stereotypes by means of piloting methods. Advance, pp. 1–82. doi:10.31124/advance.21716483.v1.
Minas, R. and Overbye, E., 2010, The territorial organisation of social assistance in Europe, in: Y. Kazepov (Ed.) Rescaling Social Policies: Towards Multilevel Governance in Europe (Abingdon and New York: Ashgate Publishing), pp. 203–240.
Padilla, J. L., Benitez, I., and van de Vijver, F., 2019, Addressing equivalence and bias in cross‐ cultural survey research within a mixed methods framework, in: T. P. Johnson B. E. Pennell, I. A. Stoop and Dorer B. (Eds) Advances in Comparative Survey Methods: Multinational, Multiregional and Multicultural Contexts (3MC) (Hoboken, NJ: John Wiley & Sons), pp. 45–64.
Pennel, B. E., Hibben, K. C., Lyberg, L. E., Mohler, P.P. and Worku, G. 2017, A total survey error perspective on surveys in multinational, multiregional, and multicultural contexts, in: P. P. Biemer E. D. de Leeuw, S. Eckman, B. Edwards, F. Kreuter, L. E. Lyberg, N. C. Tucker and B. T. West (Eds) Total Survey Error in Practice (Hoboken, NJ: John Wiley & Sons), pp. 179–201.
Peters, B. G. and Fontaine, G., 2020, Methods and applications in comparative policy analysis, in: B. G. Peters and G. Fontaine (Eds) Handbook of Research Methods and Applications in Comparative Policy Analysis (Cheltenham: Edward Elgar), pp. 1–18.
Ramo, D. E., Meacham, M., Thrul, J., Belohlavek, A., Sarkar, U. and Humfleet, G., 2019, Exploring identities and preferences for intervention among LGBTQ+ young adult smokers through online focus groups. Journal of Adolescent Health, 64 (3 ), pp. 390–397. doi:10.1016/j.jadohealth.2018.09.022.
Satyro, N. G. D. and Cunha, P. S., 2019, The coexistence of different welfare regimes in the same country: A comparative analysis of the Brazilian municipalities heterogeneity. Journal of Comparative Policy Analysis: Research and Practice, 21 (1 ), pp. 65–89. doi:10.1080/13876988.2018.1484211.
Schoua-Glusberg, A. and Villar, A., 2014, Assessing translated questions via cognitive testing, in: K. Miller, V. Chepp, S. Willson, J. L. Padilla (Eds) Cognitive Interviewing Methodology (Hoboken, NJ: John Wiley & Sons), pp. 51–67.
Sen, A., 1992, Inequality Reexamined (Cambridge, Mass: Harvard University Press).
Simonse, O., Vanderveen, G., van Dillen, L. F., Van Dijk, W. W. and van Dijk, E., 2023, Social security or insecurity? The experience of welfare participation by financially vulnerable households in the Netherlands. Social Policy & Administration, 57 (3 ), pp. 255–271. doi:10.1111/spol.12849.
Smith, T. W., 2019, Improving multinational, multiregional, and multicultural (3MC) comparability using the total survey error (TSE) paradigm, in: T. P. Johnson, B. E. Pennell, I. A. Stoop and B. Dorer (Eds) Advances in Comparative Survey Methods: Multinational, Multiregional and Multicultural Contexts (3MC) (Hoboken, NJ: John Wiley & Sons, Incorporated), pp. 13–43.
Stegmueller, D., 2011, Apples and oranges? The problem of equivalence in comparative research. Political Analysis, 19 (4 ), pp. 471–487. doi:10.1093/pan/mpr028.
Steyvers, K., 2019, Doing comparative research on local politics: At a crossroads between inertia and innovation. Croatian and Comparative Public Administration, 19 (1 ), pp. 53–78. doi:10.31297/hkju.19.1.3.
van de Vijver, F. and Poortinga, Y., 1997, Towards an integrated analysis of bias in cross-cultural assessment. European Journal of Psychological Assessment, 13 (1 ), pp. 29–37. doi:10.1027/1015-5759.13.1.29.
Willis, G. B., 2015, Research synthesis: The practice of cross-cultural cognitive interviewing. Public Opinion Quarterly, 79 (S1 ), pp. 359–395. doi:10.1093/poq/nfu092.
Wolman, H., 2008, Comparing local goverment systems across countries: Conceptual and methodological challenges to building a field of comparative local government studies. Environment and Planning C: Government and Policy, 26 (1 ), pp. 87–103. doi:10.1068/cav5.
Yerkes, M. A., Hoogenboom, M., and Javornik, J., 2020, Where’s the community in community, work and family? A community-based capabilities approach. Community, Work and Family, 23 (5 ), pp. 516–533. doi:10.1080/13668803.2020.1818547.
Yerkes, M. A., Javornik, J., and Kurowska, A. (Eds), 2019, Social Policy and the Capability Approach. Concepts, Measurement and Application (Bristol: Policy Press).
