
==== Front
Health Promot Int
Health Promot Int
heapro
Health Promotion International
0957-4824
1460-2245
Oxford University Press US

10.1093/heapro/daae111
daae111
Article
Health Promoting Environments
AcademicSubjects/MED00860
Real-world public health interventions demonstrate how research evidence informs program scale-up
https://orcid.org/0000-0002-3058-2211
Crane Melanie Sydney School of Public Health, The University of Sydney, New South Wales 2006, Australia
Charles Perkins Centre, The University of Sydney, New South Wales 2006, Australia

https://orcid.org/0000-0003-2453-2207
Lee Karen Sydney School of Public Health, The University of Sydney, New South Wales 2006, Australia
Charles Perkins Centre, The University of Sydney, New South Wales 2006, Australia

Wolfenden Luke School of Medicine and Public Health, The University of Newcastle, Callaghan, New South Wales 2308, Australia

https://orcid.org/0000-0003-2460-5031
Phongsavan Philayrath Sydney School of Public Health, The University of Sydney, New South Wales 2006, Australia
Charles Perkins Centre, The University of Sydney, New South Wales 2006, Australia

https://orcid.org/0000-0002-0369-4621
Bauman Adrian Sydney School of Public Health, The University of Sydney, New South Wales 2006, Australia
Charles Perkins Centre, The University of Sydney, New South Wales 2006, Australia

Corresponding author. E-mail: melanie.crane@sydney.edu.au
10 2024
23 9 2024
23 9 2024
39 5 daae111© The Author(s) 2024. Published by Oxford University Press.
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (https://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact journals.permissions@oup.com

Abstract

Evidence-based population interventions rely on intervention testing (efficacy and effectiveness trials) to determine what works to improve public health. We investigated the characteristics of real-world public health interventions to address obesity and explored the extent to which research testing was undertaken prior to scale-up. We identified 90 population health interventions targeting physical activity, nutrition or obesity-related health behaviours and collected publicly available information on their key characteristics and outcomes. We then assessed the differences between interventions that followed a research pathway and those that did not. Two-thirds (n = 60) of the interventions were reported as having followed a research pathway. Univariate logistic regression analysis revealed that these interventions were more likely to be health education interventions [odds ratio (OR): 5.56; 95% confidence interval (CI): 1.38–22.38], developed by research institutes (OR: 12.81; 95% CI: 3.47–47.34), delivered in North America (OR: 4.13; 95% CI: 1.61–10.62), and less likely to be owned (OR: 0.35; 95% CI: 0.14–0.88) or funded by government organizations (OR: 0.37; 95% CI: 0.14–0.95). Interventions that followed a research pathway were nearly three times more likely to have a positive impact on population health (OR: 2.72; 95% CI: 1.04–7.14). Interventions that followed a research pathway to scale-up were no more likely to be sustained longer than those that did not. Differences exist across real-world interventions between those that follow a research pathway to population-scale delivery and those that do not, regarding organizational and environmental context. A key benefit of research pathway to scale-up is the impact it has on health outcomes.

evidence-based interventions
scale-up
effectiveness
research
health promotion
dissemination
The Australian Prevention Partnership Centre GNT9100001 ACT Health 10.13039/501100001139 SA Health Tasmania Health Prevention Research Support Program
==== Body
pmcContribution to Health Promotion

We examine the characteristics of real-world interventions to explore what is driving scale-up in practice.

We identify that only two-thirds of obesity prevention interventions use a research pathway for population scale-up by testing intervention effectiveness.

Interventions that follow a research pathway are more likely to be university driven, education focused, and are not owned or funded by government organizations.

These interventions are nearly three times as likely to have beneficial outcomes on population health.

INTRODUCTION

Obesity remains a complex population health challenge for public health, with a quarter of the world’s population expected to have obesity in the coming decade (World Obesity Federation, 2023). Based on existing prevalence and trend data, the impact of obesity and overweight is predicted to reach $4.32 trillion annually by 2035 if prevention and management efforts do not improve (World Obesity Federation, 2023). Tackling obesity requires a population-level approach that targets the conditions and environments influencing the key drivers of unhealthy weight gain from physical inactivity and poor diet (World Health Organization, 2013; Rutter, 2018). This population approach necessitates the implementation of a range of interventions delivered across multiple settings, environments and sectors (World Health Organization, 2013). However, large-scale dissemination of interventions across such complex and varied contexts makes generating evidence of ‘what works’ at the population level more difficult to establish.

Determining the efficacy and effectiveness of interventions is a cornerstone of evidence-based practice. The use of ‘evidence-based’ interventions that have undergone sufficient scientific evaluation to be deemed safe and effective is important for the efficient allocation of scarce health resources and is more likely to produce community-level outcomes. As such, when implemented at the population level, it is recommended that interventions go through a series of evidence-generating stages or pathways prior to their ‘scale-up’—that is, moving from efficacy testing to real-world effectiveness trials, prior to large-scale dissemination (Indig et al., 2018). However, not all population-level interventions follow this pathway. One study reported that only half of population health interventions to prevent chronic diseases followed a ‘comprehensive research pathway’ (Indig et al., 2018). The authors found that 15% of interventions investigated were implemented at a population level without any prior research evidence derived from prior efficacy or effectiveness testing (Indig et al., 2018).

Undertaking rigorous assessments of intervention effects does not always guarantee translation and dissemination. Studies of public health intervention trials included in Cochrane reviews report that only one-third were subsequently scaled up in practice (Wolfenden et al., 2022; McCrabb et al., 2023). Surprisingly, decisions to scale-up did not depend on whether the intervention was found to be ‘effective’, whether the risk of bias in trial evaluations was large, or how much involvement end-users had in the research (Wolfenden et al., 2022). Several reasons for scaling up interventions without prior evidence of effectiveness include: limited time; resources and training of population health practitioners (Glasgow et al., 2003; Brownson et al., 2015); organizational and political pressures including the urgency of more immediate public health problems or policy (Kingdon and Stano, 1984); and a disconnect between researchers and practitioners (Brownson et al., 2018).

The routine delivery of obesity prevention initiatives that have not undergone prior testing has the potential to undermine efforts to address the global burden of obesity. However, the extent to which research testing prior to scale-up occurs, its impacts on weight-related behaviours and program sustainability are not known. This study aimed to address these questions. To do so, we examine real-world interventions, investigate their characteristics and assess differences between interventions which had utilized a research pathway and those that did not, so that we can improve research and evaluation agendas.

METHODOLOGY

Identification of interventions

We conducted a systematic search for interventions to identify published obesity prevention interventions delivered across different population settings. This process was not intended as a systematic review, nor do our results aim to be presented as a systematic review. Rather, it is a structured method for identifying a sample of real-world interventions. We searched three databases (Scopus, Embase and Medline) for papers published between January 2015 and January 2020. The search extended an earlier literature search for interventions from January 1990 to December 2014 (Indig et al., 2018). The full search strategy is summarized in Table 1 (Supplementary File for further details). For inclusion, studies needed to be in English language, should refer to population-based interventions implemented in ‘at least city or regional delivery’, and should target physical activity, nutrition or obesity prevention (PANO). Studies needed to mention the name of an intervention in the abstract or title but did not need to report on the evaluation impact or outcomes as our intention was to gather the names of interventions only. Protocol papers after 2018 were excluded, given they were likely to still be in research development stage. Further information on this process is presented elsewhere (Crane et al., 2024).

Table 1: Search criteria

Databases used	Scopus, OVID (Embase, Medline)	
Search inclusion terms	Area of focus (health promotion OR public health OR health prevention OR preventative health OR population health)
AND
Prevention (physical activity OR exercise OR sedentary behavior OR obesity OR diet OR nutrition OR diabetes OR cardiovascular disease OR noncommunicable disease OR chronic disease OR overweight)
AND
Activity (scalability OR scale up OR at scale OR translational research OR dissemination OR implementation OR adoption OR sustain OR diffusion OR institutionalisation)
AND
Intervention (policy OR program OR intervention OR innovation)	
Search exclusion terms	Cancer, HIV, maternal and child health, developing countries, secondary prevention interventions (where program enrolment required diagnosis of a disease or known at-risk factors/biomarkers)	

From the122 interventions identified across 159 papers, 32 were excluded for the following reasons: insufficient information for data collection (i.e. website no longer active and/or limited published information that could be extracted to populate data) (n = 10); still in research development/never implemented or delivered in a definable population setting (n = 17); not a primary prevention intervention (n = 11) or not PANO (n = 6); a ‘static resource’ (e.g. a training manual) with no identifiable delivery mechanism (n = 6); a grants program (n = 4); or referred to an organization with no tangible intervention or delivery method (n = 5). Umbrella strategies that included multiple interventions (n = 5) or a modification of an intervention already included (n = 4) were also excluded to reduce overlap. The remaining interventions were used for the basis of the study.

Data collection

We used the Google search engine to identify a primary website (in English) for each intervention selected. All primary intervention websites were identified within the first three pages (15 web search ‘hits’). The website was used as the primary source of information. Additional information was gathered from (i) the scientific literature on the programs (from the initial literature search); (ii) a list of scientific papers reported on the intervention’s website; and (iii) a targeted title search in Google Scholar. In Google Scholar, the first three results pages were searched using the program name and search terms such as ‘prevention’ ‘physical activity’ OR ‘nutrition’ OR ‘obesity’ and ‘program’. This was sufficient to exhaust the list of relevant research papers. No date limit was placed on the Google search up until December 2021.

For each intervention, we collected information on characteristics thought to influence decisions to scale up the programs. Specifically, we used the domains of the Consolidated Framework for Implementation Research (CFIR) as a framework for data collection (Damschroder et al., 2009, 2022). This included information on the characteristics of the intervention, the inner setting—including provider characteristics and the deliverer (adopters of the innovation and partners), the population (end-users), the implementation process and outer setting. Within each domain, we included a list of characteristics which could be determined from publicly available sources (Table 2). We collated evidence of efficacy and effectiveness testing prior to population scale-up to determine the research pathway and recorded the year in which the research was conducted and year of scale-up. Efficacy testing was defined as intervention trials conducted under ‘ideal’ research conditions, and effectiveness testing as under real-world conditions. Intervention characteristics and outcomes were determined from published scientific papers and evaluation reports. We recorded whether the intervention had undertaken both efficacy testing and real-world effectiveness testing, partial testing (i.e. either efficacy or effectiveness testing) or no testing prior to scale-up.

Table 2: Determinants of implementation success based on the CFIR

Variable	Description	Categories	
Intervention characteristics		
Intervention type	Interventions differentiated by target, action and method/technique using the World Health Organization’s ‘International Classification of Health Interventions’ (ICHI)—see ICHI for intervention definition.a	Health education programs; behaviour change skills programs; information provision; products/incentives; capacity building/advocacy; multicomponent	
Intervention components	The components of the design/resource which provides an indication of the intervention’s complexity.	Interactive digital (e.g. tracking app); interactive face/phone (e.g. counselling); static print resource (e.g. factsheet); static digital resource (e.g. video); physical product/structure (e.g. Physical Activity equipment); educational curricula; multiple types	
Program duration	The length of time the program takes to be delivered.	<6 months; 6–12 months; >1 year; ongoing/no definitive period	
Population characteristics		
Target population	The primary target group for the intervention.	Children (0–12 years) and youth (13–18 years); adults, older adults, general population	
Organization context		
Provider characteristics	The organization, individual or group who facilitate the program across delivery organizations and settings as an indicator of implementation climate and organizational culture.	Government; not-for-profit; university/research institute; private/commercial; multiple organizations.	
Conceptual development	The group or individuals who developed the intervention concept (these organizations may differ from the implementing organization, funder or deliverer) as an indicator of the source of the intervention as internally or externally developed.	Researchers/universities; policy/government organizations; practice/practitioners; co-production; non-government organizations	
Partnerships	Coalitions/collaborations/networks of stakeholders engaged in the program implementation.	Yes—includes university; private/commercial; community groups; professional groups; government; individuals; multiple partners	
Delivery characteristics		
Deliver characteristics	The organizations responsible for delivering the intervention to the user population and level of influence.	Community organizations (schools, childcare centres, community clubs, workplaces and individuals); and supra community-level organizations (government organizations, health practices, commercial organizations and across multiple community settings)	
Delivery setting	The place where the intervention was implemented to reach the target population.	Community settings (homes, recreational facilities; religious places; community centres); institutions (education settings, healthcare, workplaces)	
Implementation process		
Delivery structure	Program delivery and decision making is centralized (i.e. delivered by a central team of experts or a set curricula) or decentralized structure (delivery organizations can choose or adapt the intervention components delivered).	Centralized; decentralized delivery	
Dissemination mode	The mode by which the content of the intervention is delivered.	Face-to-face; other modes [telephone, digital media, print media, environmental structures, somatic, pull, or push, or multi-mode (where no main mode could be defined)]	
Dissemination strategy	Dissemination resources used as an indication of potential resourcing/communication needs.	Individuals; group; population	
Outer setting		
Country/region	Country is used as a proxy for policy, political-cultural and health system context.	USA/Canada; Europe/UK;
Australia/New Zealand; other countries.	
Funding	The type of organization listed as financing the intervention.	Government; commercial/user pay; non-government/not-for-profit	
Community involvement	Mention of community engagement and participation an indication of community support.	Community involvement; no community involvement	
Outcomes		
Population impact	A statistically significant measure of the primary outcomes (single and multiple outcomes) as evidence of successful population impact of the intervention.	Positive impact on primary outcome(s) measured; negative impact on primary outcome(s); mixed effect on primary outcome(s)	
Sustainment	The extent (years) the intervention has been delivered as evidence of ongoing dissemination sustainment.	Number of years sustained—higher than or less than the average (15 years)	
Implementation impact	The type of impacts reported refer to one of three constituents according to CFIR addendum.b	End-users, the program was intended to benefit; program deliverers (as those who are delivering the program to end-users); decision makers (if the evaluation provided high-level information only)	
a World Health Organization (2021).

bDamschroder et al. (2022).

We also collected evaluation evidence of program impact and outcomes. We recorded the primary outcomes evaluated and the period of evaluation. Outcome variables included information on (i) the population impact of the intervention; (ii) the intervention’s sustainment; and (iii) the implementation impact—as described by the CFIR outcomes framework (Damschroder et al., 2022). Population impact was measured by quantitative evidence of the intervention on a primary behavioural outcome (and its effect reported as a statistically significant effect at P < 0.05). Program sustainment was defined as ‘the extent the intervention has been delivered over the long-term’ (Damschroder et al., 2022). This was measured as the number of years the intervention has been delivered since scale-up. The concept ‘Implementation impact’ was defined by the type of impact reported by the available evaluation evidence. The CFIR outcomes framework suggests that an intervention will produce benefits for three key groups of constituents: (i) the end-users as intervention recipients (in this case the target population); (ii) program deliverers (those involved in delivering the intervention either as the provider organization or third party); and (iii) decision makers (Damschroder et al., 2022). We noted which group the available process and impact evaluation evidence focused on as an indicator of the evaluation purpose.

Ethics approval was not applicable for this research as all information was collected from the public domain.

Data analysis

The quantitative data were entered in Excel (M.C., K.L. and research assistants) and independently checked for consistency (M.C. and K.L.). Discrepancies in coding (7.6% of data entered) were discussed and agreement reached. For simple descriptive analysis, the data were imported into Stata version 14 (StataCorp LP, College Station, TX).

We defined a ‘research pathway’ as published evidence of efficacy and/or effectiveness testing identified prior to delivery at scale. ‘Non-research pathways’ were those delivered at scale but with no evidence of intervention testing. We then described the differences between interventions that followed a research pathway and those that were delivered at scale (the non-research pathway) across key intervention characteristics and reported program outcomes. We used chi-squared test of dependence to compare differences in nominal variables and t-test statistics with unequal variance to compare interval (year) variables. We then used univariate logistic regression analysis to examine the size of effect of these differences. Small cell counts of determinant variables prevented adjusted regression modelling.

RESULTS

Overview of interventions

In total, 90 interventions were selected for inclusion. The oldest intervention was delivered at scale in 1988, and the most recent in 2017 (Figure 1). Most interventions were scaled up between 2005 and 2012. The length of time the interventions had been operating was on average 14.6 years (SD 6.6 years). Two-thirds of the interventions followed a research pathway prior to scale-up (n = 60), and more than half of these included both effectiveness and efficacy research testing (n = 37); for the remaining one-third, we found no research pathway prior to dissemination at scale (n = 30). Of the interventions that followed a research pathway, one-third (33.3%) had been implemented in the last decade (after 2012), compared with 16.7% of interventions in the non-research pathway in the same period. However, differences were not statistically significant in terms of when the interventions were scaled up [t(65.9) = −0.893, P = 0.353] or the length of time they had been delivered [t(75.6) = −1.363, P = 0.177].

Figure 1: Obesity prevention interventions at-scale by year implemented and pathway (n = 90).

Characteristics associated with the research pathway to scale-up

Interventions that had undertaken a research pathway to population scale-up differed from those that did not on several key implementation determinants (Table 3). Univariate logistic regression analysis revealed differences in intervention characteristics, with interventions following a research pathway being five times more likely to include health education interventions compared with those in the non-research pathway (P = 0.028). Differences in provider characteristics indicated that the interventions following a research pathway were at least 12 times more likely to be developed by a university/research group rather than by a government or other organizations (P < 0.0001). For interventions that followed the research pathway, the provider organization was less likely to be a government organization (P = 0.025). In terms of the ‘Outer setting’ characteristics, geographically, interventions that followed a research pathway were four times more likely to originate from North America compared with interventions from other countries (primarily Australia/New Zealand and Europe). These other geographical locations were more likely to take the non-research pathway (P = 0.003). Differences were also observed in terms of funding sources, with government-funded interventions less likely to follow a research pathway (P = 0.038). There were no significant differences between research pathways in terms of delivery and process characteristics.

Table 3: Comparison of intervention determinants by implementation pathway and outcome

Implementation determinant/outcome	Research pathway (n = 60)	Non-research pathway
(n = 30)	Unadjusted
OR (95% CI)	P-value	
Innovation characteristics	
Intervention type	Health education type	20 (33.3%)	3 (10.0%)	5.56 (1.38–22.38)	0.028	
Behavioural skills	22 (36.7%)	15 (50.0%)	1.52 (0.57–4.08)	
Othera	18 (30.0%)	12 (40.0%)		
Intervention components	Interactive	25 (41.7%)	16 (53.3%)	0.79 (0.51–1.23)	0.296	
Static/physical product	35 (58.3%)	14 (46.7%)	
Duration	6 months or less	32 (55.2%)	19 (63.3%)	1.40 (0.57–3.47)	0.463	
More than 6 months	26 (44.8%)	11 (36.7%)	
Population characteristics	
Target population	Children and youth	33 (55.0%)	11 (36.7%)	2.11 (0.85–5.19)	0.104	
Adults and general population	27 (45.0%)	19 (63.3%)	
Organizational characteristics	
Provider characteristics	Government organization	19 (31.7%)	17 (56.7%)	0.35 (0.14–0.88)	0.025	
Non-government organization	41 (68.3%)	13 (43.3%)	
Conceptual developmentb	University-led	34 (59.7%)	3 (10.3%)	12.81 (3.47–47.34)	<0.0001	
Government/NGO	23 (40.3%)	26 (89.7%)	
Partnerships	Partnerships	37 (61.7%)	23 (76.7%)	0.49 (0.18–1.32)	0.159	
No partnerships	23 (38.3%)	7 (23.3%)	
Delivery and implementation process	
Deliver characteristicsc	Community organizations	38 (63.3%)	16 (53.3%)	1.51 (0.62–3.68)	0.362	
Supra community organizations	23 (36.7%)	14 (46.7%)	
Delivery settingd	Community centres	24 (40.0%)	17 (56.7%)	1.96 (0.81–4.77)	0.137	
Institutions (schools, workplaces, healthcare)	36 (60.0%)	13 (43.3%)	
Delivery structure	Centralized	39 (65.0%)	14 (46.7%)	0.47 (0.19–1.15)	0.098	
De-centralized	21 (35.0%)	16 (53.3%)	
Dissemination mode	Face-to-face	35 (58.3%)	12(40.0%)	2.10 (0.86–5.13)	0.103	
Other modes	25 (41.7%)	18 (60.0%)	
Dissemination strategy	Individual	13 (21.7%)	5 (16.7%)	2.47 (0.85–7.14)	0.213	
Group	37 (61.7%)	15 (50.0%)	2.6 (0.67–10.06)	
Population	10 (22.6%)	10 (33.3%)		
Outer setting	
Country/region of delivery	USA/Canada	47 (78.3%)	14 (46.7%)	4.13 (1.61–10.62)	0.003	
Other countries	13 (21.7%)	16 (53.3%)	
Funding	Government	15 (28.3%)	15 (51.7%)	0.37 (0.14–0.95)	0.038	
NGO/commercial	38 (71.7%)	14 (48.3%)	
Community involvement	Community involved	23 (38.3%)	16 (53.3%)	0.74 (0.47–1.15)	0.178	
No community involved	37 (61.7%)	14 (46.7%)	
Outcomes						
Population impact	Positive	29 (50.9%)	8 (27.6%)	2.72 (1.04–7.14)	0.042	
Mixed/negative	28 (49.1%)	21 (72.4%)	
Sustainment	Sustained < 15 years	31 (51.7%)	20 (66.7%)	1.87 (0.75–4.66)	0.178	
Sustained 15+ years	29 (48.3%)	10 (33.3%)	
Implementation impact constituente	End-users	51 (85.0%)	22 (73.3%)	2.06 (0.70–6.04)	0.188	
Not reported	9 (15.0%)	8 (26.7%)			
Program deliverers	32 (53.3%)	16 (53.3%)	0.82 (0.34–1.98)	0.654	
Not reported					
Decision makers	11 (18.3%)	12 (40.0%)	0.34 (0.13–0.90)	0.03	
Not reported					
NGO, non-government organization/not-for-profit organization.

aIncluded capacity building, information and environment interventions and multicomponent interventions.

bNon-university conceptual developers included government and non-government organization. Unable to determine lead (n = 4).

cCommunity deliverers (including individuals, clubs, religious and educational organizations, workplaces) and supra community (including government, health facilities, multiple partners, commercial).

dCommercial setting (n = 1) removed from category analysis.

eCategories are dichotomous and not mutually exclusive.

Bold indicates statistically significant result.

Intervention outcomes and pathways

Evaluation evidence used to determine population impact, program sustainment or the type of implementation impact was found for 86 interventions (Table 3). A positive effect on a primary outcome indicator was observed in 37 (43.0%) interventions in total, while a mixed effect (some positive, some negative outcomes) were observed in 46 (53.5%) interventions. Three studies reported only negative effects. Comparing interventions by the pathway undertaken, those which had been scaled up through the research pathway were almost three times as likely to have any positive impacts than those that did not follow the research pathway (P = 0.042). Focusing just on those that reported an impact on end-users behaviour (n = 73), this difference increased to over three-fold [odds ratio (OR): 3.33; 95% confidence interval (CI): 1.05–10.45].

In terms of sustainment, we found that almost half of those interventions in the research pathway had been sustained for more than 15 years (n = 29), while one-third (n = 10) of those that were delivered at scale had been sustained for the same length of time. However, these differences were not statistically significant in the univariate logistic regression analysis (P = 0.178).

The implementation impact constituents reported in the evaluation studies focused mainly on end-user outcomes (81%). These were individual-level behaviour outcomes (i.e. changes in PA or dietary behaviour or changes in biomarkers). There was no significant difference between the two pathways in terms whether the type of outcomes reported were reported for end-users or not. There were fewer reporting evaluation impacts on program deliverers (n = 48). These included measures of knowledge, or the frequency of activities delivered/polices adopted by delivery organizations (i.e. schoolteachers, workplaces), and were generally investigated as part of a Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework. Impacts related to outcomes to inform decision makers included the number of partnerships formed, or capacity building that was enabled or assessed the cost effectiveness of interventions. These types of impacts were found for only one-quarter (25.6%) of interventions. However, comparison between the research pathways showed that interventions in the non-research pathway was more likely to include research that sought to provide information for decision makers (P = 0.030).

DISCUSSION

This study examined established obesity prevention interventions to understand the influence of research pathways on population-level scale-up and sustainment. Two-thirds of our sample of interventions followed a research pathway of intervention testing before scale-up. Those interventions were almost three times more likely to have beneficial outcomes on population health compared to those that were scaled up without concerted efforts in prior research testing. This emphasizes the importance of establishing program efficacy and effectiveness to determine which interventions are likely to be effective when embedded into the often complex multilevel implementation context in which population health programs are delivered (Glasgow and Chambers, 2012).

Several differences were observed in the intervention characteristics and developmental processes between programs that followed research and non-research pathways to scale-up. Specifically, interventions developed through universities were more likely to have followed a research pathway prior to scale-up. This is unsurprising given the role of researchers in knowledge creation. However, this result supports the view that research-funded implementation is effective and supports the call for strategies to facilitate engagement with the research sector in generating evidence of intervention effectiveness before scale-up (Proctor et al., 2011; Chambers et al., 2013). Conversely, the study revealed that fewer interventions funded by government organizations followed a research pathway. These findings may reflect the challenges and demands of government organizations who are pressured to address health concerns and meet community expectations quickly despite the absence of clear evidence of effective interventions. While this may be the case, it has long been advocated that researchers need to partner more closely with decision makers to ensure that the evidence being generated is not only relevant but timely (Campbell et al., 2009). The concept of partnership and co-creation is central to this stage of scale-up, more so than in efficacy testing, but requires concerted efforts to develop sustained partnerships and trust between policymakers, researchers and community end-users (Bauman, 2022). This is to facilitate opportunities for researchers to provide information on pragmatic, feasible and effective interventions that might be delivered at scale, as often these opportunities can be unpredictable (Lee et al., 2020).

We also found differences in research pathways by country, with interventions in North America more likely to report a research pathway than those in other countries. The findings may reflect the differing levels of resources and research infrastructure for efficacy and effectiveness testing prior to scale-up, and/or differences in systems for the provision of preventive programs. More research is warranted to better understand potential reasons for such discrepancies.

While research pathways to scale-up were associated with positive intervention effects on population-level health outcomes, there was no association with how long interventions had been delivered (sustained). Our original hypothesis was that interventions that followed a comprehensive research pathway would be more beneficial, reinforcing their ongoing sustainment. However, our results indicated that sustainment was not influenced by research pathway. There is a growing concern that resources are being utilized to sustain interventions that may not have the desired effects on population health (particularly those that were scaled up without evidence of effectiveness) or, even worse, may be incurring more harm (Norton and Chambers, 2020). Not all sustained interventions that have been sustained should continue to be sustained (Brownson et al., 2015), particularly when impacts do not show positive results. A new area of research into ‘de-implementation’ is beginning to focus more on strategies to terminate or modify those that have been deemed ineffective (Norton and Chambers, 2020). It is, however, noted that a variety of strategies, such as adaptation, is a necessary process to facilitate moving interventions to scale and facilitating their sustainment (Schell et al., 2013)—hence, monitoring outcomes at scale is important in these modified programs, to ensure worthwhile outcomes are still being achieved, and if not, de-implementation may need to occur.

The time lag to conduct research and translate it into practice has largely been cited to take as long as 14–17 years in the biomedical and cancer control fields (Balas and Boren, 2000; Khan et al., 2021), though recently it has been noted that in obesity prevention interventions, the time to scale-up may indeed be much shorter (Lee et al., 2024). Conducting intervention research is generally time- and resource-intensive, and publication delays from submission to publication may result in a slower time to translating interventions into practice (Björk and Solomon, 2013; Lee et al., 2021). Despite this, while taking a non-research pathway may lead to more rapid translation, untested interventions may prove to be ineffective or, at worst, result in negative unintended consequences (McCrabb et al., 2019). Other research has shown that interventions following non-research pathways yield no difference between intervention or control groups in post-implementation studies (Beyler et al., 2014). Therefore, while a research pathway may take more time, the benefits are potentially much greater.

Building capacity for evidence-based research development approaches to intervention development is necessary for evidence-based practice (Brownson et al., 2018). The results of this study are intended to improve understanding of the contextual factors of real-world implementation. By using the CFIR, we were able to observe research pathways and intervention characteristics across key implementation determinants (Damschroder et al., 2022). Multiple implementation frameworks have been developed to improve the implementation and scale-up of public health interventions (Fleuren et al., 2004; Damschroder et al., 2009; Fixsen et al., 2009; Greenhalgh et al., 2017). However, the processes of disseminating interventions across whole populations is dependent on a broad system of external influences including political, cultural or economic forces (Greenhalgh et al., 2017; Crane et al., 2022). This is perhaps why differences in countries and types of organizations were evident. The findings also highlight the importance of connecting researchers and health policy and practice providers and developing better strategies to improve the connection between knowledge translation and evidence-based practice (Wolfenden et al., 2015, 2020).

Limitations

The data for this research were collected only from publicly available material for consistency. Other factors that may have determined scale-up and outcomes that would have involved interviewing intervention stakeholders are, therefore, not captured. It is possible that research evidence generation may have been undertaken by those we have categorized as non-research pathways, and that the evidence may not be publicly available. Our sampling and search method that relied on academic bibliographic databases, and of selecting interventions where there was some evidence-generating research, may bias our sample. Interventions that have neither undergone any research development nor impact evaluation may be under-represented in this sample, and our results are a conservative indication of differences in implementation characteristics and outcome effects. In collating intervention outcomes, we note that very few published reports and scientific literature of the interventions reported negative or no effect. This is perhaps a product of evidence bias towards the reporting of positive impacts (Dwan et al., 2013). Auditing existing large-scale programs, such as those funded by the government, and exploring the evidence supporting them would represent an alternate means of investigating the study aims in a way that may reduce some of the potential biases. While such research is warranted, previous surveys with public health practitioners suggest that just over half of programs are evidence-based, a finding similar to what we have reported (Gibbert et al., 2013).

CONCLUSIONS

Several differences were observed across intervention characteristics and developmental processes between programs that followed research and non-research pathways to scale-up. Such findings underscore the benefits of planned, evidence-based approaches for the development and scale-up of health promotion programs. Taking a research pathway, while more time-consuming, provides more likely evidence of effectiveness at-scale, while minimizing potential unintended negative consequences from untested interventions. However, it is recognized in some situations, where there is urgency to address a pressing need, the ‘rapid implementation’ of interventions tested or untested may need to occur. This is where opportunities exist for researchers and decision makers to work closely together to ensure that under such circumstances, the best possible solutions are co-developed to achieve intended outcomes. In these circumstances, undertaking quality evaluation of the impacts of programs in the context of their delivery is warranted.

Supplementary Material

daae111_suppl_Supplementary_File

ACKNOWLEDGEMENTS

We would like to thank Dr Erika Goldbaum and Caitlin Bialek for their assistance in data entry.

AUTHOR CONTRIBUTIONS

M.C. with A.B. and K.L. conceived the study. M.C. led the data collection and analysis process. All authors were involved in the final interpretation of the data. M.C. led the writing of the final manuscript, with K.L. leading the final editing of the manuscript. All authors contributed to the conceptual writing, provided individual expertise and contributed to the final review of the manuscript. All authors read and approved the final manuscript.

FUNDING

Data collection was supported by The Australian Prevention Partnership Centre through the National Health and Medical Research Council of Australia (NHMRC) Partnership Centre grant scheme (grant ID: GNT9100001). ACT Health, SA Health, Tasmania Health and VicHealth have contributed funds to support this work as part of the NHMRC Partnership Centre grant scheme. M.C. and K.L. were supported by the Prevention Research Support Program, funded by the New South Wales Ministry of Health.

CONFLICT OF INTEREST STATEMENT

M.C., K.L. and A.B. declare that they have no competing interests. L.W. has had some direct involvement in the implementation of one of the included interventions but was not involved in the design, data collection or analysis.

DATA AVAILABILITY

The data underlying this article will be shared on reasonable request to the corresponding author.
==== Refs
REFERENCES

Balas, E. A. and Boren, S. A. (2000) Managing clinical knowledge for health Care Improvement. Yearb Med Inform, 65–70.27699347
Bauman, A. (2022) A personal reflection on co-creation in public health: a dream partly realised. Public Health Research & Practice, 32 , e2022.
Beyler, N., Bleeker, M., James-Burdumy, S., Fortson, J. and Benjamin, M. (2014) The impact of Playworks on students’ physical activity during recess: findings from a randomized controlled trial. Preventive Medicine, 69 , S20–S26.25456798
Björk, B.-C. and Solomon, D. (2013) The publishing delay in scholarly peer-reviewed journals. Journal of Informetrics, 7 , 914–923.
Brownson, R. C., Allen, P., Jacob, R. R., Harris, J. K., Duggan, K., Hipp, P. R. et al . (2015) Understanding mis-implementation in public health practice. American Journal of Preventive Medicine, 48 , 543–551.25891053
Brownson, R. C., Fielding, J. E. and Green, L. W. (2018) Building capacity for evidence-based public health: reconciling the pulls of practice and the push of research. Annual Review of Public Health, 39 , 27–53.
Campbell, D. M., Redman, S., Jorm, L., Cooke, M., Zwi, A. B. and Rychetnik, L. (2009) Increasing the use of evidence in health policy: practice and views of policy makers and researchers. Australia and New Zealand Health Policy, 6 , 21.19698186
Chambers, D. A., Glasgow, R. E. and Stange, K. C. (2013) The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implementation Science, 8 , 117.24088228
Crane, M., Lee, K., Bohn-Goldbaum, E., Nathan, N. and Bauman, A. (2024) Sustaining health obesity prevention programs: lessons from real-world population settings. Evaluation and Program Planning, 103 , 102404.38244416
Crane, M., Nathan, N., McKay, H., Lee, K., Wiggers, J. and Bauman, A. (2022) Understanding the sustainment of population health programmes from a whole-of-system approach. Health Research Policy and Systems, 20 , 1–12.34980147
Damschroder, L. J., Aron, D., Keith, R., Kirsh, S., Alexander, J. and Lowery, J. (2009) Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Science, 4 , 50.19664226
Damschroder, L. J., Reardon, C., Opra Widerquist, M. and Lowery, J. (2022) Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum. Implementation Science, 17 , 1–10.34983585
Dwan, K., Gamble, C., Williamson, P. R., Kirkham, J. J. and Group, R. B. (2013) Systematic review of the empirical evidence of study publication bias and outcome reporting bias—an updated review. PLoS One, 8 , e66844.23861749
Fixsen, D. L., Blase, K. A., Naoom, S. F. and Wallace, F. (2009) Core implementation components. Research on Social Work Practice, 19 , 531–540.
Fleuren, M., Wiefferink, K. and Paulussen, T. (2004) Determinants of innovation within health care organizations: literature review and Delphi study. International Journal for Quality in Health Care, 16 , 107–123.15051705
Gibbert, W. S., Keating, S. M., Jacobs, J. A., Dodson, E., Baker, E., Diem, G. et al . (2013) Training the workforce in evidence-based public health: an evaluation of impact among US and international practitioners. Preventing Chronic Disease, 10 , E148.24007676
Glasgow, R. E. and Chambers, D. (2012) Developing robust, sustainable, implementation systems using rigorous, rapid and relevant science. Clinical and Translational Science, 5 , 48–55.22376257
Glasgow, R. E., Lichtenstein, E. and Marcus, A. C. (2003) Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. American Journal of Public Health, 93 , 1261–1267.12893608
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A’Court, C. et al . (2017) Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. Journal of Medical Internet Research, 19 , e367.29092808
Indig, D., Lee, K., Grunseit, A., Milat, A. and Bauman, A. (2018) Pathways for scaling up public health interventions. BMC Public Health, 18 , 68.
Khan, S., Chambers, D. and Neta, G. (2021) Revisiting time to translation: implementation of evidence-based practices (EBPs) in cancer control. Cancer Causes & Control: CCC, 32 , 221–230.33392908
Kingdon, J. W. and Stano,E. (1984). Agendas, Alternatives, and Public Policies. Little, Brown, Boston, MA.
Lee, K., Bauman, A., Wolfenden, L., Phongsavan, P. and Crane, M. (2024) How long does it take to scale-up obesity prevention interventions? Preventive Medicine, 185 , 108012.38821419
Lee, K., Ding, D., Grunseit, A., Wolfenden, L., Milat, A. and Bauman, A. (2021) Many papers but limited policy impact? A bibliometric review of physical activity research. Translational Journal of the American College of Sports Medicine, 6 , 1–9.
Lee, K., van Nassau,F., Grunseit,A., Conte,K., Milat,A., Wolfenden,L. et al . (2020) Scaling up population health interventions from decision to sustainability—a window of opportunity? A qualitative view from policy-makers. Health Research Policy and Systems, 18 , 118.33036633
McCrabb, S., Hall, A., McKay, H., Gonzalez, S., Milat, A., Bauman, A. et al . (2023) From trials to communities: implementation and scale-up of health behaviour interventions. Health Research Policy and Systems, 21 , 79.37525165
McCrabb, S., Lane, C., Hall, A., Milat, A., Bauman, A., Sutherland, R. et al . (2019) Scaling-up evidence-based obesity interventions: a systematic review assessing intervention adaptations and effectiveness and quantifying the scale-up penalty. Obesity Reviews, 20 , 964–982.30868745
Norton, W. E. and ChambersD. A. (2020) Unpacking the complexities of de-implementing inappropriate health interventions. Implementation Science, 15 , 2. 10.1186/s13012-019-0960-9 31915032
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A. et al . (2011) Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38 , 65–76.20957426
Rutter, H. (2018) The complex systems challenge of obesity. Clinical Chemistry, 64 , 44–46.29097510
Schell, S. F., Luke, D. A., Schooley, M. W., Elliott, M. B., Herbers, S. H., Mueller, N. B. et al . (2013) Public health program capacity for sustainability: a new framework. Implementation Science, 8 , 15.23375082
Wolfenden, L., Barnes, C., Jones, J., Finch, M., Wyse, R. J., Kingsland, M. et al . (2020) Strategies to improve the implementation of healthy eating, physical activity and obesity prevention policies, practices or programmes within childcare services. Cochrane Database of Systematic Reviews, 2 , 1–131.
Wolfenden, L., Mooney, K., Gonzalez, S., Hall, A., Hodder, R., Nathan, N. et al . (2022) Increased use of knowledge translation strategies is associated with greater research impact on public health policy and practice: an analysis of trials of nutrition, physical activity, sexual health, tobacco, alcohol and substance use interventions. Health Research Policy and Systems, 20 , 1–10.34980147
Wolfenden, L., Ziersch, A., Robinson, P., Lowe, J. and Wiggers, J. (2015) Reducing research waste and improving research impact. Australian and New Zealand Journal of Public Health, 39 , 303–304.26239576
World Health Organization (2013). Global Action Plan for the Prevention and Control of Noncommunicable Diseases 2013-2020. World Health Organization, Geneva.
World Health Organization. (2021) International Classification of Health Interventions (ICHI) [cited 20 March 2021]. https://icd.who.int/dev11/l-ichi/en.
World Obesity Federation (2023). World Obesity Atlas. World Obesity Federation, London, UK. https://data.worldobesity.org/publications/?cat=19.
