
==== Front
Implement Sci Commun
Implement Sci Commun
Implementation Science Communications
2662-2211
BioMed Central London

633
10.1186/s43058-024-00633-5
Research
A research agenda to advance the study of implementation mechanisms
http://orcid.org/0000-0001-8920-8075
Lewis Cara C. Cara.Lewis@nih.gov

1
Frank Hannah E. 2
Cruden Gracelyn 3
Kim Bo 45
Stahmer Aubyn C. 6
Lyon Aaron R. 7
Albers Bianca 8
Aarons Gregory A. 9
Beidas Rinad S. 10
Mittman Brian S. 11
Weiner Bryan J. 12
Williams Nate J. 13
Powell Byron J. 141516
MNoE GroupFernandez Maria E.
Masyuko Sarah
Pullmann Michael
Saldana Lisa
Wagner Anjuli D.
Walsh-Bailey Callie
Stirman Shannon Wiltsey

1 https://ror.org/0027frf26 grid.488833.c 0000 0004 0615 7519 Kaiser Permanente Washington Health Research Institute, 1730 Minor Avenue, Suite 1600, Seattle, WA 98101 USA
2 https://ror.org/05gq02987 grid.40263.33 0000 0004 1936 9094 The Warren Alpert Medical School, Brown University, Box G-BH, Providence, RI 02912 USA
3 https://ror.org/04jmr7c65 grid.413870.9 0000 0004 0418 6295 Chestnut Health System, Lighthouse Institute – OR Group, 1255 Pearl St, Ste 101, Eugene, OR 97401 USA
4 https://ror.org/04v00sg98 grid.410370.1 0000 0004 4657 1992 Center for Healthcare Organization and Implementation Research, VA Boston Healthcare System, 150 South Huntington Avenue, Boston, MA 02130 USA
5 grid.38142.3c 000000041936754X Department of Psychiatry, Harvard Medical School, 25 Shattuck Street, Boston, MA 02115 USA
6 grid.27860.3b 0000 0004 1936 9684 UC Davis MIND Institute, 2825 50Th St, Sacramento, CA 95819 USA
7 https://ror.org/00cvxb145 grid.34477.33 0000 0001 2298 6657 Department of Psychiatry and Behavioral Sciences, University of Washington, 1959 NE Pacific Street Box 356560, Seattle, WA 98195-6560 USA
8 https://ror.org/02crff812 grid.7400.3 0000 0004 1937 0650 Institute for Implementation Science in Health Care, University of Zurich, Zürich, Switzerland
9 https://ror.org/0168r3w48 grid.266100.3 0000 0001 2107 4242 Department of Psychiatry, University of California San Diego, 9500 Gilman Drive La Jolla California, San Diego, 92093 CA USA
10 https://ror.org/000e0be47 grid.16753.36 0000 0001 2299 3507 Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern University, 625 N Michigan Avenue, Evanston, IL 60661 USA
11 grid.280062.e 0000 0000 9957 7758 Division of Health Services Research & Implementation Science, Department of Research & Evaluation, Kaiser Permanente Southern California, 100 S Los Robles Ave, Pasadena, CA 91101 USA
12 https://ror.org/00q16t150 grid.488602.0 Department of Global Health, School of Public Health, Box 357965, Seattle, WA 98195 USA
13 https://ror.org/02e3zdp86 grid.184764.8 0000 0001 0670 228X School of Social Work, Boise State University, Boise, ID 83725 USA
14 https://ror.org/01yc7t268 grid.4367.6 0000 0004 1936 9350 Center for Mental Health Services Research, Brown School, Washington University in St. Louis, St. Louis, MO USA
15 https://ror.org/01yc7t268 grid.4367.6 0000 0004 1936 9350 Center for Dissemination & Implementation, Institute for Public Health, Washington University in St. Louis, St. Louis, MO USA
16 grid.4367.6 0000 0001 2355 7002 Division of Infectious Diseases, John T. Milliken Department of Medicine, School of Medicine, Washington University in St. Louis, St. Louis, MO USA
16 9 2024
16 9 2024
2024
5 9817 1 2024
30 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Background

Implementation science scholars have made significant progress identifying factors that enable or obstruct the implementation of evidence-based interventions, and testing strategies that may modify those factors. However, little research sheds light on how or why strategies work, in what contexts, and for whom. Studying implementation mechanisms—the processes responsible for change—is crucial for advancing the field of implementation science and enhancing its value in facilitating equitable policy and practice change. The Agency for Healthcare Research and Quality funded a conference series to achieve two aims: (1) develop a research agenda on implementation mechanisms, and (2) actively disseminate the research agenda to research, policy, and practice audiences. This article presents the resulting research agenda, including priorities and actions to encourage its execution.

Method

Building on prior concept mapping work, in a semi-structured, 3-day, in-person working meeting, 23 US-based researchers used a modified nominal group process to generate priorities and actions for addressing challenges to studying implementation mechanisms. During each of the three 120-min sessions, small groups responded to the prompt: “What actions need to be taken to move this research forward?” The groups brainstormed actions, which were then shared with the full group and discussed with the support of facilitators trained in structured group processes. Facilitators grouped critical and novel ideas into themes. Attendees voted on six themes they prioritized to discuss in a fourth, 120-min session, during which small groups operationalized prioritized actions. Subsequently, all ideas were collated, combined, and revised for clarity by a subset of the authorship team.

Results

From this multistep process, 150 actions emerged across 10 priority areas, which together constitute the research agenda. Actions included discrete activities, projects, or products, and ways to shift how research is conducted to strengthen the study of implementation mechanisms.

Conclusions

This research agenda elevates actions to guide the selection, design, and evaluation of implementation mechanisms. By delineating recommended actions to address the challenges of studying implementation mechanisms, this research agenda facilitates expanding the field of implementation science, beyond studying what works to how and why strategies work, in what contexts, for whom, and with which interventions.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-024-00633-5.

Keywords

Implementation science
Mechanisms
Mediators
Determinants
Implementation strategies
Methods
Measurement
Design
Theory
Causality
http://dx.doi.org/10.13039/100000133 Agency for Healthcare Research and Quality R13HS025632 Lewis Cara C. http://dx.doi.org/10.13039/100000025 National Institute of Mental Health P50MH126219 R01MH111950 R01MH111981 R25MH080916 Lewis Cara C. Stahmer Aubyn C. National Institute for Mental HealthK01MH113806 Powell Byron J. http://dx.doi.org/10.13039/100000054 National Cancer Institute P50CA244432 R01CA262325 P50CA244690 Lewis Cara C. Beidas Rinad S. Weiner Bryan J. issue-copyright-statement© BioMed Central Ltd. 2024
==== Body
pmcContributions to the literature This research agenda operationalizes a set of activities to strengthen the implementation science field’s focus on why and how strategies work.

The research agenda addresses the following activities: accumulating knowledge, innovating methods and overcoming design challenges, improving measurement, providing guidance for specifying causal mechanisms, increasing focus on theorizing, engaging the policy and practice community, engaging funders, building capacity, enhancing equity, and effectively disseminating methods.

Studying implementation mechanisms can promote pragmatic strategy development, equitable processes and outcomes, and policy relevance by clarifying pathways for overcoming contextually specific barriers and achieving outcomes of interest.

Background

Some see implementation science as not just a pathway, but the pathway for advancing equity in healthcare access and outcomes, and equitable population health [1]. Although this research pathway can lead to equity, it is certainly not guaranteed, and in fact, like many fields, most implementation science theories, models, and frameworks did not center equity until recently [2]. This omission leaves implementation studies and strategies vulnerable to unintended consequences (or ripple effects) that might actually exacerbate disparities [3, 4]. The field of implementation science has made significant progress in this regard. Scholars like Woodward et al. [5] offer practical guidance for incorporating health equity domains into implementation determinant frameworks, and Gaias et al. [6] proposed a process to evaluate and adapt implementation strategies to promote equity. Walsh-Bailey is developing a resource to guide the integration of equity into strategy selection, design, and specification [7]. Moreover, numerous efforts collate factors that enable or obstruct the implementation of evidence-based interventions [8–10], and compile behavior change techniques and implementation strategies that may modify these factors [11–16]. Even with these advances, little research sheds light on how or why strategies work, in what contexts, and for whom [17–21]. Studying implementation mechanisms, or the processes through which strategies exert their effects on outcomes, can address this research gap to meaningfully advance the field of implementation science and enhance its value in facilitating equitable policy and practice change. Mechanistic implementation research can identify potential mediators or moderators that illuminate differential strategy impact based on factors such as gender, race/ethnicity, socioeconomic status, and center on understanding equitable approaches to implementation science and practice.

One of the principles of implementation science is that context matters, and by nature, each context is unique. The people, their interactions, their physical environment and resources, and their history and beliefs about the future, are among the subset of aspects that are diverse among clinics in the same organization, schools in the same district, and hospitals in the same health system. As implementation science evolves, complex and costly strategies are increasingly being deployed, making equity issues especially pronounced for those receiving care in under-resourced settings [20]. Evidence suggests that tailored implementation may be superior to standardized approaches [22, 23], but tailoring in the absence of understanding strategy mechanisms may compromise outcomes for some or undermine scaling positive outcomes. Establishing strategy mechanisms of action means that the essence of how a strategy works is known and empirically supported. Therefore, when tailoring, adapting, or modifying to fit different contexts, the essence of the strategy’s operation can be retained. When strategies are streamlined to fit contextual constraints or adapted to be a better fit, the mechanism ought to be activated if we are to expect the same outcome. Conversely, if strategies underperform or fail to work in certain settings, unpacking the causal pathway can lead to isolation of contextual factors that threaten mechanism activation or demand a new mechanism altogether. This is not to say that simply studying mechanisms will guarantee equitable outcomes, but in studying them, equitable implementation processes and outcomes are more likely.

To this end, in 2017, the Society for Implementation Research Collaboration (SIRC) conference theme centered implementation mechanisms to elevate dialogue and research about, “What Makes Implementation Work and Why?” [24]. SIRC is a not-for-profit society that convenes scholars, practitioners, policy makers, and others interested in advancing rigorous evaluation of implementation initiatives. SIRC’s call to action was motivated by the observation across trials that heterogeneity is the rule, not the exception, and weak main effects result. Thus, advancing the study of implementation mechanisms may offer benefits to research and practice communities. For example, identifying and evaluating mechanisms can help researchers learn from null studies [17] and optimize strategies for subsequent efforts or different objectives (e.g., equity, effectiveness, scalability) [25]. Articulating mechanisms can guide the practice community to identify the impact that strategies might have on their outcomes and inform their design or tailoring of strategies to the local context [26, 27]. Despite this call, only 7% of abstracts included at the subsequent (2019) SIRC conference [28] explicitly “featured the study of implementation mechanisms” [29].

In response to this need to advance the study of mechanisms, we convened an Agency for Healthcare Research and Quality-funded 3-year conference series titled, “Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration” [30]. The specific aims were to (1) develop a research agenda on implementation mechanisms, and (2) disseminate the research agenda to research, policy, and practice audiences. Similar to processes used for generating related research agendas (e.g., sustainability research [31]), concept mapping was employed in the first two years of the grant to elucidate challenges to advancing implementation mechanisms research [30, 32] and to organize these ideas into conceptually distinct clusters. Reported in more detail elsewhere [30, 32], concept mapping analyses yielded a 12-cluster solution that organized 105 challenge statements within five “super clusters” of mechanism research domains: (1) Accumulating Knowledge, (2) Conceptualization and Measurement, (3) Methods and Design, (4) Strategy, Mechanisms, Determinant, Outcome Linkages, and (5) Theory, Causality, and Context. See Table 1 for a complete list of identified challenges organized by cluster. These concept mapping results provided the basis for the research agenda. This paper describes how actions that could overcome those challenges were identified and presents the resulting research agenda. Table 1 Concept mapping super-clusters, clusters, and statements

 Cluster	Statement	
THEORY, CAUSALITY, & CONTEXT	
 Insufficient Use of Theory	Not always clear whether to prioritize theory or pragmatic (stakeholder & context) input in defining mechanism	
Theories, models, and frameworks rarely specify mechanisms	
Studies of implementation strategies and/or mechanisms are not often informed by theory	
Studies rarely include explicit theories of change or logic models, and those that do rarely articulate mechanisms	
Theories and frameworks do not always align with the level of strategies and mechanisms activated	
Behavior change theories have not been sufficiently applied to identify candidate implementation mechanisms	
There is not a consistent understanding of causal pathways and templates to promote testing those pathways	
 Conceptualizing the Causal Chain and the Role of Context	Assessment of context should be linked to the implementation conceptual framework	
We don't know which mechanisms to prioritize because of lack of information about the malleability of different determinants	
We have a limited understanding of mechanisms that are generalizable across settings, populations, and interventions	
We don't understand if and how mechanisms change across cultural context	
Many conceptual and measurement models of mechanisms do not include an implementation strategy	
We do not sufficiently clarify the strategy-mechanism-outcome cascade when conceptualizing studies	
Unclear if mechanisms for implementation and de-implementation differ	
Given the lack of conceptual clarity and difficulty measuring context, we may not adequately capture context as a moderator	
Determinants can also be mechanisms depending upon what questions are being asked, which contributes to lack of conceptual clarity and adds complexity	
STRATEGY, MECHANISM, DETERMINANT, OUTCOME LINKAGES	
 Factors Influencing Strategies, Mechanisms, & Determinants	Unclear how adaptation of implementation strategies affects if or how a mechanism is activated	
We do not know how stakeholder involvement functions in activating or moderating mechanisms	
There is often a mismatch between strategies selected and the level (e.g., individual, organizational, etc.) of determinants and mechanisms identified	
How a strategy is operationalized could affect whether/how a mechanism is activated	
Complexity in disentangling implementation strategy from intervention and clearly labeling each component makes studying mechanisms challenging	
We don't have a clear understanding of how implementation strategies exert their effects in different settings	
Unexpected contextual changes may affect mechanism activation	
Limited use of systematic methods for strategy selection that include an articulation of how strategies work	
Not clear which implementation strategies address specific determinants	
 Complexity & Multiplexity in the Strategy-Mechanism-Determinant-Outcome Pathway	Mechanisms are difficult to elucidate in complex social phenomenon and may be reductionistic	
A single mechanism can be addressed by multiple strategies	
Difficult to prioritize mechanisms to study given the presence of multiple mechanisms	
Because of methodological and analytic limitations, we tend to over-simplify the complexity of causal pathways	
Different mechanisms need to be activated at different points of the implementation process	
Different outcomes might be affected by different mechanisms	
It is difficult to detect the interactive effect of strategies on mechanisms and outcomes	
Multifaceted strategies may have interactive, indirect, or direct effects	
It is difficult to determine which components of a multifaceted strategy are necessary and sufficient	
It is unclear which strategies at what dose and time are necessary to activate a mechanism	
A single strategy can address multiple mechanisms	
Studying discrete strategies may be necessary but insufficient in delineating mechanisms	
Mechanisms are difficult to elucidate in complex social phenomenon and may be reductionistic	
DESIGNS, METHODS, & MEASUREMENT	
 Where to Focus Measure Development & Use	Measures of mechanisms are not harmonized across studies	
There is an overemphasis on mechanisms at the intra-personal level	
Lack of high-quality, brief, pragmatic measures of mechanisms	
Lack of measures to assess all factors in a causal pathways	
Limited guidance for selecting measures to assess mechanisms	
Systems-level mechanisms missing from studies	
Implementation science rarely considers client/patient-level mechanisms	
 Time as it Relates to Design & Measurement	Typically not feasible to directly manipulate implementation strategies to activate mediators	
Measuring a mechanism may have an impact on the mechanism itself	
Measurement and analysis of mechanisms across sites may be hampered by differential rates of change	
Cannot establish a mechanism without carefully timing measurement	
Unclear how often we need to measure mechanisms	
Tension between precision testing of mechanisms and allowing for broader assessment of multiple mechanisms at multiple time points	
Unclear how to best study emergent mechanisms vs. those hypothesized at the onset of a study	
May be measuring mechanisms/outcomes too soon to determine if the mechanism was successfully activated	
Methods for tracking strategies are insufficient for analyzing multiple mechanisms and change in mechanisms over time	
It is difficult to measure mechanism activation	
 Methods & Design Opportunities	Assessment of time component is often missing from qualitative studies	
Methods such as random assignment to all possible combinations of implementation strategies are not feasible	
Qualitative methods not sufficient for establishing evidence for a mechanism	
Underuse of qualitative and mixed methods approaches for studying mechanisms	
Too much reliance on self-report measures	
Study of mechanisms limited mostly to earlier phases of the implementation process, and there is less understanding of later stages	
Cross-sectional measurement insufficient to fully establish or confirm mechanisms	
Current methods are challenging or insufficient for analyzing multiple mechanisms and change in mechanisms over time	
Many implementation studies are not sufficiently powered to examine multi-level mechanisms	
Lack of multi-level mediator assessment	
Observational longitudinal studies are not sufficient to establish a mechanism	
Underutilization of rapid ethnographic methods and other real-time data collection methods for studying mechanisms	
Underuse of observational designs for studying mechanisms	
Lack of prospective study designs to study mechanisms	
Commonly used study designs are not well-suited to studying mechanisms	
Conducting experimental studies to establish mechanisms using best practices is costly	
Pragmatic, stakeholder-engaged study designs are underutilized in the study of mechanisms	
Tightly controlled studies to examine hypothesized strategies and mechanisms are lacking	
 Overreliance on Problematic or Insufficient Methods	Rarely is the study of mechanisms a primary study aim	
Studies of mechanisms are of variable quality and there are few high-quality studies	
We have few studies that examine implementation mechanisms, and those that do focus on a narrow range of strategies or mechanisms	
Most studies do not assess mediators in the same way, which makes it difficult to compare across studies	
Difficult to balance gathering sufficient data to study mechanisms and participant/researcher burden	
ACCUMULATING KNOWLEDGE	
 Prioritizing Mechanism Research and Incorporating Other Knowledge	It is difficult to balance the need for rapid knowledge generation and having a methodical approach to building evidence over time	
Mechanisms are not systematically or consistently assessed across studies	
Not fully capitalizing on systems science approaches to study mechanisms	
Lack of incorporation of findings from pilot studies into conceptualizing mechanisms	
We rarely include the perspectives of clients/patients when studying mechanisms	
We are not leveraging/combining datasets to study mechanisms	
 Accumulating Knowledge Within & Across Disciplines	Poor reporting limits our understanding of mechanisms	
No central location to collate learning across studies	
Field lacks foundational training in mechanistic research	
Funding agencies may not recognize the need for research that focuses on mechanisms	
The study of mechanisms is approached from many different disciplines, making it difficult to have a set of cohesive methods and designs	
Do not have rapid, systematic ways of learning from available literature to inform study of mechanisms	
Not enough reliance on previous studies to identify mechanisms	
Not adequately capitalizing on other fields that have advanced the study of mechanisms	
By focusing on mechanisms, we risk minimizing the importance of health and public health outcomes	
CONCEPTUALIZATION & MEASUREMENT	
 Nomenclature and Associated Resources	Researchers' assumptions may interfere with consideration/selection of appropriate mechanisms	
Lack of clearly defined roles (researchers, practitioners) and relationships between roles in studying mechanisms	
Lack of common language and precise and distinct terms for mechanisms within implementation science	
The field lacks a list of mechanisms	
There is little attention paid to the criteria for establishing mechanisms	
Limited guidance on how to specify and report mechanisms	
Confusion among the terms mechanism, mediator, and moderator	
Lack of common language for mechanisms across fields	
Language for studying mechanisms is rife with jargon, which does not allow for clear communication with frontline staff and other relevant stakeholders, hindering incorporation of their perspectives	
Funders lack clear expectations and common language regarding the study of implementation mechanisms	
 Conceptual & Empirical Ambiguity	Active ingredients, core functions, and mechanisms are conceptually related, but distinct, and they are rarely sufficiently operationalized and measured	
Difficult to determine which mechanisms to prioritize due to lack of evidence on the strength of various mechanisms	
Studies do not always explicitly link causal theory to measurement	

Method

Mechanisms Network of Expertise (MNoE)

The research agenda was developed by the Mechanisms Network of Expertise (MNoE). The MNoE is composed of over 40 invited implementation scientists who are diverse with respect to several dimensions (e.g., gender, race/ethnicity, stage of career, focus on priority populations, research settings), but who are predominantly United States (US)-based (4 scholars are from outside the US); See Additional File 1. Expertise ranged across various aspects of implementation mechanism research including strategy development, measurement, design, theory, and practice. We gathered collective wisdom and engaged in reciprocal learning with these experts through immersive, multi-day “Deep Dive” meetings.

Identifying research priorities via nominal group technique: MNoE data generation

A US-based1 subset of the MNoE (N = 23) met in person for a 3-day Deep Dive to address two goals: 1) expand upon the challenges derived from the previously completed concept mapping, and 2) generate ideas or actions (hereafter just referred to as actions) organized by priority areas, which constitute the research agenda, to advance the study of implementation mechanisms. To this end, attendees received handouts with the cluster solution from concept mapping and the list of statements associated with each cluster (Table 1). These two goals were pursued through four, 120-min sessions comprised of a 75-min small-group activity followed by a 45-min large-group activity. (Table 2) Group activities were structured using evidence-informed, semi-structured group problem solving activities—called “scripts”— derived from operations, consulting, and systems science methods [33, 34] (Table 3). Scripts include discussion prompts, guidelines about how time is spent (e.g., in small versus large groups), roles to be assumed by individuals (e.g., timekeeper), and session goals (e.g., brainstorming actions for a given cluster). A core planning team (n = 5) selected scripts from a repository and tailored them to Deep Dive objectives (e.g., identifying actions for addressing challenges to studying implementation mechanisms) across the sessions. Tailoring of scripts included adjusting the time allocated for each script, the examples used, and the wording of the prompts. The planning team assigned small group membership beforehand to ensure diverse groups regarding career stage and content or methodological expertise. The small group composition changed by session to stimulate creative conversation and cross-pollinate ideas by hearing new perspectives. Table 2 Clusters reviewed during deep dive sessions

Session #	Super-cluster	Clusters included	
1	• Theory, Causality, Context

• Strategy, Mechanism, Determinant,

Outcome, Linkages

	• Insufficient use of theory

• Conceptualizing the causal chain and role of context

• Factors influencing strategies, mechanisms, and determinants

• Complexity and multiplexity in the strategy mechanism-determinant-outcome pathway

	
2	• Accumulating Knowledge

• Conceptualization & Measurement

	• Accumulating knowledge within and across disciplines

• Prioritizing mechanism research and incorporating other knowledge

• Nomenclature and associated resources

• Conceptual and empirical ambiguity

• Where to focus measure development and use

	
3	• Methods & Design	• Time as it relates to design and measurement

• Over-reliance on problematic or insufficient methods

• Methods and design opportunities

	
4	• All	• All	

Table 3 Group model building “scripts”

Script	Script Purpose (Tailored for MNoE Deep Dive)	Activity	Session(s) used	Length	
Action Ideas	“Identify action ideas (solutions, possibilities, priorities) for addressing the challenges and opportunities in each concept mapping cluster.”	Similar to Nominal Group Technique procedures, action ideas were first identified within small groups. A notetaker wrote one idea per piece of 8.5″ × 11″ paper. This allowed each idea to be visually displayed and clustered with like ideas in the large group session. For each action idea, groups were asked to consider:

• Who might be responsible for fulfilling this action?

• How might it be achieved?

• How long might it take to realize this action idea? (e.g., 1 project over a year, multiple projects over 5 years, etc.)

During the large group session, a facilitator asked groups to share one complex idea that might require group discussion and one favorite idea per concept mapping super cluster for a total of 2–4 ideas. Participants were asked to “briefly describe the idea and how it would advance the study of mechanisms and IS.” Other groups had the opportunity to ask questions about the shared action idea or express if they had a similar idea. The order in which small groups were asked to share was varied across each large group session. A person assigned the role of “wall builder” identified themes among ideas and clustered ideas on the wall as they were shared out. Ideas that were not shared were clustered after the small group finished sharing their priority ideas. The wall-builder reflected on these themes after all small groups shared, and invited corrections or suggestions

	Sessions 1–3	75 min (Small Group)

45 min (Large Group)

	
Dots	Identify action ideas of priority to discuss during final small group and large group session	Each participant received 5 stickers to vote on action ideas from earlier sessions. Participants could vote on a specific action idea or on a cluster of ideas by placing the sticker on the small piece of paper (action ideas) or large pieces of paper labeled with action idea themes (idea clusters). Participants voted simultaneously and could put multiple stickers on a single action idea or cluster

The action ideas and clusters with the highest votes (i.e., dots) were identified by facilitators. The large group briefly met to identify: a) whether the prioritized action ideas and clusters could be further clustered to allow for efficient discussion of as many prioritized ideas as possible, and b) who would participate in each small group

	Immediately prior to final small group (Session 4)	20 min (10 min voting, 10 min group discussion and organization)	
Text in quotes was explicitly stated to participants

A tailored Nominal Group Technique process was followed for the first three sessions. Instead of first brainstorming individually, as in the traditional Nominal Group Technique [35], small groups first generated action ideas before sharing, discussing, and voting on priority ideas with the large group. Attendees did the following in small groups before converging as a large group (Table 3): 1) Assign group roles, including scribe (to record discussion), reporter for large group, and timekeeper. Individuals could assume more than one role. 2) Brainstorm actions for inclusion in the research agenda and address the challenges from the five super-clusters (the planning team assigned which super-clusters were discussed during each of these sessions). Actions could include methods, tools, activities, meetings, research products, research foci, disciplines, or people/perspectives to be engaged. 3) Prioritize two actions for full group discussion: based on consensus, one idea favored by the group and one idea that was complex, underdeveloped, or surprising to work through were selected. Small groups were encouraged to spend approximately 60 min brainstorming and 15 min prioritizing actions. Each prioritized action was submitted on paper for sharing in the large group session. Groups were encouraged to write as many actions as they could generate. Scribes’ notes were later analyzed (see below). All actions generated, not just those prioritized for deeper discussion, were considered in developing the research agenda.

During the first three large-group discussions, each group’s reporter briefly described how their two prioritized actions would advance the study of implementation mechanisms. Each group had 5 min to share and take questions. Simultaneous with sharing out, facilitators collected the papers and grouped similar actions on a wall visible to all. After all small groups shared, the facilitator summarized the action themes. The large group collectively reflected on these and used the remaining time to further develop prioritized actions.

The fourth (final) session synthesized and expanded actions brought forth in the preceding sessions. Each attendee used five votes to indicate preferred actions (or group of actions) [36]. The highest-voted actions (n = 6) were prioritized for this session. Attendees self-selected into small groups based on which prioritized actions they wanted to discuss. During the final large-group session, each group shared how the actions had evolved or whether new actions emerged. One facilitator synthesized actions and asked clarifying questions, while another captured actions and priorities on large pieces of paper for the large group to see and discuss.

Data extraction and consolidation

To populate the research agenda, a subgroup (N = 6) of attendees extracted data from notes taken across the Deep Dive. Please refer to Table 4 for terms (and definitions) used to organize the research agenda. All unique actions were extracted from each session note that covered at least one super-cluster. Each session note was assigned a primary and a secondary coder. Coders met monthly as a group to refine the process and discuss emergent content. The primary coder extracted action data and refined the language to represent a succinct, coherent action based on: (1) the content of the notes, (2) the context of the larger discussion in the notes, (3) discussion with colleagues (during and/or after the Deep Dive), and (4) consideration of the broader literature. The secondary coder checked data accuracy, separated or grouped actions to ensure each reflected a singular activity, and refined the action verbiage. Coders were encouraged to interpret data to generate additional actions. Coders then worked across sessions to clarify and condense the list of actions, reduce redundancy, and organize actions into priority areas (“priorities”). Given the number of actions identified for each priority area, it became clear that organizing actions within priorities by goals could offer a useful, high-level summary. Coders reviewed all actions in a priority and articulated 2–4 goals that could be achievable by a subset of actions. Each action was then labeled with its corresponding goal. Lastly, the first author synthesized all actions and associated goals within each priority, solicited input from the full authorship team, and refined the data to yield the final research agenda. Table 4 Terms and Definitions

Term	Definition	
Implementation mechanisms	Mechanisms are the processes responsible for change. In this research agenda, implementation mechanisms refer to mechanisms of implementation strategy operations. This term is used interchangeably with “mechanisms of action.”	
Research Agenda	A research agenda provides a framework or map for advancing a topic, including actions that can be taken [37]	
Deep Dive	This term represents an immersive retreat that brought together the Mechanisms Network of Expertise to identify research priorities via modified nominal group technique and group model building	
Cluster	This term refers to a group of items/statements that are grouped together in concept mapping via multivariate statistical techniques including multidimensional scaling and cluster analysis [38]	
Super Cluster	This term refers to a group of clusters that are conceptually related	
Theme	This term refers to conceptually related actions that organically emerged among the actions in the Deep Dive	
Action	This term refers to discrete ideas, activities, projects, or products, and ways to shift how research is conducted to strengthen the study of implementation mechanisms. These are the discrete elements within the Research Agenda	
Priority	This term (also referred to as “priority area”) reflects a group of actions that are conceptually related and unique from other priority areas within the Research Agenda	
Goal	This term refers to an outcome that could come about should related actions be executed in the Research Agenda	

Results

Table 5 presents the refined list of the MNoE-generated actions, organized by priorities and goals, into a research agenda to advance the study of implementation mechanisms. Although not required per our method, priorities reflected all five super-clusters from the concept mapping solution. In addition, priorities emerged specific to Engagement (of policy and practice communities, as well as funders) and Growing the Field in terms of capacity (number of knowledgeable researchers) and skills specific to studying mechanisms. The MNoE generated 150 unique actions across 10 priority areas (range: 11–19 actions per area). These actions included a mix of discrete activities, projects, or products, as well as ways to shift how research is conducted to center implementation mechanisms. Wherever possible, citations are included in the table to offer exemplars that represent the intention behind the possibility. Table 5 Research agenda to advance the study of implementation mechanisms

Priority & Goals	Actions	
Accumulating Knowledge	
Accumulate Knowledge within and across Disciplines (N = 19)

Goal 1: Synthesize existing knowledge from completed studies

Goal 2: Coordinate conduct of ongoing and future studies

	(1.1) Conduct a systematic review of how implementation strategies are operationalized and associated putative mechanisms [39]

(1.2) Conduct systematic or integrative reviews of discrete implementation strategies with a focus on mechanism articulation or activation and factors influencing equitable impact [40–42]

(1.3) Conduct a meta-analysis of mechanisms to explore effect sizes [43]

(1.4) Develop a case series of completed studies that use the same analytic strategy to examine mechanisms; explore relevant issues (e.g., measurement, implementation fidelity [44–46]; cultural relevance)

(1.5) Test discrete implementation strategies in diverse contexts while assessing mechanisms for confirmation/replication, or manipulate the same mechanisms in different settings to test the variation in activation toward generalizability [47]

(1.6) Empirically examine how context, and specifically cultural context, influence mechanism activation to ascertain if different mechanisms are needed for activation

(1.7) Determine whether strategy-mechanism pairings are universal, or if pathways vary across contexts (e.g., service system, community) and how strategy operationalization occurs (i.e., form versus function)

(1.8) Determine which implementation strategies are needed for specific types of interventions (e.g., practice & feedback needed for psychotherapy implementation) similarly across contexts

(1.9) Triangulate across studies to curate and prioritize mechanisms that are most potent in resolving certain barriers including those preventing equitable access to care

(1.10) Consolidate examples of strategy-barrier mismatches and highlight alternative pairings

(1.11) Synthesize knowledge of putative mechanisms across studies using qualitative methods [48]

(1.12) Synthesize knowledge of which strategies have evidence of resolving specific determinants across diverse operationalizations

(2.1) Identify factors that affect mechanisms and their activation (e.g., context, level of analysis/where in the pathway we are focusing, degree of mutability)

(2.2) Generate a mechanism-focused study repository to enable sharing of information: share measures of mechanisms; report impact/effect of strategies focusing on how and why; provide exemplar studies across a range of study designs and contexts, especially those that engage community/practice partners from diverse groups

(2.3) Create meta-labs [33] for widely used implementation strategies to pool samples for mediation analysis; strategies to examine might include audit and feedback [34]; learning collaboratives, and practice facilitation

(2.4) Gather observational data across a large number of studies for simulation model development to inform what to focus on for measurement and how to measure

(2.5) Conduct more simulation studies to understand how multifaceted strategies exert their effects

(2.6) Examine behavior change principles in different contexts and uncover variability in predictive validity [49]

(2.7) Demonstrate how implementation mechanisms drive public health benefits by resolving barriers and realizing implementation outcomes

	
Prioritize Mechanism Research & Incorporate Other Knowledge (N = 11)

Goal 3: Draw on approaches taken by other disciplines

Goal 4: Collaborate with experts from other disciplines

	(3.1) Consider the following fields for their definitions and their specific approaches to studying mechanisms (list is non-exhaustive):

• Behavioral psychology – principles of learning and behavior change

• Clinical psychology – case conceptualization

• Engineering – practical language of inquiry

• Epidemiology – directed acyclic graphs

• Medical anthropology – ethnographic methods

• Systems science – network simulation [50]

• Marketing – communicating complex phenomenon

• Systems engineering – design, integrate, and manage complex systems

• Ecosystems science – complex systems with dynamic contexts

• Political science—process tracing, agent-based modeling

(3.2) Look across disciplines to fields in which one entity helps another do something differently to understand how they explore mechanisms (e.g., governance, natural resources, education), including their approaches to pursuing equity

(3.3) Bring in methods from multiple disciplines for articulating and representing mechanisms (e.g., directed acyclic graphs (DAGs) [17], causal pies, quality improvement 5 why’s, root cause analysis, user story mapping, logic model, key driver diagrams, and other methods for depicting mechanisms visually [51])a

(3.4) Acknowledge that systems science might clarify where the biggest opportunities for change exist; Make clear for what purpose (good for multilevel models, can inform pragmatic decisions, can identify moderators) and the limitations of these methods (curse of dimensionality)

(4.1) Convene a multidisciplinary group (e.g., a CTSA workgroup on advancing implementation mechanisms) to identify similar concepts and complementary methods, and accumulate knowledge

(4.2) Conduct mechanisms-focused workshops led by experts from adjacent/different fields to introduce “new” ways to study mechanisms in implementation science

(4.3) Create affinity group for people who are using systems science and complexity theory methods such as agent-based modeling, system dynamics, and social network analysis

(4.4) Model a conference/workshop after the Veteran Affairs State-of-the-Art (SOTA) conference that develops research priorities by engaging multidisciplinary teams

(4.5) Engage mechanism experts from peripheral disciplines as partners in research

(4.6) Host a "hackathon" or Shark Tank with teams working on same problems using the same data to surface novel understandings about mechanisms

(4.7) Develop 1-page documents to introduce to implementation science to different fields where mechanisms are a strong focus and inspire bi-directional learning

	
Methods & Design	
Overcome design challenges & Innovate methods (N = 18)

Goal 5: Guide selection and refinement of study designs

Goal 6: Enable measurement of pertinent and feasible data

Goal 7: Leverage strengths of different research methods

	(5.1) Surface range of designs (build a compendium), their trade-offs [52–55], and offer guidance for which to use when depending on research question, stage of implementation, readiness/resources of the context, including: Agent-based modeling [56], pragmatic trials, QI/PDSA, micro-randomized trials, AB designs, single case experimental designs, SMART design, coincidence analysis, QCA, constant comparison analysisa

(5.2) Generate design guidance to center mechanisms in complex contexts & long causal chains

(5.3) Emphasize how SMART designs should center the study of mechanisms

(5.4) Simplify the causal chain from implementation strategies to outcomes to promote harmony in design and synthesis across studies

(5.5) Employ comparative case study designs using qualitative or ethnographic methods to create hypotheses about complex pathways

(5.6) Offer benchmarks for meaningful effect sizes for strategy-mechanism-outcome pathways

(6.1) Generate tools to help model direct and indirect effects, and systematic evaluation of mechanisms

(6.2) Generate a methods paper that elucidates what data (e.g., emails, team process meeting notes) and approaches can support the study and analysis of mechanisms across diverse contexts [57]

(6.3) Prioritize testing for signal (i.e., early indicator offering a sign that something is/is not working) in shorter time frames with reasonable level of evidence [58]

(6.4) Articulate hypothesized and, eventually, empirical “go/no-go” indicators of mechanism activation based on theory-informed hypotheses

(6.5) Promote study of intermediate outcomes (i.e., outcomes more proximal to the implementation strategy than implementation outcomes)a

(6.6) Overcome limitations of quantitative assessment of relevant factors to better account for time or when change occurs

(6.7) Track contextual changes and document interdependencies between strategy, context, and outcomes to isolate mechanisms

(7.1) Establish when qualitative inquiry is needed for surfacing mechanisms versus quantitative “activation” evaluation [59]

(7.2) Leverage qualitative inquiry in a formative manner to understand why something did or did not work (i.e., explore mechanisms) and confer with relevant theory; consider issues related to structural racism and inequities in service access, for example

(7.3) Conduct qualitative work focused on implementation mechanism hypothesis generation by surfacing the lived experience of diverse community partners

(7.4) Conduct powered, multilevel tests of mechanisms if initial studies provide signal of impact

(7.5) Overcome the limitations of frequentist statistical methods by leveraging Bayesian methods, practical significance standards

	
Conceptualization & Measurement	
Improve Measurement (N = 13)

Goal 8: Develop grounded and generalizable measures

Goal 9: Recommend best practices for measurement

Goal 10: Clarify and troubleshoot measurement challenges

	(8.1) Align measures to implementation science-relevant elements of the Theoretical Domains Framework and Behavior Change Wheel, for example

(8.2) Develop brief measures that allow for repeated, real-time assessment of changes in mechanisms

(8.3) Use culturally responsive measures that are mechanism-specific but generalizable across multiple studies

(9.1) Employ methods less prone to reporter bias (e.g., ethnography/observation, secondary reporters) and to measurement bias (e.g., objective data capture)

(9.2) Generate passive data collection approaches for continuous monitoring of mechanisms

(9.3) Utilize ecological momentary assessment [60] or low-burden near-continuous assessments

(9.4) Design robust data capture systems to identify unanticipated mechanisms with a focus on diverse populations and contexts

(9.5) Elucidate and measure proximal outcomes, as mechanisms are difficult to assess directly

(10.1) Build a living repository of measures that are consistently used across studies, tagging each measure for its relevance to strategies, levels, settings, frameworks/theories, etc

(10.2) Offer guidance for measurement across multiple levels of analysis and varying timepoints of change

(10.3) Clarify mechanisms that can be measured directly versus via proximal outcomes

(10.4) Identify mechanisms (at multiple levels) for which we do or do not yet have measures

(10.5) Work through measurement issues across teams in a meeting series

	
Strategy, Mechanism, Determinant, Outcome Linkages	
Provide guidance for specifying mechanisms (N = 15)

Goal 11: Determine definitions and reporting standards

Goal 12: Employ cross-context and multi-level approaches

	(11.1) Generate an initial repository of putative mechanisms and implementation strategies with an emphasis on the critical need for it to be continuously updated with emerging evidencea

(11.2) Build out the ERIC compilation to link strategies to determinants via mechanismsa

(11.3) Establish a list of questions and associated criteria to derive/articulate mechanismsa

(11.4) Develop clear, field-specific criteria for establishing implementation mechanisms

(11.5) Examine A Guideline for Reporting Mediation Analyses of Randomized Trials (AGReMA) guidelines to see if they can be enhanced to support the reporting of implementation mechanisms

(11.6) Produce guidance to ensure validity, equity, and value of implementation mechanism research

(12.1) Prioritize precise definitions and simple language decisions to make room for different terms across fields. Within this, differentiate mechanisms (theory of solution) and determinants (theory of the problem) and similarly, the causal mechanism for determinant (why does it exist?) versus causal mechanism of strategy (how does it work?)a

(12.2) Assess mechanisms of fundamental behavioral phenomena that are likely more stable (e.g., improved knowledge after training), and thus may be applicable across very different implementation settings

(12.3) Use theory to guide selection of putative mechanisms

(12.4) Specify not only whether mechanisms vary by context, but how they might be activated differently across contexts with causal pathway diagrams

(12.5) Clarify the benefits and opportunities for prospective versus retrospective mechanism evaluation/analysis

(12.6) Consider the relevance of mechanisms at the community or policy levels and how they might support or break down inequitable structures

(12.7) Conduct a meta-ethnography of case studies (such as those in organizational literature) to identify common mechanisms that are referred to differently

(12.8) Use case-based examples to identify mechanisms, including walking through a story and choices made about mechanism and design along the way and why

(12.9) Use the salvage strategy: Walk through an example where implementation failed and explore different decisions

	
Theory, Causality, & Context	
Increase Focus on Theorizing (N = 12)

Goal 13: Capitalize on opportunities to incorporate theory

Goal 14: Provide examples and guidance on incorporating theory

	(13.1) Articulate the role of causal theory and program theory in implementation sciencea

(13.2) Develop/revise extant implementation theories to better represent mechanisms

(13.3) Consider whether a single theory spanning individual and organizational levels is relevant and applicable

(13.4) Engage a group in reviewing common theories to identify which strategies could realize theoretical impact, including to address structural inequities

(13.5) Explore data through theoretical lens, develop or modify theory to explain the data, then test theory with new real-world data

(13.6) Making the notion of timing more explicit in the theory of change

(13.7) Describe theoretical justification for mechanistic pathway that links back to determinant frameworks (frameworks → theories)

(14.1) Make theory explicit by requiring it in the strategy design phase

(14.2) Clarify how implementation mechanisms are “theoretical” or how theory is used to articulate mechanisms

(14.3) Provide guidance about assessing the quality of theories and best practices for “full” theory integration in implementation

(14.4) Prepare guidance for how to choose or develop a theory and how theory complements other forms of knowledge, emphasizing what value theory offers

(14.5) Surface examples of theories from other fields (e.g., epidemiology) that include mechanisms

	
Engagement	
Engage the Policy & Practice Community (N = 12)

Goal 15: Engage partners in mechanism identification, validation, and testing

Goal 16: Use innovative methods to obtain practice-based data and confirm theory

	(15.1) Generate a list of questions to assess mechanisms with partners and minimize jargon (e.g., what happened next?)a

(15.2) Develop clear language so the purpose resonates with practice community partners, and they can contribute their knowledge/perspective

(15.3) Use plain language [61] when describing, specifying, and measuring mechanisms to help with translation to the practice community and across contexts

(15.4) Draw from community-partnered participatory research [62], participatory action research, user centered design, group model building, and related methods to develop best practices, methodological examples, and possibly a toolkit for how to engage community partners in elucidating/testing mechanisms

(15.5) Develop strategies to excite practice partners from diverse communities to study mechanisms

(15.6) Clarify which community partners can/should contribute to mechanism articulation and testing

(15.7) Ensure sufficient funding for partners to play an active role in tracking and communicating their “data” about mechanisms [63]

(15.8) Engage the practice community to obtain feedback on the relevance of implementation mechanisms for their contexts

(15.9) Engage the practice community in drawing workflows or causal pathways (e.g., using Group Model Building)

(15.10) Assess practitioners’ experience with why a strategy has worked to inform theorizing and explore generalizability

(15.11) Honor that partnership engagement may be a mechanism

(16.1) Partner with diverse community groups for mechanism generation, then explore alignment with theory

(16.2) Make Implementation Mapping [64, 65] scalable and relevant to non-specialist partners (and/or the practice community)

(16.3) Consider using cognitive walkthroughs [66] and user-centered design [67] to facilitate clinicians' sharing of causal information, while being careful not to lead clinicians toward coming up with a cause for their actions when there is no specific cause

(16.4) Obtain logs from clinicians that describe concerns; receive feedback in small chunks (e.g., clinician centric inbox for implementation feedback); analyze justification data to get at causal relationships (e.g., "this order was placed per practice guidelines" or "this order was recommended by an expert colleague")

(16.5) Compare the “ground up” elucidation of mechanisms to the “top down” or theory-driven approach; what do we learn from each?

(16.6) Articulate processes for integrating empirical and practical knowledge when operationalizing mechanisms

	
Engage Funders & Need for New Funding (N = 17)

Goal 17: Make the study of mechanisms a funding priority

Goal 18: Grow mechanism expertise among peer reviewers

Goal 19: Create funding models to support mechanism-focused research

	(17.1) Create single calls for proposals and/or dedicated funding streams for mechanism-focused research studies or administrative supplements

(17.2) Advocate for prioritizing mechanisms in implementation science focused funding announcements

(17.3) Ensure ability to study mechanisms and their role in increasing equity is within budget limits; consider de-prioritizing focus on evidence-based intervention impact on patient outcomes when an evidence base is strong

(17.4) Encourage grant reviewers and funders to prioritize mechanism evaluation in implementation trials

(17.5) Clarify with policy makers and funders that mechanisms-focused research can lead to a more parsimonious and efficient implementation approach and to reproducibility

(18.1) Offer an orientation to mechanism-focused research for grant reviewers

(18.2) Present workshops on mechanisms at implementation science conferences to educate the next generation of reviewers

(18.3) Conduct mock study section reviews with implementation mechanism researchers

(18.4) Create guidelines for assessing the rigor of studies intended to examine mechanisms

(18.5) Host events about mechanism-focused research with program officers as respondents or panel moderators

(18.6) Build Board of Experts with national grant funders on implementation mechanisms

(18.7) Encourage mechanism-savvy implementation scientists to sit on study sections

(19.1) Create grant mechanism in which to test theories and putative mechanisms on a small scale, with the ability to adjust the study/design if the signal is not detected as expected in initial tests

(19.2) Prioritize grant funding to pay community partners, fund their time, and incentivize their involvement in mechanism exploration

(19.3) Demonstrate for funders the time it takes to engage in ethnography, which is a promising approach to unearthing mechanisms

(19.4) Increase project length to create necessary time for deep mechanistic evaluation (could be focus of renewal)

(19.5) Fund a coordinating center to integrate findings across numerous studies examining implementation strategy mechanisms

	
Growing the Field	
Build Capacity (N = 17)

Goal 20: Build resources for identifying and understanding mechanisms

Goal 21: Offer trainings on what mechanisms are and how to study them

	(20.1) Prepare guidelines or criteria for how to generate or test an implementation strategy causal pathway

(20.2) Create guidance/guidelines for how to choose the most appropriate outcome for a given mechanism

(20.3) Produce guidance surfacing related terms and offer conceptual distinctions

(20.4) Produce a decision tree, roadmap, or conceptualization tool to ascertain if something is or is not a mechanism

(20.5) Develop a tool (decision tree, flow chart) for disentangling the intervention from implementation strategies

(20.6) Clarify how/when to adapt the intervention versus modify the implementation strategya

(20.7) Clarify ways to unpack how multi-level, multi-component implementation strategies work

(20.8) Provide guidance on when to change the strategy for the context versus change the context using the strategy based on our understanding of mechanisms

(20.9) Clarify how to understand misalignment in study findings (e.g., mechanism not activated but outcome achieved, or mechanism activated but no improvement in outcome) and the critical role of context

(20.10) Clarify how to surface a mismatch or misalignment between strategies and barriers to reveal untargeted or unnecessarily targeted mechanisms

(20.11) Clarify how to know when tailoring strategies is needed and when changes are extensive enough that a strategy becomes a new strategy with unproven effectiveness

(20.12) Clarify how to glean generalizable knowledge about mechanisms from quality improvement and case studies

(21.1) Develop asynchronous trainings on implementation mechanisms (what they are, why they are important, how to study them)a

(21.2) Develop training that teaches people how to specify links in causal chains by creating web-based templates with graphical depictions for how mechanisms relate to implementation problems with instructions (e.g., terms and definitions, questions/prompts)

(21.3) Train study teams to regularly reflect about why and how an implementation is (or is not) working, including examining equity in the context and/or population

(21.4) Generate a workgroup to support a series of training grants (e.g., F32s) that study mechanisms

(21.5) Host a Big Pile of Data Summit; showcase un-utilized data from grants and pair with postdocs with shared mentoring structure

	
Emphasize Dissemination (N = 17)

Goal 22: Write mechanism-focused manuscripts

Goal 23: Partner with journals to promote mechanisms

Goal 24: Establish forums for mechanism-focused dialog

Goal 25: Generate broader interest in mechanisms

	(22.1) Publish annual updates to reviews of implementation mechanisms in implementation science journals

(22.2) Publish commentaries in response to studies examining mechanisms to encourage cross-disciplinary/cross-team dialogue

(22.3) Write a commentary calling for continuous action, theorizing, and innovating as the study of mechanisms evolves

(22.4) Write a paper that describes the non-exhaustive nature of lists (of determinants, for example) and the importance of articulating putative mechanismsa

(22.5) Write a “Mechanisms Made Too Simple” papera

(22.6) Create an Implementation Mechanisms 101 reference document

(22.7) Develop a manuscript describing how understanding implementation strategy mechanisms can address health equity by ensuring that strategies are effective in multiple contexts and with multiple populations

(23.1) Invite journals to encourage the study of implementation mechanisms

(23.2) Develop a new paper type: From Failure to Wisdom

(23.3) Create a "Less is More" column in journals that feature the study of mechanisms

(23.4) Include call out boxes or columns within papers that feature the story of mechanism activation

(24.1) Host multi-day conference with national funders (e.g., National Institutes of Health) that is a publicly streamed event with engaged note takers to develop a consensus statement about the import, implications, and how to study mechanisms

(24.2) Host annual pre-conference meetings on implementation mechanisms for several years to share latest developments

(25.1) Share examples of how mechanistic research can be used to drive pragmatic decisions: which strategy for whom, when and in what contexts/conditions?

(25.2) Clarify why people want to know about mechanisms in promoting the study of mechanisms—develop the “why” with community partners

(25.3) Emphasize the benefits to a reductionist approach for science and practice

(25.4) Bring communication experts to build capacity for talking across partners about mechanisms

	
“Actions” refer to discrete actions, ideas, or products. “Priorities” refer to higher-level directions for future research and associated activities. Many actions are relevant to more than 1 priority but are only listed once 

aIndicates actions that are in progress, led by MnoE members coming out of the 2021 Deep Dive. The term “context” is used throughout the table. Context includes key features of the environment in which the work is immersed and which are interpreted as meaningful to the success, failure, and unexpected consequences of the intervention(s), as well as the relationship of these to partners

Here, we briefly describe each priority and the types of associated actions. Table 5 presents additional details—including goals that each priority area might achieve. The first set of actions are directly aligned with the concept mapping solution super-clusters.Accumulate Knowledge within and Across Disciplines includes 19 actions that feature specific systematic reviews and meta-analyses, for example, and research questions that would drive this type of evidence synthesis (e.g., determine whether mechanisms are universal, or if variation across contexts is observed).

Prioritize Mechanism Research and Incorporate Other Knowledge includes 11 actions that would bring together transdisciplinary teams across fields where mechanisms are likely a prominent area of research, such as psychology and epidemiology.

Overcome Design Challenges and Innovate Methods includes 18 actions where new methods are needed (e.g., modeling time in quantitative assessment to isolate specific mechanisms) and identifies underused methods offering specific value (e.g., comparative case studies to generate hypotheses about complex mechanistic pathways).

Improve Measurement includes 13 actions, such as pragmatic approaches for objective data collection and those that capture lived experiences—an essential measurement component to understand when disparities might be addressed or exacerbated through implementation research and practice.

Provide Guidance for Specifying Mechanisms includes 15 actions reflecting mostly tools/aids to improve researchers’ approach to examining mechanisms (e.g., a list of questions and criteria for articulating mechanisms).

Increase Focus on Theorizing includes 12 ways to capitalize on developing, incorporating, and refining theory into mechanistic research to better characterize mechanisms (e.g., make theory explicit in the strategy design phase).

The emergent actions related to Engagement and Growing the Field provide further priorities for action.Engaging the Policy and Practice Community includes 12 actions or methods for understanding the perspective of these potential partners (e.g., cognitive walkthroughs, plain language, Implementation Mapping [64, 65]) and questions about when to include whom and how (e.g., compare “ground up” elucidation of mechanisms to the “top down” or theory-driven approach).

Engaging Funders and the Need for New Funding includes 17 actions to garner interest and expertise (e.g., mock study sections) and inspire novel use of new grant mechanisms (e.g., administrative supplements, trainee funding mechanisms).

Build Capacity includes 17 actions to offer clarification/guidance (e.g., how to understand conceptual/theoretical misalignment between strategies, mechanisms, and outcomes) and avenues to build the field’s capacity (e.g., postdoctoral training grants).

Emphasize Dissemination includes 17 actions like specific manuscript ideas, ways to engage journals to support mechanism-focused manuscripts, forums to host this dialogue, and other methods for generating broader interest beyond academia. Such methods are intended to foster iterative and collaborative advancements in mechanism research across interdisciplinary groups.

Discussion

This paper articulates opportunities to advance the study of implementation mechanisms in a research agenda organized by priorities for the field and specific actions to advance those priorities. Actions range from those that can be acted upon now by way of shifting the research paradigm (e.g., always articulate mechanisms when designing implementation strategies) to those that may need targeted funding and specialized knowledge/expertise (e.g., conduct sufficiently powered, multilevel tests of mechanisms with multidisciplinary input). What follows is a discussion of each priority area by highlighting actions (represented by A# corresponding to Table 5) or exemplars organized by goals (represented by G# in Table 5). These actions were articulated by the MNoE (a group of experts) as ways to address challenges identified in their prior concept mapping work.

Accumulating knowledge

With 100 + discrete implementation strategies and behavior change techniques from which to choose [12–16], balanced with evidence that rarely will a single strategy suffice in realizing sustained and robust change [68, 69], accumulating basic knowledge about how strategies work is crucial. Although the MNoE acknowledged that a starting place could be to curate a list of implementation mechanisms, they also emphasized that there is a risk in overreliance on static lists and frameworks at the expense of theorizing or broader critical thinking [70, 71] (A22.4), particularly where evidence for strategy functioning and causal processes is thin. To this end, the MNoE prioritized knowledge synthesis across completed studies (G1) and coordination of future studies (G2). Specifically, the MNoE prioritized accumulating knowledge to yield practical information such as: (i) which strategies are needed for specific types of interventions across most contexts (e.g., ‘practice & feedback’ needed for evidence-based psychotherapy implementation) (A1.8); (ii) which strategies hold promise in addressing certain barriers across diverse operationalizations [72, 73] (A1.12) (e.g., educational training to address knowledge deficits); (iii) whether strategy-mechanism pairings are universal, or if and how pathways vary across contexts (e.g., service system, level of actor, community, culture) or strategy operationalization (i.e., form versus function [74, 75]) (A1.7).

Not only are individual studies needed to test strategy pathways to yield this information (P1.5), which could be done in practical and efficient simulation studies (A2.5), but evidence syntheses are needed to curate this practical information (A1.1, 1.2, 1.3, 1.6, 1.8, 1.9, 1.10, 1.11, 1.12). These possible actions are ripe for those interested in secondary data analysis. Alternatively, meta-laboratories (meta-labs) [76] offer an approach to testing implementation strategies at scale with the possibility of pooling samples for mediation analyses (A2.3). Meta-labs can harness practical implementation efforts in health systems, for example, where different operationalizations of commonly deployed strategies can be examined using harmonized implementation process, service, and patient-level health outcomes contained in electronic medical records. Grimshaw and colleagues are pioneering the meta-lab by convening subject matter experts to accumulate evidence about audit and feedback [76, 77]. It is unclear whether existing grant funding mechanisms can accommodate the infrastructure necessary for multi-study, global coordination, and data sharing in such efforts (A19.5).

To accumulate knowledge efficiently, the MNoE recommended a mechanism-focused study repository for sharing information, evidence, and methods (A2.2). A repository could be used to share measures of mechanisms for cross-study testing and comparison; report impact/effect of strategies with how and why data; and provide diverse exemplar studies, especially those that engage community/practice partners. Web-based resources for implementation science are mounting (e.g., measure repositories [78, 79]), but to our knowledge, few living repositories or systematic reviews exist perhaps because they are a relatively novel methodology [80] expedited into action by the COVID-19 pandemic [81, 82].

Finally, the MNoE prioritized drawing on other disciplines (G3) and collaborating with experts from other disciplinary backgrounds (G4), such as scholars who study mechanisms using a multilevel perspective (A3.1). There are dozens of fields in which one entity helps another do something differently (A3.2) (e.g., governance, natural resources, education, health promotion) to integrate evidence-based interventions and strive for equity. The MNoE cautioned against our field ‘recreating the methodological wheel,’ and underscored the utility of multidisciplinary workgroups (A4.1) and workshops (A4.2). The MNoE prioritized actions to make implementation science more accessible (e.g., 1-page documents such as an SBAR: Situation, Background, Assessment, Recommendation [83] that conveys the importance of studying implementation mechanisms) to support bidirectional learning and springboard convenings. A recent commentary expressed concern that our field borrows superficially from others when interdisciplinarity or trans-disciplinarity is warranted [20]. Funders have recently made deep interdisciplinary collaboration a priority through opportunities such as the National Cancer Institute Implementation Science Centers [84] in which their Research Program Cores bring together numerous disciplines in a Methods Unit to test, refine, and disseminate new approaches [85] throughout 5-year awards [86].

Methods and design

The MNoE asserted the importance of overcoming design challenges (e.g., multiple multi-level mechanisms) and innovating methods (e.g., to address the time-varying nature of mechanism activation) specific to the study of mechanisms. They prioritized activities to guide selection and refinement of study designs (G5), enable measurement of pertinent and feasible data (G6), and leverage strengths of different research methods (G7) to enable establishing strategy mechanisms. For instance, much like the overview of designs that emerged from an NIH working session in 2014 [52], guidance is needed regarding when to use different designs and methods specifically for the purpose of establishing implementation mechanisms (A5.1). The MNoE suggested mechanism activation may offer an earlier signal along the causal pathway to indicate whether a strategy is working as hypothesized (A6.3). Designing trials for early signal testing demands methodological guidance regarding what constitutes reasonable levels of evidence (go/no-go indicators) (A6.4), how to time mechanism measurement or measure intermediate outcomes (A6.5), and how to pivot if the signal is not detected, particularly in a grant-funded study where adapting/changing the implementation strategy (i.e., independent variable) could be deemed a protocol deviation [58]. Fortunately, methods experts are beginning to apply adaptive trial designs that directly answer this call [87]. The MNoE also acknowledged the power of qualitative methods [88–91] to inform theory development and surface candidate mechanisms (A7.1) and to offer formative evidence for why a strategy did not work as intended (A7.2). The MNoE highlighted that qualitative methods provide richness, unique insights, and critical perspectives of those with lived experience [57, 89–92]. Engagement with diverse partners will yield more specific, contextualized, and experientially-informed hypotheses of how strategies are working (A7.3) that may be more acceptable and appropriate for a given context and innovation compared to researcher-derived hypotheses. For example, a secondary analysis of a large implementation trial of measurement-based care revealed no significant mediators from the quantitative data but identified important candidate mechanisms from qualitative analyses [93].

Conceptualization and measurement

In general, great strides have been made to enhance the quality, access, and utility of measurement in implementation science through systematic reviews, guidance documents, and web-based repositories [78, 79, 94]. The MNoE prioritized actions specific to studying mechanisms to develop grounded and generalizable measures (G8), recommend best practices regarding measurement (G9), and clarify ongoing measurement challenges (G10). The MNoE articulated the need to deploy measurement methods that allow for multiple, real-time assessments to detect changes that unfold over time (A8.2), as mechanisms are hypothesized to be activated at varying rates by population and context. The MNoE elevated the possible use of passive data collection approaches for continuous monitoring of mechanisms (A9.2), ecological momentary assessment (EMA), or lower-burden, near-continuous assessments to track changes in mechanisms and determinants (A9.3). As an example, EMA was used to identify predictors of noncompliance of event-based reporting of tobacco use [95]. Although this example is implementation-adjacent, it reveals how underused approaches like EMA can overcome measurement challenges critical to studying mechanisms such as timing (e.g., multiple, repeated measures) and self-report (e.g., bias, memory).

Strategy, mechanism, determinant, outcome linkages

The MNoE was initially organized to include a subset of scholars who focused on understanding the linkages between strategies, mechanisms, determinants, and outcomes [30]. Recognizing that strategies are too often disconnected from determinants [96, 97] and overpromising outcomes [69], the MNoE articulated the role of mechanisms in the causal pathway in terms of how a strategy exerts its effects on target outcomes by overcoming barriers [98]. The MNoE prioritized defining mechanisms as distinct from determinants and establishing reporting standards for mechanisms research (G11) to support deployment of cross-context and multilevel approaches (G12). The MNoE remarked on this as critical “foundational work” for scientific and practical progress to be made. For instance, the MNoE encouraged consideration of which strategies (from compilations such as Expert Recommendations for Implementing Change (ERIC) [13] and Effective Practice and Organization of Care (EPOC) [99]) have evidence of activating specific mechanisms to resolve particular barriers and achieve specific outcomes. Such foundational knowledge of discrete strategies would be instrumental in designing a practical implementation plan, but no synthesis or repository exists to our knowledge (A11.1), although a 2016 review does offer preliminary evidence on a subset of strategy-mediator pairings [21]. One activity to contribute this knowledge may be the “salvage strategy” [100, 101] in which journals or conferences feature implementation failures and invite exploration of mechanism activation or lack thereof [17] (A12.9). The MNoE also prioritized using theory to guide articulation of putative mechanisms (A12.3) and the examination of mechanisms across diverse contexts to explore how mechanisms might be activated differently or over a different timeframe across contexts, populations, or interventions (A12.4). Moreover, the MNoE acknowledged the potential to hyperfocus on intrapersonal mechanisms of behavior change, which has a mounting evidence base [16, 72, 102]. To complement this individually focused work, the MNoE explicitly prioritized exploring mechanisms at aggregate levels of analysis that are less studied (e.g., community or policy levels), but where structures should be targeted to improve (A12.6) equitable outcomes [32, 50, 52, 58, 75, 87, 103, 104].

Theory, causality, and context

Because implementation science is a convergence of many disciplines, there are relevant classic theories (e.g., from social psychology, business, economics, education, anthropology) that articulate mechanisms [105]. Most utilized are frameworks, from which the theoretical underpinnings that depict relationships among constructs and enable prediction through propositions are absent, leaving a list of measurable factors organized by conceptual coherence, as in the case of the Theoretical Domains Framework [106] and the updated Consolidated Framework for Implementation Research [107]. Kislov et al. [108] wrote about the importance of theorizing as a process that could enable implementation scientists to bidirectionally inform and learn from empirical data to test and advance generalizable knowledge and theory working at the mid-range level to develop and refine grand theories. More recently, Meza and colleagues [109] attempted to make theorizing more accessible to researchers, and although they use theorizing about determinants as their use case, they name mechanisms as a critical component of causal chains that explain how an implementation initiative is successful. Toward this goal, the MNoE prioritized activities that would incorporate theory (G13) through examples and guidance (G14). Actions included differentiating causal theory from program theory (A13.1), modifying implementation science “grand” theories to better represent mechanisms (A13.2), and making the notion of timing more explicit in the theory of change (A13.6). Consistent with the above-mentioned calls to prioritize theory, the MNoE prioritized guidance to choose relevant theories for study planning (A14.4), to fully integrate theory in an implementation study of mechanisms (A14.3), and to clarify how theory is used to articulate mechanisms (A14.2).

Beyond the five priority clusters initially identified in the concept mapping of challenges stymying the field, two new priority clusters of actions emerged through MNoE discussions: Engagement and Growing the Field. These priorities reflect critical areas of work to advance the study of implementation mechanisms. The Engagement cluster represents actions that, if prioritized early, would amplify the impact of actions in other clusters. Growing the Field actions are foundational and/or underpin the work of the other clusters, which might not be possible otherwise.

Engagement

In terms of Engagement, the MNoE thought it critical to engage the policy and practice community, as well as funders of implementation science. The MNoE emphasized that the policy and practice communities are critical to establishing mechanisms, yet this area of science can feel obscure and pedantic to those communities. Funders were identified as a separate target for engagement because many of the prioritized actions do not fit neatly within traditional funding mechanisms.

The MNoE articulated priorities for engaging policy and practice partners in mechanism identification, validation, and testing (G15) and in using methods to obtain practice-based data and confirm theory (G16). The MNoE recommended plain-language mechanism definitions and de-jargonized questions for identifying mechanisms with community partners to help scientific teams learn from their perspectives (A15.1). Plain language was repeatedly emphasized because the term “mechanisms” itself may limit idea generation or perceptions of applicability as it tends to surface mechanical or biological underpinnings (A15.2, 15.3). The MNoE saw the policy and practice communities, broadly construed, as central to unearthing putative mechanisms and generated actions for facilitating their engagement, including motivating them to study mechanisms (A15.5), supporting them to collect and track data on mechanisms (A15.7), providing feedback (A15.8), and constructing causal pathways (A15.9). For instance, group model building presents a directed approach to engaging participants in articulating implementation mechanisms [110]. There are several more general frameworks, models, and approaches that can guide this kind of policy and practice community engagement, including community-based participatory research [111], community partnered participatory research [112], participatory action research [113], integrated knowledge translation [114], and user-centered design [67, 115].

The MNoE articulated goals for engaging funders including emphasizing the study of mechanisms as a priority (G17), growing mechanism expertise (G18), and considering new funding models to support mechanism-focused research (G19). The MNoE suggested that it might be important to clarify, or confirm, that mechanism-focused research can lead to more parsimonious and efficient implementation approaches and reproducibility (A17.5). To this end, the MNoE surfaced the possibility of using scientific administrative supplements for mechanism data testing (A17.1) and making the study of mechanisms an explicit priority in funding opportunities (A17.2). To ensure mechanism evaluation fits within grant budget limits, the MNoE suggested deprioritizing patient and clinical outcomes when the intervention’s efficacy and/or effectiveness is robust and adaptation is minimal (A17.3).

The MNoE highlighted the importance of ensuring that grant reviewers are familiar with implementation mechanisms and can critically review grant proposals on these topics. To grow the capacity of reviewers (and the extramural community more broadly), the MNoE proposed specialized training for reviewers or the reviewer pipeline (A18.1), including conference workshops (A18.2) and mock study sections that center applications proposing implementation mechanisms research (A18.3). The MNoE envisioned a guideline document that would support assessing a study proposal’s plan to evaluate implementation mechanisms and scaffold learning key elements for mechanisms testing for those writing grant applications (A18.4).

Finally, the MNoE articulated several ideas for funding opportunities or suggested elements to emphasize within planned/existing funding opportunities. These included funding a coordinating center to harmonize measures, create the infrastructure for data collection, and integrate findings across numerous studies examining implementation strategy mechanisms (A19.5). The MNoE also wondered about the possibility of mechanism evaluation occurring during a follow-up (e.g., renewal) grant funding period, leveraging the longitudinal nature of the evaluation and the need to engage multiple partners (A19.4). In addition to large cross-study or longer initiatives, the MNoE suggested small and nimble grant opportunities that allow for discrete strategy testing and the need to pivot if the strategy “signal” is not detected (A19.1).

Growing the field

Throughout the Deep Dive, the MNoE called for multi-pronged efforts to grow the field. The MNoE recommended resources for evaluating mechanisms that could scaffold scientists’ efforts (G20) as well as more robust training that would help scholars grow new skillsets in the study of implementation mechanisms (G21). The MNoE prioritized guidance and resources regarding topics such as: how to test a strategy causal pathway (A20.1), how to choose the most appropriate outcome for a given mechanism (A20.2), how to isolate a mechanism from other factors in a causal pathway (A20.3, 20.4), how to disentangle the intervention from implementation strategies (A20.5), when to adapt the intervention versus modify the implementation strategy (A20.6), and when to change the strategy for the context versus change the context using the strategy based on our understanding of mechanisms (A20.8). With respect to this last topic, many scholars see contextual targets that, if changed, boast greater societal benefit (e.g., consideration of social determinants of health; addressing structural racism) as being inappropriate targets for implementation scientists, unless the evidence-based intervention itself is directed at those higher levels. Yet, implementing within existing structures can exacerbate inequities. These are critical questions, answers to which would have serious practical implications if, indeed, empirical guidance could be curated. Moreover, these questions are faced by numerous research teams, making the investment in generating such guidance even more valuable. These are the types of empirical evidence and associated resources that might come from larger investments to support the study of mechanisms, such as center grant awards, from which the scientific field and practice community stand to benefit.

The MNoE also generated several actions that were characterized as training-like approaches to build capacity. These included efforts like brief, recorded, didactic sessions regarding definitional issues surrounding mechanisms (A21.1), as well as more process-oriented training on, for example, how to specify causal chains (A21.2) and how to regularly reflect on why an implementation strategy is or is not working throughout the course of a study (A21.3). A team at the IMPACT Center has begun to produce videos aligned with these actions with funding from the US National Institute of Mental Health (P50MH126219) [116]. Acknowledging that videos might not be sufficient, this team has also offered in-person workshop training followed by office hours and one year of expert consultation around causal pathway diagramming [117]. Multipronged training and consultation will be critical for capacity building in new areas like the study of implementation mechanisms. Somewhat innovative actions were also shared, including a workgroup to support a series of training grants focused on the study of mechanisms (A21.4) and a data summit in which underutilized data from grants could be made available for secondary analysis paired with postdoctoral researchers using a shared mentoring model. The sentiment was that the expertise required to advance the study of mechanisms is sparse and approaches that extend the reach to new teams and data sources would be critical.

Although several of the above suggestions function as dissemination, the MNoE articulated four specific dissemination-related goals: produce focused manuscripts (G22); partner with journals to generate new paper types (G23); establish forums for dialog (G24); and generate broad interest using strategies that reach community partners (G25). The MNoE articulated numerous manuscripts that would be helpful such as Mechanisms Made Too Simple, inspired by Curran’s article [118]. They also imagined new paper types, such as one that centered on “learning from failure with wisdom,” which would essentially unpack implementation failures with a mechanistic lens. An example of such a commentary was written by researchers (not members of the original research team) regarding a recently published null trial that appears fruitful [17], and yet another approach is to ensure that implementers have opportunities to share “salvage strategies” that make the most out of opportunities to retain rigor when unexpected events threaten to derail studies that could shed light on mechanisms [100, 101]. Finally, the MNoE underscored the importance of clarifying the “why” behind the study of mechanisms, particularly given the importance of learning from and supporting the policy and practice community. As they discussed dissemination, they surfaced a marketing problem in that not all would agree that the study of mechanisms could advance both science and practice, and some members believed this reductionist approach is misaligned with the very nature of implementation [20].

Limitations

Importantly, the MNoE may not be representative of those who could contribute and/or stand to benefit from this work. Although we made efforts to engage researchers from outside the United States (US; e.g., open attendance during a SIRC breakout; international representation in MNoE paper writing groups), the inputs and outputs of this research agenda largely reflect a US perspective. Indeed, parallel and complementary work from scholars in the United Kingdom (UK) includes an ontology of mechanisms of action in behavior change interventions that begins to address several aspects of the Research Agenda [119]. We hope readers with different perspectives will consider building from the US and UK work, for example, writing a commentary to further the dialogue and/or pursuing research that advances some of the priorities discussed above. Moreover, although some of the MNoE identify more as clinically or practically oriented researchers, the MNoE did not include policy and practice community members. Thus, it is likely that new actions across the priority clusters would have emerged if different groups were engaged in the process of generating this content. Also, the focus of this research agenda is on implementation strategy mechanisms, or the processes through which strategies exert their effects to achieve outcomes [30]. This focus overlooks contextual mechanisms, such as those surfaced through realist reviews [120]. This focus is consistent with prior work by our team [19], but can limit the field’s ability to explain how and why implementation occurs.

Conclusion

Implementation science needs to further expand from what works to how and why certain strategies work, for whom, when, and in which contexts [121]. This research agenda outlines a roadmap of concrete actions for advancing the study of mechanisms. To carry out this research agenda, concerted and strategic effort is needed. There are numerous training forums that grow implementation research capacity [122]. We hope some will highlight the priorities articulated herein, bring together transdisciplinary experts with mechanism-specific expertise, and contribute to the study of implementation mechanisms.

Supplementary Information

Supplementary Material 1.

Abbreviations

SIRC Society for Implementation Research Collaboration

MNoE Mechanisms Network of Expertise

Acknowledgements

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

In addition to federal funding, we are grateful to have received funding support from the Society for Implementation Research Collaboration, which helped make our Deep Dive meetings possible. Moreover, we would like to thank attendees at SIRC’s 2019 conference who provided input into the challenges associated with studying implementation mechanisms. Finally, we wish to thank each member of the MNoE, some of whom contributed to the development of the R13 grant proposal (Gregory Aarons, Rinad Beidas, Aaron Lyon, Brian Mittman, Byron Powell, Bryan Weiner, Nate Williams), and the rest of whom participated in one or more Deep Dive meetings, between event virtual sessions, or in the paper writing groups; see Additional File 1. Gracelyn Cruden began this work at her prior institution, Oregon Social Learning Center, and completed it at her current institution.

Authors’ contributions

CCL, HEF, BK, AS, GC, and BJP contributed to the conceptualization of the manuscript, engaged in the coding, and participated in data interpretation. CCL drafted the introduction, results, and discussion. HEF and GC drafted the method section. ARL and BA reviewed preliminary results and contributed to revisions to the results table. BJP and CCL worked the manuscript through several cycles of review by all coauthors. All authors (CCL, HEF, BK, AS, GC, BJP GAA, RSB, BSM, BJW, NJW, MF, SM, MP, LS, AW, CWB, SWS) reviewed, edited, and approved the final content of the manuscript.

Funding

This research was supported by funding from the Agency for Healthcare Research and Quality (R13HS025632), National Institute of Mental Health (P50MH126219, R01MH111950, R01MH111981, K01MH128769, R25MH080916, K01MH113806, K01MH128761), and National Cancer Institute (P50CA244432 and R01CA262325, P50CA244690).

Availability of data and materials

N/A.

Declarations

Ethics approval and consent to participate

This study was reviewed and approved by Kaiser Permanente Washington Health Research Institute’s IRB and was deemed Not Human Subjects Research.

Consent for publication

N/A.

Competing interests

Drs. Lewis and Weiner receive royalties from Springer Publishing. Dr. Beidas is principal at Implementation Science & Practice, LLC. She receives royalties from Oxford University Press, consulting fees from United Behavioral Health and OptumLabs, and serves on the advisory boards for Optum Behavioral Health, AIM Youth Mental Health Foundation, and the Klingenstein Third Generation Foundation outside of the submitted work. Dr. Aarons is a Co-Editor-in-Chief, Dr. Beidas is an Associate Editor, and Drs. Powell and Weiner are on the Editorial Board of Implementation Science, none of whom will play a role in the editorial process of this manuscript.

1 MNoE members from other countries were invited, but unable to attend due to COVID restrictions.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Shelton RC Chambers DA Glasgow RE An extension of RE-AIM to enhance sustainability: Addressing dynamic context and promoting health equity over time Front Public Health 2020 8 1 8 10.3389/fpubh.2020.00134 32117846
Shelton RC, Chambers DA, Glasgow RE. An extension of RE-AIM to enhance sustainability: Addressing dynamic context and promoting health equity over time. Front Public Health. 2020;8:1–8.32117846 10.3389/fpubh.2020.00134
2. Gustafson P Abdul Aziz Y Lambert M Bartholomew K Rankin N Fusheini A A scoping review of equity-focused implementation theories, models and frameworks in healthcare and their application in addressing ethnicity-related health inequities Implement Sci. 2023 18 51 10.1186/s13012-023-01304-0 37845686
Gustafson P, Abdul Aziz Y, Lambert M, Bartholomew K, Rankin N, Fusheini A, et al. A scoping review of equity-focused implementation theories, models and frameworks in healthcare and their application in addressing ethnicity-related health inequities. Implement Sci. 2023;18:51.37845686 10.1186/s13012-023-01304-0
3. Pullmann MD Dorsey S Duong MT Lyon AR Muse I Corbin CM Expect the unexpected: A qualitative study of the ripple effects of children’s mental health services implementation efforts Implementation Research and Practice 2022 3 26334895221120797 10.1177/26334895221120797 36504561
Pullmann MD, Dorsey S, Duong MT, Lyon AR, Muse I, Corbin CM, et al. Expect the unexpected: A qualitative study of the ripple effects of children’s mental health services implementation efforts. Implementation Research and Practice. 2022;3:26334895221120796.36504561 10.1177/26334895221120797
4. Dadich A, Vaughan P, Boydell K. The unintended negative conse- quences of knowledge translation in healthcare: A systematic scoping review. Health Sociol Rev. 2023;32:75–93.
5. Woodward EN Singh RS Ndebele-Ngwenya P Melgar Castillo A Dickson KS Kirchner JE A more practical guide to incorporating health equity domains in implementation determinant frameworks Implementation Science Communications 2021 2 61 10.1186/s43058-021-00146-5 34090524
Woodward EN, Singh RS, Ndebele-Ngwenya P, Melgar Castillo A, Dickson KS, Kirchner JE. A more practical guide to incorporating health equity domains in implementation determinant frameworks. Implementation Science Communications. 2021;2:61.34090524 10.1186/s43058-021-00146-5
6. Gaias LM Arnold KT Liu FF Pullmann MD Duong MT Lyon AR Adapting strategies to promote implementation reach and equity (ASPIRE) in school mental health services Psychol Sch 2022 59 2471 2485 10.1002/pits.22515
Gaias LM, Arnold KT, Liu FF, Pullmann MD, Duong MT, Lyon AR. Adapting strategies to promote implementation reach and equity (ASPIRE) in school mental health services. Psychol Sch. 2022;59:2471–85.10.1002/pits.22515
7. Building health equity into implementation strategies and mechanisms. Available from: https://reporter.nih.gov/search/0xJrdiFN8EeuIS8eEAKZCA/project-details/10597777. Cited 2023 Aug 9
8. Flottorp SA Oxman AD Krause J Musila NR Wensing M Godycki-Cwirko M A checklist for identifying determinants of practice: A systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice Implement Sci 2013 8 1 11 10.1186/1748-5908-8-35 23279972
Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, et al. A checklist for identifying determinants of practice: A systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:1–11.23279972 10.1186/1748-5908-8-35
9. Squires JE Graham ID Santos WJ Hutchinson AM Backman C Bergström A The Implementation in Context (ICON) framework: a meta-framework of context domains, attributes and features in healthcare Health Res Pol Syst 2023 21 81 10.1186/s12961-023-01028-z
Squires JE, Graham ID, Santos WJ, Hutchinson AM, Backman C, Bergström A, et al. The Implementation in Context (ICON) framework: a meta-framework of context domains, attributes and features in healthcare. Health Res Pol Syst. 2023;21:81.10.1186/s12961-023-01028-z
10. Birken SA Powell BJ Presseau J Kirk MA Lorencatto F Gould NJ Combined use of the Consolidated Framework for Implementation Research (CFIR) and the Theoretical Domains Framework (TDF): A systematic review Implement Sci. 2017 12 2 10.1186/s13012-016-0534-z 28057049
Birken SA, Powell BJ, Presseau J, Kirk MA, Lorencatto F, Gould NJ, et al. Combined use of the Consolidated Framework for Implementation Research (CFIR) and the Theoretical Domains Framework (TDF): A systematic review. Implement Sci. 2017;12:2.28057049 10.1186/s13012-016-0534-z
11. Powell BJ McMillen JC Proctor EK Carpenter CR Griffey RT Bunger AC A compilation of strategies for implementing clinical innovations in health and mental health Med Care Res Rev 2012 69 123 157 10.1177/1077558711430690 22203646
Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med Care Res Rev. 2012;69:123–57.22203646 10.1177/1077558711430690
12. Michie S Richardson M Johnston M Abraham C Francis J Hardeman W The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: Building an international consensus for the reporting of behavior change interventions Ann Behav Med 2013 46 81 95 10.1007/s12160-013-9486-6 23512568
Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: Building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013;46:81–95.23512568 10.1007/s12160-013-9486-6
13. Powell BJ Waltz TJ Chinman MJ Damschroder LJ Smith JL Matthieu MM A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project Implement Sci 2015 10 1 14 10.1186/s13012-015-0209-1 25567289
Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:1–14.25567289 10.1186/s13012-015-0209-1
14. Kok G Gottlieb NH Peters GY Mullen PD Parcel GS Ruiter RAC A taxonomy of behaviour change methods: An Intervention Mapping approach Health Psychol Rev 2016 10 297 312 10.1080/17437199.2015.1077155 26262912
Kok G, Gottlieb NH, Peters GY, Mullen PD, Parcel GS, Ruiter RAC, et al. A taxonomy of behaviour change methods: An Intervention Mapping approach. Health Psychol Rev. 2016;10:297–312.26262912 10.1080/17437199.2015.1077155
15. McHugh S Presseau J Luecking CT Powell BJ Examining the complementarity between the ERIC compilation of implementation strategies and the behaviour change technique taxonomy: a qualitative analysis Implement Sci 2022 17 56 10.1186/s13012-022-01227-2 35986333
McHugh S, Presseau J, Luecking CT, Powell BJ. Examining the complementarity between the ERIC compilation of implementation strategies and the behaviour change technique taxonomy: a qualitative analysis. Implement Sci. 2022;17:56.35986333 10.1186/s13012-022-01227-2
16. Marques M, Wright A, Johnston M, West R, Hastings J, Zhang L, et al. The Behaviour Change Technique Ontology: Transforming the Behaviour Change Technique Taxonomy v1. PsyArXiv; 2023. Available from: https://psyarxiv.com/vxypn/. Cited 2023 Aug 8
17. Geng EH Baumann AA Powell BJ Mechanism mapping to advance research on implementation strategies PLoS Med 2022 19 e1003918 10.1371/journal.pmed.1003918 35134069
Geng EH, Baumann AA, Powell BJ. Mechanism mapping to advance research on implementation strategies. PLoS Med. 2022;19:e1003918.35134069 10.1371/journal.pmed.1003918
18. Lewis CC Klasnja P Powell BJ Lyon AR Tuzzio L Jones S From classification to causality: advancing understanding of mechanisms of change in implementation science Front Public Health 2018 6 1 6 10.3389/fpubh.2018.00136 29404319
Lewis CC, Klasnja P, Powell BJ, Lyon AR, Tuzzio L, Jones S, et al. From classification to causality: advancing understanding of mechanisms of change in implementation science. Front Public Health. 2018;6:1–6.29404319 10.3389/fpubh.2018.00136
19. Lewis CC Boyd MR Walsh-Bailey C Lyon AR Beidas R Mittman B A systematic review of empirical studies examining mechanisms of implementation in health Implement Sci 2020 15 1 25 10.1186/s13012-020-00983-3 31900167
Lewis CC, Boyd MR, Walsh-Bailey C, Lyon AR, Beidas R, Mittman B, et al. A systematic review of empirical studies examining mechanisms of implementation in health. Implement Sci. 2020;15:1–25.31900167 10.1186/s13012-020-00983-3
20. Beidas RS Dorsey S Lewis CC Lyon AR Powell BJ Purtle J Promises and pitfalls in implementation science from the perspective of US-based researchers: learning from a pre-mortem Implement Sci 2022 17 55 10.1186/s13012-022-01226-3 35964095
Beidas RS, Dorsey S, Lewis CC, Lyon AR, Powell BJ, Purtle J, et al. Promises and pitfalls in implementation science from the perspective of US-based researchers: learning from a pre-mortem. Implement Sci. 2022;17:55.35964095 10.1186/s13012-022-01226-3
21. Williams NJ Multilevel mechanisms of implementation strategies in mental health: Integrating theory, research, and practice Admin Pol Mental Health Mental Health Serv Res 2016 43 783 798 10.1007/s10488-015-0693-2
Williams NJ. Multilevel mechanisms of implementation strategies in mental health: Integrating theory, research, and practice. Admin Pol Mental Health Mental Health Serv Res. 2016;43:783–98.10.1007/s10488-015-0693-2
22. Baker R Comosso-Stefinovic J Gillies C Shaw EJ Cheater F Flottorp S Tailored interventions to address determinants of practice Cochrane Database Syst Rev 2015 4 1 118
Baker R, Comosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. Tailored interventions to address determinants of practice. Cochrane Database Syst Rev. 2015;4:1–118.
23. Lewis CC Marti CN Scott K Walker MR Boyd M Puspitasari A Standardized versus tailored implementation of measurement-based care for depression in community mental health clinics PS 2022 73 1094 1101 10.1176/appi.ps.202100284
Lewis CC, Marti CN, Scott K, Walker MR, Boyd M, Puspitasari A, et al. Standardized versus tailored implementation of measurement-based care for depression in community mental health clinics. PS. 2022;73:1094–101.10.1176/appi.ps.202100284
24. Lewis CC Stanick C Lyon A Darnell D Locke J Puspitasari A Proceedings of the Fourth Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2017: implementation mechanisms: what makes implementation work and why? part 1 Implement Sci 2018 13 Suppl 2 1 5 29301543
Lewis CC, Stanick C, Lyon A, Darnell D, Locke J, Puspitasari A, et al. Proceedings of the Fourth Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2017: implementation mechanisms: what makes implementation work and why? part 1. Implement Sci. 2018;13(Suppl 2):1–5.29301543
25. Guastaferro K Collins LM Optimization methods and implementation science: an opportunity for behavioral and biobehavioral interventions Implement Res Pract 2021 2 26334895211054363 37089990
Guastaferro K, Collins LM. Optimization methods and implementation science: an opportunity for behavioral and biobehavioral interventions. Implement Res Pract. 2021;2:26334895211054364.37089990
26. McHugh SM, Riordan F, Curran GM, Lewis CC, Wolfenden L, Presseau J, et al. Conceptual tensions and practical trade-offs in tailoring implementation interventions. Frontiers in Health Services. 2022;2. Available from: https://www.frontiersin.org/articles/10.3389/frhs.2022.974095. Cited 2022 Nov 19
27. Riordan F, Kerins C, Pallin N, Albers B, Clack L, Morrissey E, et al. Characterising processes and outcomes of tailoring implementation strategies in healthcare: a protocol for a scoping review. HRB Open Research; 2022. Available from: https://hrbopenresearch.org/articles/5-17. Cited 2022 Nov 19
28. Landes SJ Kerns SEU Pilar MR Walsh-Bailey C Yu SH Byeon YV Proceedings of the Fifth Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2019: where the rubber meets the road: the intersection of research, policy, and practice - part 1 Implement Sci 2020 15 Suppl 3 1 5 31900167
Landes SJ, Kerns SEU, Pilar MR, Walsh-Bailey C, Yu SH, Byeon YV, et al. Proceedings of the Fifth Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2019: where the rubber meets the road: the intersection of research, policy, and practice - part 1. Implement Sci. 2020;15(Suppl 3):1–5.31900167
29. Vejnoska SF Mettert K Lewis CC Mechanisms of implementation: An appraisal of causal pathways presented at the 5th biennial Society for Implementation Research Collaboration (SIRC) conference Implement Res Pract 2022 3 26334895221086271 37091081
Vejnoska SF, Mettert K, Lewis CC. Mechanisms of implementation: An appraisal of causal pathways presented at the 5th biennial Society for Implementation Research Collaboration (SIRC) conference. Implement Res Pract. 2022;3:26334895221086270.37091081
30. Lewis CC Powell BJ Brewer SK Nguyen AM Schriger SH Vejnoska SF Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration: protocol for generating a research agenda BMJ Open 2021 11 e053474 10.1136/bmjopen-2021-053474 34663668
Lewis CC, Powell BJ, Brewer SK, Nguyen AM, Schriger SH, Vejnoska SF, et al. Advancing mechanisms of implementation to accelerate sustainable evidence-based practice integration: protocol for generating a research agenda. BMJ Open. 2021;11:e053474.34663668 10.1136/bmjopen-2021-053474
31. Proctor EK Luke D Calhoun A McMillen JC McCrary S Padek M Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support Implement Sci 2015 10 1 13 10.1186/s13012-015-0274-5 25567289
Proctor EK, Luke D, Calhoun A, McMillen JC, McCrary S, Padek M. Sustainability of evidence-based healthcare: research agenda, methodological advances, and infrastructure support. Implement Sci. 2015;10:1–13.25567289 10.1186/s13012-015-0274-5
32. Powell BJ Challenges related to the study of implementation mechanisms: A concept mapping approach 2022 San Diego, CA Conference presentation at the Society for Implementation Research Collaboration Biennial Conference
Powell BJ. Challenges related to the study of implementation mechanisms: A concept mapping approach. San Diego, CA: Conference presentation at the Society for Implementation Research Collaboration Biennial Conference; 2022.
33. Vennix JA Group model building. System Dynamics 1996 2 123 132
Vennix JA. Group model building. System. Dynamics. 1996;2:123–32.
34. Hovmand PS, Rouwette EAJA, Andersen DF, Richardson GP, and Kraus A. Scriptapedia 4.0.6. Cambridge: The 31st International Conference of the System Dynamics Society; 2013. Available from: https://proceedings.systemdynamics.org/2013/proceed/papers/P1405.pdf. Cited 2024 Sept 6.
35. Cantrill JA Sibbald B Buetow S Delphi and nominal group techniques in health services research Int J Pharm Pract 1996 4 67 74 10.1111/j.2042-7174.1996.tb00844.x
Cantrill JA, Sibbald B, Buetow S. Delphi and nominal group techniques in health services research. Int J Pharm Pract. 1996;4:67–74.10.1111/j.2042-7174.1996.tb00844.x
36. American Society for Quality. What is multivoting? 2023. Available from: https://asq.org/quality-resources/multivoting. Cited 2023 Aug 8
37. Ertmer PA Glazewski KD Developing a research agenda: contributing new knowledge via intent and focus J Comput High Educ 2014 26 54 68 10.1007/s12528-013-9076-4
Ertmer PA, Glazewski KD. Developing a research agenda: contributing new knowledge via intent and focus. J Comput High Educ. 2014;26:54–68.10.1007/s12528-013-9076-4
38. Kane M Trochim WMK Concept mapping for planning and evaluation 2007 Thousand Oaks, CA Sage
Kane M, Trochim WMK. Concept mapping for planning and evaluation. Thousand Oaks, CA: Sage; 2007.
39. Albers B Metz A Burke K Bührmann L Bartley L Driessen P The mechanisms of implementation support - findings from a systematic integrative review Res Soc Work Pract 2022 32 259 280 10.1177/10497315211042375
Albers B, Metz A, Burke K, Bührmann L, Bartley L, Driessen P, et al. The mechanisms of implementation support - findings from a systematic integrative review. Res Soc Work Pract. 2022;32:259–80.10.1177/10497315211042375
40. Miech EJ Rattray NA Flanagan ME Damschroder L Schmid AA Damush TM Inside help: An integrative review of champions in healthcare-related implementation SAGE Open Med 2018 6 2050312118773261 10.1177/2050312118773261 29796266
Miech EJ, Rattray NA, Flanagan ME, Damschroder L, Schmid AA, Damush TM. Inside help: An integrative review of champions in healthcare-related implementation. SAGE Open Med. 2018;6:2050312118773261.29796266 10.1177/2050312118773261
41. Zamboni K Baker U Tyagi M Schellenberg J Hill Z Hanson C How and under what circumstances do quality improvement collaboratives lead to better outcomes? A systematic review  Implement Sci 2020 15 27 10.1186/s13012-020-0978-z 32366269
Zamboni K, Baker U, Tyagi M, Schellenberg J, Hill Z, Hanson C. How and under what circumstances do quality improvement collaboratives lead to better outcomes? A systematic review. Implement Sci. 2020;15:27.32366269 10.1186/s13012-020-0978-z
42. Kilbourne AM Geng E Eshun-Wilson I Sweeney S Shelley D Cohen DJ How does facilitation in healthcare work? Using mechanism mapping to illuminate the black box of a meta-implementation strategy Implement Sci Commun 2023 4 53 10.1186/s43058-023-00435-1 37194084
Kilbourne AM, Geng E, Eshun-Wilson I, Sweeney S, Shelley D, Cohen DJ, et al. How does facilitation in healthcare work? Using mechanism mapping to illuminate the black box of a meta-implementation strategy. Implement Sci Commun. 2023;4:53.37194084 10.1186/s43058-023-00435-1
43. Ivers NM Grimshaw JM Jamtvedt G Flottorp S O’Brien MA French SD Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care J Gen Intern Med 2014 29 1534 1541 10.1007/s11606-014-2913-y 24965281
Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O’Brien MA, French SD, et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med. 2014;29:1534–41.24965281 10.1007/s11606-014-2913-y
44. Akiba CF Powell BJ Pence BW Nguyen MXB Golin C Go V The case for prioritizing implementation strategy fidelity measurement: benefits and challenges Translational Behavioral Medicine 2022 12 335 342 10.1093/tbm/ibab138 34791480
Akiba CF, Powell BJ, Pence BW, Nguyen MXB, Golin C, Go V. The case for prioritizing implementation strategy fidelity measurement: benefits and challenges. Transl Behav Med. 2022;12:335–42.34791480 10.1093/tbm/ibab138
45. Akiba CF Powell BJ Pence BW Muessig K Golin CE Go V “We start where we are”: a qualitative study of barriers and pragmatic solutions to the assessment and reporting of implementation strategy fidelity Implement Sci Commun 2022 3 117 10.1186/s43058-022-00365-4 36309715
Akiba CF, Powell BJ, Pence BW, Muessig K, Golin CE, Go V. “We start where we are”: a qualitative study of barriers and pragmatic solutions to the assessment and reporting of implementation strategy fidelity. Implement Sci Commun. 2022;3:117.36309715 10.1186/s43058-022-00365-4
46. Akiba CF Go VF Powell BJ Muessig K Golin C Dussault JM Champion and audit and feedback strategy fidelity and their relationship to depression intervention fidelity: A mixed method study SSM - Mental Health 2023 3 100194 10.1016/j.ssmmh.2023.100194 37485235
Akiba CF, Go VF, Powell BJ, Muessig K, Golin C, Dussault JM, et al. Champion and audit and feedback strategy fidelity and their relationship to depression intervention fidelity: A mixed method study. SSM - Mental Health. 2023;3:100194.37485235 10.1016/j.ssmmh.2023.100194
47. Brookman-Frazee L Stahmer AC Effectiveness of a multi-level implementation strategy for ASD interventions: study protocol for two linked cluster randomized trials Implement Sci 2018 13 66 10.1186/s13012-018-0757-2 29743090
Brookman-Frazee L, Stahmer AC. Effectiveness of a multi-level implementation strategy for ASD interventions: study protocol for two linked cluster randomized trials. Implement Sci. 2018;13:66.29743090 10.1186/s13012-018-0757-2
48. Brookman-Frazee L Chlebowski C Suhrheinrich J Finn N Dickson KS Aarons GA Characterizing shared and unique implementation influences in two community services systems for autism: applying the EPIS framework to two large-scale autism intervention community effectiveness trials Adm Policy Ment Health 2020 47 176 187 10.1007/s10488-019-00931-4 30905009
Brookman-Frazee L, Chlebowski C, Suhrheinrich J, Finn N, Dickson KS, Aarons GA, et al. Characterizing shared and unique implementation influences in two community services systems for autism: applying the EPIS framework to two large-scale autism intervention community effectiveness trials. Adm Policy Ment Health. 2020;47:176–87.30905009 10.1007/s10488-019-00931-4
49. Rothman AJ Sheeran P What is slowing us down? Six challenges to accelerating advances in health behavior change Ann Behav Med 2020 54 948 959 10.1093/abm/kaaa090 33416843
Rothman AJ, Sheeran P. What is slowing us down? Six challenges to accelerating advances in health behavior change. Ann Behav Med. 2020;54:948–59.33416843 10.1093/abm/kaaa090
50. Luke DA Powell BJ Paniagua-Avila A Bridges and mechanisms: Integrating systems science thinking into implementation research Ann Rev Public Health. 2024 45 7 10.1146/annurev-publhealth-060922-040205 38100647
Luke DA, Powell BJ, Paniagua-Avila A. Bridges and mechanisms: Integrating systems science thinking into implementation research. Ann Rev Public Health. 2024;45:7.38100647 10.1146/annurev-publhealth-060922-040205
51. Swedberg R Can you visualize theory? On the use of visual thinking in theory pictures, theorizing diagrams, and visual sketches Sociol Theory 2016 34 250 275 10.1177/0735275116664380
Swedberg R. Can you visualize theory? On the use of visual thinking in theory pictures, theorizing diagrams, and visual sketches. Sociol Theory. 2016;34:250–75.10.1177/0735275116664380
52. Brown CH Curran G Palinkas LA Aarons GA Wells KB Jones L An overview of research and evaluation designs for dissemination and implementation Annu Rev Public Health 2017 38 1 22 10.1146/annurev-publhealth-031816-044215 28384085
Brown CH, Curran G, Palinkas LA, Aarons GA, Wells KB, Jones L, et al. An overview of research and evaluation designs for dissemination and implementation. Annu Rev Public Health. 2017;38:1–22.28384085 10.1146/annurev-publhealth-031816-044215
53. Mercer SL DeVinney BJ Fine LJ Green LW Dougherty D Study designs for effectiveness and translation research: Identifying trade-offs Am J Prev Med 2007 33 139 154 10.1016/j.amepre.2007.04.005 17673103
Mercer SL, DeVinney BJ, Fine LJ, Green LW, Dougherty D. Study designs for effectiveness and translation research: Identifying trade-offs. Am J Prev Med. 2007;33:139–54.17673103 10.1016/j.amepre.2007.04.005
54. Mazzucca S Tabak RG Pilar M Ramsey AT Baumann AA Kryzer E Variation in research designs used to test the effectiveness of dissemination and implementation strategies: A review Front Public Health 2018 6 1 10 10.3389/fpubh.2018.00032 29404319
Mazzucca S, Tabak RG, Pilar M, Ramsey AT, Baumann AA, Kryzer E, et al. Variation in research designs used to test the effectiveness of dissemination and implementation strategies: A review. Front Public Health. 2018;6:1–10.29404319 10.3389/fpubh.2018.00032
55. Hwang S Birken SA Melvin CL Rohweder CL Smith JD Designs and methods for implementation research: advancing the mission of the CTSA program J Clin Transl Sci 2020 4 159 167 10.1017/cts.2020.16 32695483
Hwang S, Birken SA, Melvin CL, Rohweder CL, Smith JD. Designs and methods for implementation research: advancing the mission of the CTSA program. J Clin Transl Sci. 2020;4:159–67.32695483 10.1017/cts.2020.16
56. Institute of Medicine. Assessing the Use of Agent-Based Models for Tobacco Regulation. Washington, DC: The National Academies Press; 2015. Available from: 10.17226/19018
57. Bonell C Warren E Melendez-Torres G Methodological reflections on using qualitative research to explore the causal mechanisms of complex health interventions Evaluation 2022 28 166 181 10.1177/13563890221086309
Bonell C, Warren E, Melendez-Torres G. Methodological reflections on using qualitative research to explore the causal mechanisms of complex health interventions. Evaluation. 2022;28:166–81.10.1177/13563890221086309
58. Frank HE Kemp J Benito KG Freeman JB Precision implementation: an approach to mechanism testing in implementation research Adm Policy Ment Health 2022 49 1084 1094 10.1007/s10488-022-01218-x 36167942
Frank HE, Kemp J, Benito KG, Freeman JB. Precision implementation: an approach to mechanism testing in implementation research. Adm Policy Ment Health. 2022;49:1084–94.36167942 10.1007/s10488-022-01218-x
59. Glaser J, Laudel G. The Discovery of Causal Mechanisms: Extractive Qualitative Content Analysis as a Tool for Process Tracing. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research. 2019;20. Available from: https://www.qualitative-research.net/index.php/fqs/article/view/3386. Cited 2023 Aug 9
60. Stone AA Schneider S Smyth JM Evaluation of pressing issues in ecological momentary assessment Annu Rev Clin Psychol 2023 19 107 131 10.1146/annurev-clinpsy-080921-083128 36475718
Stone AA, Schneider S, Smyth JM. Evaluation of pressing issues in ecological momentary assessment. Annu Rev Clin Psychol. 2023;19:107–31.36475718 10.1146/annurev-clinpsy-080921-083128
61. Kerwer M Chasiotis A Stricker J Günther A Rosman T Straight From the scientist’s mouth—plain language summaries promote laypeople’s comprehension and knowledge acquisition when reading about individual research findings in psychology Collabra: Psychology 2021 7 18898 10.1525/collabra.18898
Kerwer M, Chasiotis A, Stricker J, Günther A, Rosman T. Straight From the scientist’s mouth—plain language summaries promote laypeople’s comprehension and knowledge acquisition when reading about individual research findings in psychology. Collabra: Psychology. 2021;7:18898.10.1525/collabra.18898
62. Jones L Wells K Strategies for academic and clinician engagement in community-participatory partnered research JAMA 2007 297 407 410 10.1001/jama.297.4.407 17244838
Jones L, Wells K. Strategies for academic and clinician engagement in community-participatory partnered research. JAMA. 2007;297:407–10.17244838 10.1001/jama.297.4.407
63. London RA, Glass RD, Chang E, Sabati S, Nojan S. “We Are About Life Changing Research”: Community Partner Perspectives on Community-Engaged Research Collaborations. Journal of Higher Education Outreach and Engagement. 2022;26. Available from: https://openjournals.libs.uga.edu/jheoe/article/view/2512. Cited 2023 Aug 9
64. Fernandez ME ten Hoor GA van Lieshout S Rodriguez SA Beidas RS Parcel G Implementation mapping: Using intervention mapping to develop implementation strategies Front Public Health 2019 7 1 15 10.3389/fpubh.2019.00158 30809516
Fernandez ME, ten Hoor GA, van Lieshout S, Rodriguez SA, Beidas RS, Parcel G, et al. Implementation mapping: Using intervention mapping to develop implementation strategies. Front Public Health. 2019;7:1–15.30809516 10.3389/fpubh.2019.00158
65. Fernandez ME Powell BJ Ten Hoor GA Editorial: Implementation Mapping for selecting, adapting and developing implementation strategies Front Public Health 2023 11 1 4 10.3389/fpubh.2023.1288726
Fernandez ME, Powell BJ, Ten Hoor GA. Editorial: Implementation Mapping for selecting, adapting and developing implementation strategies. Front Public Health. 2023;11:1–4.10.3389/fpubh.2023.1288726
66. Lyon AR Coifman J Cook H McRee E Liu FF Ludwig K The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability Implement Sci Commun 2021 2 78 10.1186/s43058-021-00183-0 34274027
Lyon AR, Coifman J, Cook H, McRee E, Liu FF, Ludwig K, et al. The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability. Implement Sci Commun. 2021;2:78.34274027 10.1186/s43058-021-00183-0
67. Graham AK Wildes JE Reddy M Munson SA Taylor CB Mohr DC User-centered design for technology-enabled services for eating disorders Int J Eat Disord 2019 52 1095 1107 10.1002/eat.23130 31313370
Graham AK, Wildes JE, Reddy M, Munson SA, Taylor CB, Mohr DC. User-centered design for technology-enabled services for eating disorders. Int J Eat Disord. 2019;52:1095–107.31313370 10.1002/eat.23130
68. Oxman AD Thomson MA Davis DA Haynes B No magic bullets: A systematic review of 102 trials of interventions to improve professional practice Can Med Assoc J 1995 153 1424 1431
Oxman AD, Thomson MA, Davis DA, Haynes B. No magic bullets: A systematic review of 102 trials of interventions to improve professional practice. Can Med Assoc J. 1995;153:1424–31.
69. Shojania KG Grimshaw JM Still no magic bullets: pursuing more rigorous research in quality improvement Am J Med 2004 116 778 780 10.1016/j.amjmed.2004.03.003 15144917
Shojania KG, Grimshaw JM. Still no magic bullets: pursuing more rigorous research in quality improvement. Am J Med. 2004;116:778–80.15144917 10.1016/j.amjmed.2004.03.003
70. Walsh Bailey C Wiltsey Stirman S Helfrich CD Moullin J Nilsen P Oladunni O The hazards of overreliance on theories, models, and frameworks and how the study of mechanisms can offer a solution 2022 San Diego, CA Society for Implementation Research Collaboration Conference
Walsh Bailey C, Wiltsey Stirman S, Helfrich CD, Moullin J, Nilsen P, Oladunni O, et al. The hazards of overreliance on theories, models, and frameworks and how the study of mechanisms can offer a solution. San Diego, CA: Society for Implementation Research Collaboration Conference; 2022.
71. Connors EH Martin JK Aarons GA Barwick M Bunger AC Bustos TE Proceedings of the Sixth Conference of the Sixth Conference of the Society for Implementation Research Collaboration (SIRC) 2022: from implementation foundations to new frontiers Implement Res Pract 2023 4 S1 185
Connors EH, Martin JK, Aarons GA, Barwick M, Bunger AC, Bustos TE, et al. Proceedings of the Sixth Conference of the Sixth Conference of the Society for Implementation Research Collaboration (SIRC) 2022: from implementation foundations to new frontiers. Implement Res Pract. 2023;4:S1–185.
72. Johnston M Carey RN Connell Bohlen LE Johnston DW Rothman AJ de Bruin M Development of an online tool for linking behavior change techniques and mechanisms of action based on triangulation of findings from literature synthesis and expert consensus Transl Behav Med 2021 11 1049 1065 10.1093/tbm/ibaa050 32749460
Johnston M, Carey RN, Connell Bohlen LE, Johnston DW, Rothman AJ, de Bruin M, et al. Development of an online tool for linking behavior change techniques and mechanisms of action based on triangulation of findings from literature synthesis and expert consensus. Transl Behav Med. 2021;11:1049–65.32749460 10.1093/tbm/ibaa050
73. Waltz TJ Powell BJ Fernández ME Abadie B Damschroder LJ Choosing implementation strategies to address contextual barriers: Diversity in recommendations and future directions Implement Sci 2019 14 1 15 10.1186/s13012-019-0892-4 30611302
Waltz TJ, Powell BJ, Fernández ME, Abadie B, Damschroder LJ. Choosing implementation strategies to address contextual barriers: Diversity in recommendations and future directions. Implement Sci. 2019;14:1–15.30611302 10.1186/s13012-019-0892-4
74. Perez Jolles M Lengnick-Hall R Mittman BS Core functions and forms of complex health interventions: A patient-centered medical home illustration J Gen Intern Med 2019 34 1032 1038 10.1007/s11606-018-4818-7 30623387
Perez Jolles M, Lengnick-Hall R, Mittman BS. Core functions and forms of complex health interventions: A patient-centered medical home illustration. J Gen Intern Med. 2019;34:1032–8.30623387 10.1007/s11606-018-4818-7
75. Lengnick-Hall R Stadnick NA Dickson KS Moullin JC Aarons GA Forms and functions of bridging factors: specifying the dynamic links between outer and inner contexts during implementation and sustainment Implement Sci 2021 16 34 10.1186/s13012-021-01099-y 33794956
Lengnick-Hall R, Stadnick NA, Dickson KS, Moullin JC, Aarons GA. Forms and functions of bridging factors: specifying the dynamic links between outer and inner contexts during implementation and sustainment. Implement Sci. 2021;16:34.33794956 10.1186/s13012-021-01099-y
76. Grimshaw JM Ivers N Linklater S Foy R Francis JJ Gude WT Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback BMJ Qual Saf 2019 28 416 423 10.1136/bmjqs-2018-008355 30852557
Grimshaw JM, Ivers N, Linklater S, Foy R, Francis JJ, Gude WT, et al. Reinvigorating stagnant science: implementation laboratories and a meta-laboratory to efficiently advance the science of audit and feedback. BMJ Qual Saf. 2019;28:416–23.30852557 10.1136/bmjqs-2018-008355
77. The Audit & Feedback MetaLab. Available from: https://www.ohri.ca/auditfeedback/. Cited 2023 Aug 9
78. Implementation Outcome Repository. Available from: https://implementationoutcomerepository.org. Cited 2023 Aug 9
79. Systematic Reviews of Methods to Measure Implementation Constructs. Available from: https://journals.sagepub.com/topic/collections-irp/irp-1-systematic_reviews_of_methods_to_measure_implementation_constructs/irp. Cited 2023 Aug 9
80. Living systematic reviews | Cochrane Community. Available from: https://community.cochrane.org/review-development/resources/living-systematic-reviews. Cited 2023 Aug 9
81. Maguire BJ Guérin PJ A living systematic review protocol for COVID-19 clinical trial registrations Wellcome Open Res 2020 5 60 10.12688/wellcomeopenres.15821.1 32292826
Maguire BJ, Guérin PJ. A living systematic review protocol for COVID-19 clinical trial registrations. Wellcome Open Res. 2020;5:60.32292826 10.12688/wellcomeopenres.15821.1
82. Negrini S Ceravolo MG Côté P Arienti C A systematic review that is ``rapid’’ and ``living’’: A specific answer to the COVID-19 pandemic J Clin Epidemiol 2021 138 194 198 10.1016/j.jclinepi.2021.05.025 34089780
Negrini S, Ceravolo MG, Côté P, Arienti C. A systematic review that is ``rapid’’ and ``living’’: A specific answer to the COVID-19 pandemic. J Clin Epidemiol. 2021;138:194–8.34089780 10.1016/j.jclinepi.2021.05.025
83. Compton J Copeland K Flanders S Cassity C Spetman M Xiao Y Implementing SBAR across a large multihospital health system Joint Comm J Qual Pat Safety 2012 38 261 268
Compton J, Copeland K, Flanders S, Cassity C, Spetman M, Xiao Y, et al. Implementing SBAR across a large multihospital health system. Joint Comm J Qual Pat Safety. 2012;38:261–8.
84. Oh A Vinson CA Chambers DA Future directions for implementation science at the National Cancer Institute: Implementation Science Centers in Cancer Control Transl Behav Med 2021 11 669 675 10.1093/tbm/ibaa018 32145023
Oh A, Vinson CA, Chambers DA. Future directions for implementation science at the National Cancer Institute: Implementation Science Centers in Cancer Control. Transl Behav Med. 2021;11:669–75.32145023 10.1093/tbm/ibaa018
85. Lewis CC Hannon PA Klasnja P Baldwin L-M Hawkes R Blackmer J Optimizing Implementation in Cancer Control (OPTICC): protocol for an implementation science center Implement Sci Commun 2021 2 44 10.1186/s43058-021-00117-w 33892822
Lewis CC, Hannon PA, Klasnja P, Baldwin L-M, Hawkes R, Blackmer J, et al. Optimizing Implementation in Cancer Control (OPTICC): protocol for an implementation science center. Implement Sci Commun. 2021;2:44.33892822 10.1186/s43058-021-00117-w
86. Oh AY Emmons KM Brownson RC Glasgow RE Foley KL Lewis CC Speeding implementation in cancer: The National Cancer Institute’s Implementation Science Centers in Cancer Control J Natl Cancer Inst 2023 115 131 8 10.1093/jnci/djac198 36315080
Oh AY, Emmons KM, Brownson RC, Glasgow RE, Foley KL, Lewis CC, et al. Speeding implementation in cancer: The National Cancer Institute’s Implementation Science Centers in Cancer Control. J Natl Cancer Inst. 2023;115:131–8.36315080 10.1093/jnci/djac198
87. Kilbourne AM Almirall D Eisenberg D Waxmonsky J Goodrich DE Fortney JC Protocol: Adaptive Implementation of Effective Programs Trial (ADEPT): cluster randomized SMART trial comparing a standard versus enhanced implementation strategy to improve outcomes of a mood disorders program Implement Sci 2014 9 1 14 10.1186/s13012-014-0132-x 24398253
Kilbourne AM, Almirall D, Eisenberg D, Waxmonsky J, Goodrich DE, Fortney JC, et al. Protocol: Adaptive Implementation of Effective Programs Trial (ADEPT): cluster randomized SMART trial comparing a standard versus enhanced implementation strategy to improve outcomes of a mood disorders program. Implement Sci. 2014;9:1–14.24398253 10.1186/s13012-014-0132-x
88. Ramanadhan S Revette AC Lee RM Aveling EL Pragmatic approaches to analyzing qualitative data for implementation science: an introduction Implement Sci Commun 2021 2 1 10 10.1186/s43058-021-00174-1 33413699
Ramanadhan S, Revette AC, Lee RM, Aveling EL. Pragmatic approaches to analyzing qualitative data for implementation science: an introduction. Implement Sci Commun. 2021;2:1–10.33413699 10.1186/s43058-021-00174-1
89. National Cancer Institute. Qualitative methods in implementation science. Bethesda, Maryland; 2019. Available from: https://cancercontrol.cancer.gov/sites/default/files/2020-09/nci-dccps-implementationscience-whitepaper.pdf
90. Gertner AK Franklin J Roth I Cruden GH Haley AD Finley EP A scoping review of the use of ethnographic approaches in implementation research and recommendations for reporting Implement Res Pract 2021 2 1 13
Gertner AK, Franklin J, Roth I, Cruden GH, Haley AD, Finley EP, et al. A scoping review of the use of ethnographic approaches in implementation research and recommendations for reporting. Implement Res Pract. 2021;2:1–13.
91. Hamilton AB Finley EP Qualitative methods in implementation research: An introduction Psychiatry Res 2019 280 112516 10.1016/j.psychres.2019.112516 31437661
Hamilton AB, Finley EP. Qualitative methods in implementation research: An introduction. Psychiatry Res. 2019;280:112516.31437661 10.1016/j.psychres.2019.112516
92. Lemire S Kwako A Nielsen SB Christie CA Donaldson SI Leeuw FL What is this thing called a mechanism? Findings from a review of realist evaluations N Dir Eval 2020 167 73 86 10.1002/ev.20428
Lemire S, Kwako A, Nielsen SB, Christie CA, Donaldson SI, Leeuw FL. What is this thing called a mechanism? Findings from a review of realist evaluations. N Dir Eval. 2020;167:73–86.10.1002/ev.20428
93. Lewis CC Boyd MR Marti CN Albright K Mediators of measurement-based care implementation in community mental health settings: results from a mixed-methods evaluation Implement Sci 2022 17 71 10.1186/s13012-022-01244-1 36271404
Lewis CC, Boyd MR, Marti CN, Albright K. Mediators of measurement-based care implementation in community mental health settings: results from a mixed-methods evaluation. Implement Sci. 2022;17:71.36271404 10.1186/s13012-022-01244-1
94. Rabin BA Lewis CC Norton WE Neta G Chambers D Tobin JN Measurement resources for dissemination and implementation research in health Implement Sci 2016 11 1 9 26727969
Rabin BA, Lewis CC, Norton WE, Neta G, Chambers D, Tobin JN, et al. Measurement resources for dissemination and implementation research in health. Implement Sci. 2016;11:1–9.26727969
95. Kendall AD Robinson CSH Diviak KR Hedeker D Mermelstein RJ Introducing a real-time method for identifying the predictors of noncompliance with event-based reporting of tobacco use in ecological momentary assessment Ann Behav Med 2023 57 399 408 10.1093/abm/kaac070 36541688
Kendall AD, Robinson CSH, Diviak KR, Hedeker D, Mermelstein RJ. Introducing a real-time method for identifying the predictors of noncompliance with event-based reporting of tobacco use in ecological momentary assessment. Ann Behav Med. 2023;57:399–408.36541688 10.1093/abm/kaac070
96. Bosch M van der Weijden T Wensing M Grol R Tailoring quality improvement interventions to identified barriers: A multiple case analysis J Eval Clin Pract 2007 13 161 168 10.1111/j.1365-2753.2006.00660.x 17378860
Bosch M, van der Weijden T, Wensing M, Grol R. Tailoring quality improvement interventions to identified barriers: A multiple case analysis. J Eval Clin Pract. 2007;13:161–8.17378860 10.1111/j.1365-2753.2006.00660.x
97. Wensing M Grol R Knowledge translation in health: How implementation science could contribute more BMC Med 2019 17 1 6 10.1186/s12916-019-1322-9 30651111
Wensing M, Grol R. Knowledge translation in health: How implementation science could contribute more. BMC Med. 2019;17:1–6.30651111 10.1186/s12916-019-1322-9
98. Lewis CC Klasnja P Lyon AR Powell BJ Lengnick-Hall R Buchanan G The mechanics of implementation strategies and measures: advancing the study of implementation mechanisms Implement Sci Commun 2022 3 114 10.1186/s43058-022-00358-3 36273224
Lewis CC, Klasnja P, Lyon AR, Powell BJ, Lengnick-Hall R, Buchanan G, et al. The mechanics of implementation strategies and measures: advancing the study of implementation mechanisms. Implement Sci Commun. 2022;3:114.36273224 10.1186/s43058-022-00358-3
99. EPOC Taxonomy. Available from: https://epoc.cochrane.org/epoc-taxonomy. Cited 2023 Aug 9
100. Hoagwood KE, Chaffin M, Chamberlain P, Bickman L, Mittman B. Implementation salvage strategies: maximizing methodological flexibility in children's mental health research. In 4th Annual NIH conference on the Science of Dissemination and Implementation: Policy and Practice; 2011.
101. Dunbar J Hernan A Janus E Davis-Lameloise N Asproloupos D O’Reilly S Implementation salvage experiences from the Melbourne diabetes prevention study BMC Public Health 2012 12 1 9 10.1186/1471-2458-12-806 22214479
Dunbar J, Hernan A, Janus E, Davis-Lameloise N, Asproloupos D, O’Reilly S, et al. Implementation salvage experiences from the Melbourne diabetes prevention study. BMC Public Health. 2012;12:1–9.22214479 10.1186/1471-2458-12-806
102. Carey RN Connell LE Johnston M Rothman AJ de Bruin M Kelly MP Behavior change techniques and their mechanisms of action: a synthesis of links described in published intervention literature Ann Behav Med 2019 53 693 707 30304386
Carey RN, Connell LE, Johnston M, Rothman AJ, de Bruin M, Kelly MP, et al. Behavior change techniques and their mechanisms of action: a synthesis of links described in published intervention literature. Ann Behav Med. 2019;53:693–707.30304386
103. Crable EL Lengnick-Hall R Stadnick NA Moullin JC Aarons GA Where is “policy” in dissemination and implementation science? Recommendations to advance theories, models, and frameworks: EPIS as a case example Implement Sci 2022 17 80 10.1186/s13012-022-01256-x 36503520
Crable EL, Lengnick-Hall R, Stadnick NA, Moullin JC, Aarons GA. Where is “policy” in dissemination and implementation science? Recommendations to advance theories, models, and frameworks: EPIS as a case example. Implement Sci. 2022;17:80.36503520 10.1186/s13012-022-01256-x
104. Purtle J Moucheraud C Yang LH Shelley D Four very basic ways to think about policy in implementation science Implement Sci Commun 2023 4 111 10.1186/s43058-023-00497-1 37700360
Purtle J, Moucheraud C, Yang LH, Shelley D. Four very basic ways to think about policy in implementation science. Implement Sci Commun. 2023;4:111.37700360 10.1186/s43058-023-00497-1
105. Nilsen P Making sense of implementation theories, models and frameworks Implement Sci 2015 10 1 13 10.1186/s13012-015-0242-0 25567289
Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015;10:1–13.25567289 10.1186/s13012-015-0242-0
106. Cane J O’Connor D Michie S Validation of the theoretical domains framework for use in behaviour change and implementation research Implement Sci 2012 7 1 17 10.1186/1748-5908-7-37
Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:1–17.10.1186/1748-5908-7-37
107. Damschroder LJ Reardon CM Widerquist MAO Lowery J The updated consolidated framework for implementation research based on user feedback Implement Sci 2022 17 75 10.1186/s13012-022-01245-0 36309746
Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated consolidated framework for implementation research based on user feedback. Implement Sci. 2022;17:75.36309746 10.1186/s13012-022-01245-0
108. Kislov R Pope C Martin GP Wilson PM Harnessing the power of theorising in implementation science Implement Sci 2019 14 1 8 10.1186/s13012-019-0957-4 30611302
Kislov R, Pope C, Martin GP, Wilson PM. Harnessing the power of theorising in implementation science. Implement Sci. 2019;14:1–8.30611302 10.1186/s13012-019-0957-4
109. Meza RD, Moreland JC, Pullmann MD, Klasnja P, Lewis CC, Weiner BJ. Theorizing is for everybody: Advancing the process of theorizing in implementation science. Frontiers in Health Services. 2023;3. Available from: https://www.frontiersin.org/articles/10.3389/frhs.2023.1134931. Cited 2023 Mar 19
110. Northridge ME Metcalf SS Enhancing implementation science by applying best principles of systems science Health Res Pol Syst 2016 14 1 8
Northridge ME, Metcalf SS. Enhancing implementation science by applying best principles of systems science. Health Res Pol Syst. 2016;14:1–8.
111. Wallerstein N Duran B Community-based participatory research contributions to intervention research: The intersection of science and practice to improve health equity Am J Public Health 2010 100 S40 S46 10.2105/AJPH.2009.184036 20147663
Wallerstein N, Duran B. Community-based participatory research contributions to intervention research: The intersection of science and practice to improve health equity. Am J Public Health. 2010;100:S40–6.20147663 10.2105/AJPH.2009.184036
112. Wells K Jones L “Research” in community-partnered Participatory Res JAMA 2009 302 320 321
Wells K, Jones L. “Research” in community-partnered. Participatory Res JAMA. 2009;302:320–1.
113. Baum F MacDougall C Smith D Participatory action research J Epidemiol Community Health 2006 60 854 10.1136/jech.2004.028662 16973531
Baum F, MacDougall C, Smith D. Participatory action research. J Epidemiol Community Health. 2006;60:854.16973531 10.1136/jech.2004.028662
114. Nguyen T Graham ID Mrklas KJ Bowen S Cargo M Estabrooks CA How does integrated knowledge translation (IKT) compare to other collaborative research approaches to generating and translating knowledge? Learning from experts in the field Health Res Pol Syst. 2020 18 35 10.1186/s12961-020-0539-6
Nguyen T, Graham ID, Mrklas KJ, Bowen S, Cargo M, Estabrooks CA, et al. How does integrated knowledge translation (IKT) compare to other collaborative research approaches to generating and translating knowledge? Learning from experts in the field. Health Res Pol Syst. 2020;18:35.10.1186/s12961-020-0539-6
115. Lyon AR Bruns EJ User-centered redesign of evidence-based psychosocial interventions to enhance implementation: Hospitable soil or better seeds? JAMA Psychiat 2019 76 3 4 10.1001/jamapsychiatry.2018.3060
Lyon AR, Bruns EJ. User-centered redesign of evidence-based psychosocial interventions to enhance implementation: Hospitable soil or better seeds? JAMA Psychiat. 2019;76:3–4.10.1001/jamapsychiatry.2018.3060
116. Kaiser Permanente Washington Health Research Institute. Causal Pathway Diagrams. 2022. Available from: https://vimeo.com/740549106
117. Klasnja P, Meza RD, Pullmann MP, Mettert KD, Hawkes R, Palazzo L, et al. 1354 Getting cozy with causality: A causal pathway diagramming method to 1355 enhance implementation precision. Implement Res Pract. 2024; 5:1–14.
118. Curran GM Implementation science made too simple: a teaching tool Implement Sci Commun 2020 1 1 3 10.1186/s43058-020-00001-z 32885176
Curran GM. Implementation science made too simple: a teaching tool. Implement Sci Commun. 2020;1:1–3.32885176 10.1186/s43058-020-00001-z
119. Schenk PM Wright AJ West R Hastings J Lorencatto F Moore C An ontology of mechanisms of action in behaviour change interventions Wellcome Open Res. 2024 8 337 10.12688/wellcomeopenres.19489.2 38481854
Schenk PM, Wright AJ, West R, Hastings J, Lorencatto F, Moore C, et al. An ontology of mechanisms of action in behaviour change interventions. Wellcome Open Res. 2024;8:337.38481854 10.12688/wellcomeopenres.19489.2
120. Salter KL Kothari A Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review Implement Sci 2014 9 115 10.1186/s13012-014-0115-y 25190100
Salter KL, Kothari A. Using realist evaluation to open the black box of knowledge translation: a state-of-the-art review. Implement Sci. 2014;9:115.25190100 10.1186/s13012-014-0115-y
121. Hamilton AB Mittman BS Brownson RC Colditz GA Proctor EK Implementation science in health care Dissemination and implementation research in health: Translating science to practice 2018 2 New York Oxford University Press 385 400
Hamilton AB, Mittman BS. Implementation science in health care. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: Translating science to practice. 2nd ed. New York: Oxford University Press; 2018. p. 385–400.
122. Viglione C Stadnick NA Birenbaum B Fang O Cakici JA Aarons GA A systematic review of dissemination and implementation science capacity building programs around the globe Implement Sci Commun 2023 4 34 10.1186/s43058-023-00405-7 36973832
Viglione C, Stadnick NA, Birenbaum B, Fang O, Cakici JA, Aarons GA, et al. A systematic review of dissemination and implementation science capacity building programs around the globe. Implement Sci Commun. 2023;4:34.36973832 10.1186/s43058-023-00405-7
