
==== Front
Adm Policy Ment Health
Adm Policy Ment Health
Administration and Policy in Mental Health
0894-587X
1573-3289
Springer US New York

38467950
1363
10.1007/s10488-024-01363-5
Original Article
Leveraging Single-Case Experimental Designs to Promote Personalized Psychological Treatment: Step-by-Step Implementation Protocol with Stakeholder Involvement of an Outpatient Clinic for Personalized Psychotherapy
http://orcid.org/0000-0002-1439-8684
Scholten Saskia saskia.scholten@rptu.de

1
http://orcid.org/0000-0001-7830-4889
Schemer Lea 1
http://orcid.org/0000-0002-8267-2083
Herzog Philipp 12
http://orcid.org/0000-0002-6501-0221
Haas Julia W. 1
Heider Jens 1
http://orcid.org/0000-0002-5096-7995
Winter Dorina 1
http://orcid.org/0000-0002-0027-9713
Reis Dorota 3
Glombiewski Julia Anna 1
1 grid.519840.1 Department of Psychology, Pain and Psychotherapy Research Lab, RPTU Kaiserslautern-Landau, Ostbahnstr. 10, 76829 Landau, Germany
2 https://ror.org/03vek6s52 grid.38142.3c 0000 0004 1936 754X Department of Psychology, Harvard University, 33 Kirkland Street, Cambridge, MA 02138 USA
3 https://ror.org/01jdpyv68 grid.11749.3a 0000 0001 2167 7588 Applied Statistical Modeling, Universität des Saarlandes, Campus, 66123 Saarbrücken, Germany
11 3 2024
11 3 2024
2024
51 5 702724
27 2 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by/4.0/ Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Our objective is to implement a single-case experimental design (SCED) infrastructure in combination with experience-sampling methods (ESM) into the standard diagnostic procedure of a German outpatient research and training clinic. Building on the idea of routine outcome monitoring, the SCED infrastructure introduces intensive longitudinal data collection, individual effectiveness measures, and the opportunity for systematic manipulation to push personalization efforts further. It aims to empower psychotherapists and patients to evaluate their own treatment (idiographic perspective) and to enable researchers to analyze open questions of personalized psychotherapy (nomothetic perspective). Organized around the principles of agile research, we plan to develop, implement, and evaluate the SCED infrastructure in six successive studies with continuous stakeholder involvement: In the project development phase, the business model for the SCED infrastructure is developed that describes its vision in consideration of the context (Study 1). Also, the infrastructure's prototype is specified, encompassing the SCED procedure, ESM protocol, and ESM survey (Study 2 and 3). During the optimization phase, feasibility and acceptability are tested and the infrastructure is adapted accordingly (Study 4). The evaluation phase includes a pilot implementation study to assess implementation outcomes (Study 5), followed by actual implementation using a within-institution A-B design (Study 6). The sustainability phase involves continuous monitoring and improvement. We discuss to what extent the generated data could be used to address current questions of personalized psychotherapy research. Anticipated barriers and limitations during the implementation processes are outlined.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10488-024-01363-5.

Keywords

Single-case experimental design
Experience sampling method
Ecological momentary assessment
Personalized psychotherapy
Stakeholder involvement
Implementation research
Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau (6375)Open Access funding enabled and organized by Projekt DEAL.

issue-copyright-statement© Springer Science+Business Media, LLC, part of Springer Nature 2024
==== Body
pmcIntroduction

Clinicians face the challenge that they work with individuals, while most research is still conducted on the group level. Nomothetic research was intended to identify general psychological processes (Allport, 1937). However, even relatively homogenous groups are composed of heterogeneous individuals. Trying to understand intraindividual variation inferring from results based on interindividual differences may lead to invalid conclusions (Molenaar, 2004). In psychotherapy research, heterogeneity of treatment effects (Varadhan et al., 2013) indicates that patients respond differently to psychological treatments even if they show similar symptoms and are treated by the same therapist with similar methods (Kiesler, 1966). Recently, meta-analyses empirically demonstrated the considerable heterogeneity in treatment effects for several mental disorders such as depression (Kaiser et al., 2022a, 2022b), posttraumatic stress disorder (Herzog & Kaiser, 2022) and borderline personality disorder (Kaiser & Herzog, 2023) with small to medium effects on average depending on the mental disorder studied. These findings underly the need for optimization efforts to maximize treatment outcomes of the individual patient. In this regard and analogous to personalized medicine (Simon & Perlis, 2010), personalizing psychological treatments to the individual patient might be a promising approach to achieve this goal (Chekroud et al., 2021).

Personalized Treatment Approaches

Personalized psychological treatments aim at investigating the dynamics in psychopathology (Fisher, 2015) and mechanisms of change within an individual patient over the course of treatment (Altman et al., 2020). Generally speaking, mental disorders are considered complex systems of contextualized dynamic processes that are specific to the individual and need to be considered in deriving personalized treatment decisions and recommendations (Wright & Woods, 2020).

Personalized treatment approaches can be based on intuitive, theoretical, data-informed, or data-driven models (Cohen et al., 2021). Sources of information for clinical decision-making are intuition and anecdotes for intuitive models, and theory and concepts for theoretical models, respectively (Cohen et al., 2021). Yet, the resulting clinical judgements may be error-prone due to several clinician biases such as representativeness heuristic, selection, and confirmation biases, as well as overoptimism (Lutz et al., 2022). In contrast, data-informed and data-driven models are based on evidence or statistical algorithms (Cohen et al., 2021). One data-based line of research led to the development of empirically derived decision-support tools such as systematic routine outcome monitoring (Lutz et al., 2022). Objective feedback allows individuals to form accurate pattern-recognition abilities (Kahneman & Klein, 2009). Another line of research comprises treatment selection procedures based on clinical prediction models. They aim to achieve a better match between the individual and the treatment received (Cohen & DeRubeis, 2018). Both approaches are highly valuable to address clinician biases, as a complementary tool, and could yield a potential to increase treatment outcomes. Limitations to these data-informed approaches include that many studies are underpowered or lack hold-out samples and are therefore likely to overestimate effects of personalized treatment (Lorenzo-Luaces et al., 2021). In addition, routine outcome monitoring is currently observational, raising questions about the internal validity, and thus limiting the causal inferences (Kaiser et al., 2023). Prediction models are also limited due to their reliance on group level information and large datasets which are not feasible to collect in routine clinical care and private practices.

To push personalization efforts even further, clinician scientists or scientific practitioners need to be empowered to implement successful personalized psychological treatment (Piccirillo & Rodebaugh, 2019). An experimental setting within general practice that focuses on the individual patient is needed (Howard et al., 1996). New technologies, such as experience sampling methods, wearables, open-source software for analyzing time-series data, and more advanced statistical models could assist the integration of individual level designs into clinical practice (Piccirillo & Rodebaugh, 2019).

Promoting Idiographic Research with Single-Case Designs

Single-case experimental designs (SCEDs1) are time- and cost-efficient, yet methodologically sound, prospective idiographic research designs (Nikles et al., 2021) that could be particularly promising to be implemented in routine clinical care (Schemer et al., 2022). Per definition, SCEDs have the following key features: (1) One entity (e.g., a person) that serves as its own control, (2) is observed repeatedly during a certain period of time (for example via experience sampling methodology, ESM2), and (3) under systematic manipulation (e.g., introduction of a psychological treatment) (Vlaeyen et al., 2022). SCEDs seek to estimate meaningful treatment effects for individual patients while safeguarding internal validity and causal inference (Kazdin, 2019; Tanious & Onghena, 2019; Vlaeyen et al., 2020). In combination with ESM, the methodological rigor of SCEDs can be further increased (Schemer et al., 2022). Graphical examples of SCEDs are presented as Supplemental Material (S1 Design illustrations, https://osf.io/dkytg).

SCEDs, especially in combination with ESM, are considered powerful and adaptive research designs for routine clinical care that could reduce the research-practice gap (Bentley et al., 2019; Berg et al., 2023; Kravitz et al., 2014). Both methodological approaches bring different advantages. ESM has shown to support clinical care usefully (e.g., problematic cannabis use: Piccirillo et al., 2023). In particular, both researchers and practitioners highlight that ESM has an incremental value for mental health care, with the assessment of context specificity of symptoms as the most useful part (Piot et al., 2022). SCEDs on the other hand allow to evaluate the efficacy of existing interventions or intervention packages for a particular patient in clinical practice and to easily pilot novel interventions or modifications of known interventions (Krasny-Pacini & Evans, 2018; Selker et al., 2022).

Existing evidence indicates that 79% of participating patients found n-of-1 trials useful (Kaplan & Gabler, 2014) with high perceived system usability (Kravitz et al., 2020). In a medical context, SCEDs are used for example to study the ideal dose of melatonin for sleep disturbance in Parkinson's disease in a way that individual study participants also gained valuable insights (Nikles et al., 2019). Likewise, SCEDs are also particularly suitable for treating mental health problems lacking (adequate) evidence for the efficacy of specific psychological treatments (Duan et al., 2013). Due to high comorbidities and the fact that individuals are not alike, it is often the case that existing protocols are not suitable for patients in routine clinical practice (Berg et al., 2023). By quickly identifying ineffective psychological treatments, SCEDs can help to reduce mental health care costs (Kravitz et al., 2014). In addition, patients may be involved more closely in treatment planning through self-monitoring (Kravitz et al., 2014) and shared decision-making is encouraged through continuous discussion (Riese et al., 2021). The result is a patient-centered, genuine learning system in mental health care (Selker et al., 2022).

In practice, this means that the effects of specific interventions carried out by a practitioner on a particular patient could be evaluated throughout the ongoing treatment (mind: Paul, 1967). A SCED infrastructure allows to set up research studies more easily. For example, a multiple baseline design could be implemented to evaluate isolated intervention strategies (Schemer et al., 2018). Collected data could then be analyzed on an individual level (e.g., single-case randomization tests) and on a group level (e.g., single-case meta-analysis) (Heyvaert & Onghena, 2014).

Objective

Our ultimate objective is to establish an Outpatient Clinic for Personalized Psychotherapy within a German outpatient research and training center. Its key component is a SCED infrastructure creating an experimental setting centered on individual patients within routine clinical care serving the following aims:Idiographic perspective: Enriching current diagnostic procedures mainly comprising clinical interviews and questionnaires by ESM methods that allow continuous monitoring of relevant psychological processes.

Causal inference: Facilitating systematic manipulation going beyond current standards of routine outcome monitoring systems to find effective treatments for challenging or non-responsive patients and to optimize new psychological treatments for future studies.

Nomothetic perspective: Over time, as data accumulates, large numbers of individual intensive-longitudinal datasets could be used for data-driven personalization efforts.

To achieve these long-term goals, we plan to develop, implement, and evaluate an SCED infrastructure following agile research principles (Wilson et al., 2018a). The current paper serves as a study protocol for this step-by-step implementation process. The infrastructure serves two areas of application: routine clinical care and future research projects. While this study protocol outlines the planned procedure in routine clinical care, future research projects may adapt their methodological approaches. Unlike typical implementation research focused on effective interventions (Pinnock et al., 2017), we plan to implement a methodological infrastructure. Reporting standards for implementation studies recommend a twofold reporting strategy always reporting on the implementation strategy (e.g., training of therapists) and the intervention (e.g., exposure therapy). Yet, the SCED infrastructure is not seen as an intervention. Therefore, we focus on typical implementation outcomes such as feasibility, acceptability, and sustainability of the infrastructure (Proctor et al., 2011) as oppose to its effect on clinical outcomes.

Methods and Analysis

To promote an open research culture (Nosek et al., 2015), an Open Science Framework (OSF) project is created to bundle the projects that will be realized using the SCED infrastructure, including respective pre-registrations using the SCED infrastructure, research materials, and preprints (https://osf.io/yex48/).

Setting

The SCED infrastructure will be implemented in the outpatient clinic of the RPTU Kaiserslautern-Landau in Germany. The outpatient clinic is part of a project for coordinating research efforts in German university outpatient clinics for psychotherapy that guides current standard diagnostic procedures (Velten et al., 2017). Standard diagnostic procedures comprise up to four diagnostic sessions and include an intake interview by a licensed psychotherapist, further clinical interviews and questionnaires. The Symptom Checklist 90 (SCL-90, Derogatis & Unger, 2010; Franke, 2014) together with symptom-specific questionnaires is administered at intake, during the diagnostic sessions, during treatment (every 10th session), at posttreatment and at 6-month follow-up.

Around 800 adults with mental disorders are treated per year. Eighty therapists in training and 18 licensed psychotherapists deliver about 45 treatments per day. In Germany, public health insurance usually covers short-term treatment of 24 sessions, which can be extended to up to 80 sessions in total, depending on the mental state of patients. In addition, therapies are implemented within the framework of third-party funded and self-funded psychotherapy studies. The treatment delivered is Cognitive Behavioral Therapy (CBT). The outpatient clinic is a facility that teaches bachelor's graduates who are pursuing their master's degree in psychotherapy and master's graduates who are training to become licensed psychotherapists with a special emphasis on CBT. Adherence to standard CBT methods is safeguarded by regular supervision. Bachelor graduates receive intensive one-on-one supervision by a licensed psychotherapist, who could directly intervene in the case of an emergency, while working with patients. Master graduates receive regular group and individual supervision by a licensed psychotherapist with more than five years of work experience.

Identification of Stakeholders

The following stakeholders have to be considered during the implementation process (Eslick & Sim, 2014; Krasny-Pacini & Evans, 2018; Selker et al., 2022): Patients (and their family members, caregivers…) have to be able to easily enter and access their own data and to view and interpret their results. Therapists (and their supervisors) should be able to recruit and manage a sample of patients, to set up their own SCDs, monitor the data collection progress, intervene if needed, and to view and interpret results. Clinical researchers should be able to create and implement SCEDs, intervene if needed, capture deviations and adjust the protocol, and assess fidelity measures and interrater-reliability. The administrative team provides institutional oversight and management (e.g., creating user accounts). System administrators and developers support the operation of the IT system, provide user tech support, and maintain and problem-solve the operational code. The statistician reviews the trial design and collected data for validity and/or aggregate analysis, download identified or de-identified data for offline analysis, and runs the analysis. Other potentially relevant stakeholders are healthcare payers, healthcare delivery systems, and regulatory agencies. In addition, experts in the field form the scientific advisory board that supervises the implementation of the Outpatient Clinic for Personalized Psychotherapy and training staff of the psychotherapy training consider SCD and m-Path in the curriculum to enable future therapists to easily use the methodology. Stakeholders and their key functions for the SCED infrastructure are summarized in the Supplemental Material (S2 Stakeholder identification, https://osf.io/z4uf5).

Implementation Strategy: Step-by-Step Implementation with Stakeholder Involvement

A successful implementation of the SCED infrastructure needs to emphasize the context where the SCED infrastructure is introduced (Bauer & Kirchner, 2020). Implementation strategies are methods or techniques used to enhance the adoption, sustainability, and scaling up (Powell et al., 2015; Proctor et al., 2013). Discrete implementation strategies involve single approaches or techniques, while the complexity of implementing clinical innovations often necessitates multifaceted strategies that combine two or more of these discrete strategies (Kirchner et al., 2020a). Implementation strategies should be considered from the outset of the planned research (Pinnock et al., 2017), even though they are iterative by nature and need to be adapted when facing unexpected challenges (Kirchner et al., 2020a). We plan to use a step-by-step implementation with stakeholder involvement as multifaceted implementation strategy (compare “stage implementation scale up” and “using advisory boards and workgroups” as recommended implementation strategies by the Expert Recommendations for Implementing Change (ERIC) project; Powell et al., 2015).

Following the recommendations of agile research (Bartels et al., 2022; Wilson et al., 2018), different steps in the implementation process can inform and potentially modify subsequent steps (Fig. 1): In the project identification phase, the SCED infrastructure is envisioned (see Objectives) while acknowledging the context of the SCED infrastructure (see Setting and Stakeholder Identification). It is dedicated to plan the step-by-step implementation and to describe it in this study protocol. In the project development phase, a prototype of the infrastructure is specified. This includes finalizing a business model in consideration of the context (Study 1), the SCED procedure, ESM protocol and ESM survey (Study 2 and 3). In the optimization phase, feasibility and acceptability are tested and the infrastructure is customized accordingly (Study 4). The evaluation phase includes a pilot implementation study to assess implementation outcomes (Study 5) and the actual implementation using a within-institutional A-B design to evaluate the implementation (Study 6). In the sustainability phase, the application is continuously monitored and improved regarding relevance, safety, and effectiveness. Throughout the entire process, we will establish and consult an advisory board consisting of international experts in SCEDs that meets twice a year to guide project decision making.Fig. 1 Step-by-step implementation process

Stakeholders will serve as research partners (Anampa-Guzmán et al., 2022) who will be consulted at several stages through various methods (e.g., via stakeholder meetings and interviews) to develop, optimize, and implement the SCED infrastructure. This involvement provides stakeholders with the opportunity to express what their group needs during the implementation process (such as special trainings or workshops). As outlined above, we consider different types of stakeholders: patients, therapists, supervisors, and clinical researchers as users of the SCED infrastructure; statisticians, administrative staff, and developers as user support (e.g., administrators, developers), and the scientific advisory board and our training staff as collaborators. Stakeholders are engaged using various methods (see separate pre-registrations for detailed descriptions of the respective methods) such as the World Café Method (Brown & Isaacs, 2005; Schiele et al., 2022), stakeholder meetings (Doria et al., 2018), cognitive interviews (Beatty & Willis, 2007), and semi-structural interviews (Domecq et al., 2014). Involvement will be evaluated using the Public and Patient Engagement Evaluation Tool (PPEET) (Abelson et al., 2016) and reported according to the GRIPP2 checklist (Staniszewska et al., 2017).

Study 1: Exploring Barriers and Facilitators to Develop a Business Model

The aim of this study is (1) to explore barriers and facilitators regarding the prospective implementation process and (2) to develop a business model that enables sustainable implementation. For this purpose, a preliminary business model is being formulated, initial technical requirements are being considered, and ethical and privacy issues are being assessed to identify potential questions and challenges. We will conduct a World Café (Brown & Isaacs, 2005; Schiele et al., 2022) at least three persons of all relevant stakeholder groups (patients, therapists, supervisors, researchers) to identify further questions and challenges. In qualitative research, information saturation in interview studies is reached after 6–15 interviews (Guest et al., 2006; Turner-Bowker et al., 2018). Because recommendations for group sizes in World Cafés are missing, we aimed at a group size of 12 persons based on this information. The World Café is followed by in-depth stakeholder interviews with key experts for specific questions (e.g., regarding data protection, psychotherapy training) to develop a business model that addresses these issues.

Draft of the Business Model

The business model is intended to describe how the Outpatient Clinic for Personalized Psychotherapy will create, deliver, and capture value (Osterwalder et al., 2010). Business models have a long tradition in entrepreneurial practice and have been recommended for clinical trials (McDonald et al., 2011) and health care service planning (Sibalija et al., 2021). Developing a business model obliges to think through economic, operational, and strategic aspects that will lead to sustainable benefits in defined markets such as health care services (Morris et al., 2005). Figure 2 shows the first draft of the business model we developed for the Outpatient Clinic for Personalized Psychotherapy following Osterwalder and colleagues' nine building blocks (2010). Subsequently, we will outline each block and highlight questions and challenges (Q&C) that need to be clarified in the development phase:The Outpatient Clinic for Personalized Psychotherapy aims to reach and serve patients with mental health conditions and their family members, therapists in training and licensed therapists, their supervisors, and clinical researchers (called “users”, see “stakeholders” for more information). Q&C: How interested are these users in the Outpatient Clinic for Personalized Psychotherapy? What are the incentives and the support needed to use the Outpatient Clinic for Personalized Psychotherapy?

The Outpatient Clinic for Personalized Psychotherapy provides the following services that create value for the users: an easy-to-use data assessment tool, intra- and interindividual ESM data, ongoing progress feedback, and data-based clinical decision-making support. Q&C: What is needed to make the service (organizationally and technically) easily accessible? How should it be set up that it is accepted and feasible?

Communication or channels connect the users with the services within the Outpatient Clinic for Personalized Psychotherapy. They include the research co-design process to raise awareness and evaluate the concept of the Outpatient Clinic for Personalized Psychotherapy, the ESM app m-Path to use and deliver the services, and personal communication to provide support and to stay in touch (Mestdagh et al., 2022). Q&C: How are different channels connected to each other, e.g., how is feedback provided within m-Path or to a particular therapist channeled to the administrative staff or the research team?

Types of relationships between the Outpatient Clinic for Personalized Psychotherapy and its users (formerly: customer relationships) range from personal assistance to automated services. For a successful implementation, the Outpatient Clinic for Personalized Psychotherapy needs a user support system that includes a statistician, an administrative team, system administrators, developers, the healthcare delivery system, and regulatory agencies (see “stakeholders” for more information). Q&C: How can the Outpatient Clinic for Personalized Psychotherapy be embedded into the existing routine clinical care structure and procedures?

Anticipated benefits are improvements of personalized psychological treatments that can continuously be adapted to the patient’s specific needs and an increase in patient engagement. These benefits are intended to produce reduced mental health care costs in the long run allowing to negotiate with healthcare insurances to pay for the service. Q&C: How can empirical evidence be provided that examine these anticipated benefits? What are the relevant outcomes?

Key resources describe the most important assets (physical, human, financial, and intellectual) that are required to make the business model work. The outpatient clinic is a major asset because it provides the facilities, but also the infrastructure of the healthcare provision including patients, therapists, supervisors, and staff members, likewise generating financial resources. The affiliation to the university is also a key resource because it includes clinical researchers, student assistants, and an intellectual network, e.g., to the scientific advisory board. Q&C: How the Outpatient Clinic for Personalized Psychotherapy integrated into the existing workflow and -packages to ensure long-term implementation?

Key activities that are needed to make the business model work include the implementation of a platform that allows to collect and visualize ESM data, the evaluation and analysis of the collected data, and the publication strategy. Q&C: Which platform should be selected? How are workflows covered that are not implemented in the platform yet?

Key partnerships or collaborators (see “stakeholders” for more information) make the business model work. Q&C: How are stakeholders selected, engaged, and continuously involved into the process?

The cost structure describes all the costs incurred to operate the business model. In our case, the costs to buy and maintain the license for the platform, the salaries of the employees and student assistants, and, if needed, publication fees. Currently, all costs are covered by the university and the outpatient clinic. Q&C: How can costs be covered in the long run to guarantee sustainable funding for the Outpatient Clinic for Personalized Psychotherapy?

Fig. 2 Draft of the Business Model of the Outpatient Clinic for Personalized Psychotherapy

Establishment of Technical Requirements

The Outpatient Clinic for Personalized Psychotherapy's key service is a platform facilitating all SCED phases: goal selection, treatment, measures, trial design (analysis, sampling, randomization, blinding, scheduling, and reviewing), data collection, and analysis (Eslick & Sim, 2014). Frequent graphing of data and adaptability to new information are crucial (Hayes, 1981). To facilitate shared decision-making, data presentation (e.g., treatment effect size, confidence intervals, graphical trajectories, means) must avoid overinterpretation, incorporate user preferences, and maintain scientific rigor (Duan et al., 2013). Using Eslick and Sim's (2014) checklist for platform selection, we decided to use m-Path (Mestdagh et al., 2022). M-path is an easy-to-use, closed-source platform with flexible ESM protocols, varied item formats, data collection and visualization, and just-in-time interventions, ideal for personalized research in clinical settings. m-Path's flexibility and support allowed us to implement a hierarchical account structure embedding therapists', supervisors', and patients' accounts under the research team. Patient data adheres to EU General Data Protection Regulation. Although m-Path visualizes patient data well, it lacks specific features for single case designs, like various trial designs or automated visual/statistical analysis. Q&C: How will the participant flow be organized (especially, if a randomization is used in the design?) How will data analysis and visualization be conducted?

Ethical Considerations and Data Protection

For all studies of the implementation process, the ethical approval is currently being obtained by the institutional ethics committee of the Department of Psychology at the RPTU Kaiserslautern-Landau. However, using the SCED infrastructure in routine clinical care and for research purposes comes with the challenge to meet requirements of treatment-related data and research data. For example, documentation and retention obligation apply to treatment-related data, research data may have to be deleted if the patient requests it. Specific in-depth interviews of Study 1 will be used to clarify questions regarding ethical considerations and data protection. An updated ethical approval will be obtained based on the results. Q&C: Should there be a line between treatment-related data and research data? Can patients opt-out of using the SCED infrastructure and still receive psychological treatment for their condition? How is privacy secured including data gathering, transport, storage, analyses, and presentation?

Study 2: Development of SCED Procedure and ESM Protocol for Routine Clinical Care

The aim of the study is (1) to develop a SCED procedure for routine clinical care, (2) to develop an ESM protocol that describes the different parameters that need to be considered such as assessment duration, frequency, and sampling scheme, and (3) to find a balance between optimal and pragmatic study settings. The ESM protocol will be reviewed in discussion groups with a minimum of six psychotherapists and six patients as the stakeholders who are mostly affected by the ESM protocol (for a justification of the sample size see Study 1). The SCED protocol, along with the outcomes of the discussion groups, will be presented to the international advisory board and to the management board of the outpatient clinic to come to a final decision.

SCED Procedure

We created a working version based on the Risk of Bias in N-of-1 Trials (RoBiNT) scale, a critical evaluation tool to evaluate the methodological quality of intervention studies using single-case methodology (Tate et al., 2013). We outlined the procedure for routine clinical care and research projects that could use the infrastructure of the single-case clinic in the future. For routine clinical care, we plan to use a replicated ABCDE design in which EMA data is collected after the initial appointment (phase A: baseline), during up to five diagnostic sessions (phase B: diagnostic), and during the first treatment sessions (phase C: intervention). Whenever psychotherapists alter their treatment focus or change their therapeutic strategies, they can initiate a new intervention phase (phase C’, phase C’’, etc.). This allows them to directly continuously evaluate their respective therapeutic approach. Furthermore, ESM data will be collected after treatment has ended (phase D: posttreatment) and at 6-months follow-up (phase E: follow-up). Further details can be found in Table 1.Table 1 Working version for the implementation of the single-case clinic based on the RoBiNT Scale (Study 2)

Item	RoBINT Scale	Suggestions for the implementation into routine clinical care	Possibilities for future research projects	
Internal validity subscale		
1	Design	Replicated ABCDE Design

A: baseline phase

B: diagnostic phase

C: intervention phase (C’, C’’, etc.)

D: posttreatment phase

E: follow-up phase

	Replicated randomized designs

Multiple baseline designs (concurrent and non-concurrent)

	
2	Randomization	No randomization due to practice-oriented focus	Randomization of treatment start per patient; randomization of treatment components (and their order)	
3	Sampling behavior (all phases)	Baseline, diagnostic, posttreatment, and follow-up phase: 14 surveys per phase

Intervention phase: open end

	At least 5 data points per phase; patients with less than 3 data points per phase should be excluded from any statistical analysis	
4	Blinding patient/psychotherapist	Blinding not possible and unethical	Blinding options are restricted in psychological treatments; in the case that interventions are compared against each other, neither patients, psychotherapists, nor supervisors should be informed about the research hypothesis	
5	Blinding assessors	Blinding is not possible due to the use of mainly self-report measures; patients, psychotherapists, and supervisors are not blinded to the treatment phase	Blinding options are restricted in psychological treatments; for example, an independent statistician could be blinded to the treatment phase, although the time stamps (informative for data cleaning) will always show which data points belong to the baseline phase	
6	Inter-rater reliability	Treatment progress will be monitored by the patient, psychotherapist, and supervisor; there will be no formal criterium, e.g. when to change the treatment strategy	Future studies might use additional assessment methods (e.g., video recordings, external ratings, passive sensing) that allow to assess inter-rater reliability	
7	Treatment adherence	Adherence to CBT is secured by regular (video-based) supervision of psychotherapists (in training) by experienced psychotherapists (> 5 years’ work experience; completion of a specialized training for supervisors or otherwise accredited based on extensive requirements)	In future research projects, the existing infrastructure for video recordings could be used to formally check adherence via independent raters to a specific treatment protocol	
External validity and interpretation subscale		
8	Baseline characteristics	Baseline characteristics (e.g., demographics, diagnoses, symptom severity) will be routinely assessed; during the diagnostic phase, the psychotherapist will conduct a functional analysis to develop a case conceptualization	Inclusion criteria could be chosen according to the respective research questions	
9	Therapeutic setting	University-affiliated outpatient clinic in Germany (INCLUDE NAME AFTER PEER-REVIEW) in which bachelor graduates can obtain their master degree in psychotherapy and master graduates can obtain their license as psychotherapists with a special focus in CBT	Future studies could also use the SCED infrastructure to cooperate with other clinics	
10	Dependent variable (target behavior)	Emotion regulation processes as a transdiagnostic mechanism; based on the functional analysis in the diagnostic phase, the psychotherapist and patients have the option to create additional individualized items to monitor the treatment progress	Dependent variables could be chosen according to the respective research questions	
11	Independent variable (intervention)	The treatment starts with a diagnostic phase with a subsequent intervention phase; diagnostical tools and interventions will mainly be rooted in a CBT approach	Future studies could implement specific treatment manuals and protocols, as well as compare isolated treatment elements and interventions	
12	Raw data record	Data sets for each individual patient will be recorded with an indication about which data points are missing; data will be stored while respecting data protection regulations	Future studies should strive to increase compliance of assessments (e.g., have regular compliance checks by research assistants)	
13	Data analysis	Patients and psychotherapists are encouraged to look at the data collaboratively through visual inspection; psychotherapists will be encouraged to look at the data with their supervisors through visual inspection	Future research questions and their respective analytic approach should be preregistered in the OSF framework before beginning the statistical analyses	
14	Replication	All patients and psychotherapists will be offered to the SCED infrastructure	One or two replications are considered good, more than three replications are considered optimal; with more and more data collected, future studies could divide a large data set into different subsets (e.g., training vs. test sample) to see whether the effect is replicated in different samples and determine cross-validation	
15	Generalization	The posttreatment and follow-up phase intend to assess generalization effects; different patients will be treated by different psychotherapist and psychotherapists will be supervised by different supervisors	Generalization effect should be preferably monitored throughout treatment but the evaluation before and after treatment is also acceptable; for example, future studies could investigate whether treatment effect generalizes across skills (e.g., do skills to regulate negative emotion automatically translate in better skills to evoke and maintain positive emotions) or situations (e.g., do skills to regulate interpersonal situations automatically translate in better skills to regulation intrapersonal situations)	

ESM Protocol

We followed systematic guidelines to consider different arguments when making methodological decisions (Janssens et al., 2018). A decision matrix was created that organizes different arguments along methodological questions (Supplemental Material, S3 Decision matrix for methodological questions, https://osf.io/54ewu). Arguments include the nature of the variable of interest, reliability and feasibility issues, as well as statistical requirements. Open methodological questions include study duration, measurement frequency, number of items, sampling scheme, instruction of items, and delay allowed to respond. The matrix was filled with typical practices or recommendations for ESM studies found in the literature (Eisele et al., 2020; Janssens et al., 2018; Wrzus & Neubauer, 2023). Based on the matrix, the research team formulated methodical suggestions for the SCED infrastructure regarding each methodological question. We decided to omit the issue of statistical requirements. Instead, future studies using the SCED infrastructure should discuss statistical requirements for the respective research question and analysis in a separate pre-registration in the Open Science Framework (https://osf.io/yex48/).

We propose to collect ESM data on 14 consecutive days during each SCED phase. We decided to not limit the study duration during the intervention phase with the aim to gather information about the extent that the daily surveys are used during treatment. Alternatively, a second (or more) 14-day assessment periods could be prompted during treatment (phase C’ or C’’ etc.). We suggest to collect one signal-contingent survey per day with the option of additional event-contingent surveys whenever the patient feels something important had happened. Each event-contingent survey will trigger a measurement burst with additional five surveys, randomly presented within the following hour. This measurement burst enables to monitor momentary mood on a microlevel. We decided to apply a semi-random sampling scheme with random beeps during an individualized time in the evening (e.g., 18-22 h). As delay allowed to respond, we suggest 60 min for the signal-contingent survey with a reminder after 30 min. For the event-contingent survey, we decided to close the survey after 30 min.

Study 3: Development of the ESM Survey

This study aims (1) to identify a relevant and suitable clinical behavior and (2) to develop an ESM survey to assess this target behavior in routine clinical care. The survey will be part of our standard diagnostic procedure, allowing patients and psychotherapists to add individualized items if desired. The survey's specific content is less critical for implementing the SCED infrastructure and can be replaced or expanded in future research studies based on other variables of interest. However, it is a significant gap in SCED infrastructure implementation that validated ESM surveys are still limited (for a positive example see https://esmitemrepositoryinfo.com/). It is needed if it should be possible to aggregate the generated SCED data across patients (e.g., to investigate overarching research questions). We plan to follow recommendations in questionnaire development and validation, following sequential phases of item generation, scale development and scale validation (Boateng et al., 2018). During this process, a minimum of six stakeholders each will be consulted at several stages until information saturation is reached (for a justification of the sample size see Study 1). For example, patients will be involved via cognitive interviews to ensure the comprehensibility of the items (Darnall et al., 2017). Psychotherapists will be involved to ensure that the m-path data dashboard is used in a meaningful way for clinicians (Guest et al., 2006). Besides content validity, it will be crucial to assess its suitability as a SCED instrument, e.g. in terms of sensitivity to intrapersonal change (Lavefjord et al., 2021), and to consider the potential non-stationarity of the data (Ryan et al., 2023).

We propose to apply a twofold assessment strategy which yields the potential to understand the individual’s model in reference to what is normative (Wright & Zimmermann, 2019) and to capture complex change processes that are of greatest relevance to individual clients being also most consistent with the clinical reality of psychotherapeutic work (Lloyd et al., 2019).

Emotion Regulation

As a first draft of an ESM survey, we decided to focus on emotion regulation as an important transdiagnostic mechanism to complement the thus far primarily symptom-oriented diagnostics with a mechanistically informed approach. Emotion regulation refers to the ‘ability to modulate the intensity, frequency, and duration of positive and/or negative emotions’ (Boemo et al., 2022, p. 1). It is a dynamic, multi-stage process (Gross, 2015), making it well-suited for ESM measurement. Emotion regulation is crucial in psychological disorder development and maintenance (Fernandez et al., 2016). Understanding its complexity in everyday life can reveal regulation difficulties and inform targeted therapeutic interventions (Aldao et al., 2015; Gross & Jazaieri, 2014). Moreover, emotion regulation could be studied more rigorously as mechanism of change in psychotherapy (Palmieri et al., 2022). We plan to include the following emotion regulation processes in the ESM survey: momentary affect (Cloos et al., 2022), information on the appraisal of specific situational contexts (Doré et al., 2016), emotion regulation strategies (Boemo et al., 2022), emotion regulation motives (Tamir, 2016), and emotion regulation success (Gruber et al., 2012). This would allow a differentiated assessment of an important transdiagnostic mechanism within and across patients. Data provides important insight for psychotherapy independent from specific diagnosis and could still be aggregated across patients.

Idiographic Items

Above that, two idiographic items are formulated, a problem-focused and goal-focused idiographic measure. Idiographic measures offer personalized insights and empower clients (Elliott et al., 2016; Wright & Zimmermann, 2019). Especially, goal-setting has been shown to have effects of d = 0.34-0.40 of its own (Epton et al., 2017; Harkin et al., 2016). However, the item generation is not without challenges (Sales et al., 2023). The identification of relevant problems and goals is a difficult task as patients might not always be able to report their problems and goals in clear and precise manners. Also, problems and goals might change throughout the process of psychotherapy. For those reasons, we plan to build upon existing measures. According to two recent systematic reviews (Lloyd et al., 2019; Sales & Alves, 2016) examples for short problem-focused measures that could serve as a starting point are the Simplified Personal Questionnaire (PQ; Elliott et al., 2016; Shapiro, 1961) and the Psychological Outcome Profiles (PSYCHLOPS; Ashworth et al., 2005) and goal-focused measures are Goal Attainment Scaling (Kiresuk & Sherman, 1968), Goals Form (Cooper & Xu, 2023), and Youth TOP Problems (Weisz et al., 2011).

Study 4: Testing Usability and Feasibility

In the optimization phase, the aim is to test usability and feasibility and to customize the SCED infrastructure accordingly. We plan to carry out several SCED iterations using a replicated ABC(DE) design (see Table 1; Tate et al., 2013). Each iteration will comprise 6–12 patient-therapist dyads (Bartels et al., 2022). To accelerate the optimization process, only phases A, B, and C are initially included to identify critical points of improvement. Phases D and E will be carried out, but they will not be decisive for the optimization process. Based on the draft of the ESM protocol, the respective phases should comprise 14 assessment days. Patient-therapist dyads are informed that participation also includes a subsequent stakeholder meeting. The results of each iteration will be presented at the respective stakeholder meeting. Stakeholders will collaboratively identify areas for improvement in each component of the infrastructure. Based on this, they may choose to initiate further rounds of iterations. The optimization loop continues until stakeholders agree that the infrastructure is ready for the implementation phase. Decisions will be consensus-based, requiring agreement from all stakeholders. Figure 3 presents an overview about the design and process of the optimization phase.Fig. 3 Design and process of the optimization phase

Usability Measures

A mixed methods approach will be used to determine the usability of the SCED infrastructure, including questionnaires, technical data, and semi-structured interviews (see Table 2). Questionnaires will be administered at the end of each SCED phase with the concluding assessment comprising questionnaires and a semi-structural interview 6 weeks after the intervention started. To facilitate its evaluation, we decomposed the infrastructure into the following components: initial expectations, the onboarding process, the practical use of the m-path app, the feasibility of the ESM protocol and survey (including possible negative side effects), the clarity of the visual dashboard, the utility for clinical practice, and data management aspects.Table 2 Usability Measures (Study 4)

Questionnaires	ESM data	Semi-structural interview	
• Adapted System Usability Scale (SUS) (Brooke, 1996; Lyon et al., 2020) to evaluate the usability of the m-path app (e.g., ‘I think I would like to use this app frequently.’) and dashboard with the visual presentation of the ESM data (e.g., ‘I think I would like to use this dashboard frequently.’) on a 5-point Likert Scale (1 = strongly disagree to 5 = strongly agree)

• User Experience Questionnaire (UEQ-S) (Schrepp et al., 2017) to assess the subjective impression of users towards the user experience of the m-path app on a 7-point Likert scale (-3 = fully agree with negative term) to + 3 = fully agree with positive term) grouped into 6 subscales: attractiveness, perspicuity, efficiency, dependability, stimulation, novelty

	• Patients have to rate of how burdensome (0 = neutral to 10 = disturbing) and helpful (0 = neutral to 10 = helpful) each survey is (‘How do you rate the survey today?’)

• Response times, compliance rates, and indicators of careless responding will be retrieved from the ESM surveys

• ESM data will be merged with data routinely collected at the outpatient clinic (such as clinical diagnosis, severity of symptoms, and sociodemographic data); difficulties emerging during this process will be recorded in a protocol to be able to resolve the issues for future iterations

	Interview questions are intended to cover different components of the SCED infrastructure:

• Initial expectations (e.g., ‘What did you expect at the beginning? Have these expectations been met?’)

• Onboarding process (e.g., ‘How can we improve the onboarding process?’)

• Practical use of the m-path app (e.g., ‘How did you get on with the m-path app?’)

• Feasibility of the ESM protocol and survey (e.g., ‘Would you change anything about the survey, including the frequency of assessment?’)

• Negative side effects (e.g., ‘Did you experience any negative side effects? How could negative side effects be reduced?’)

• Clarity of the dashboard (e.g., ‘Did you use the dashboard? What would you like to change about the dashboard?’)

• Utility for clinical practice (e.g., ‘Was the app part of the treatment/supervision? What could be changed to maximize its utility?’)

	
Note. Usability measures will be administered to both patients and psychotherapists after each SCED phase; supervisors will also be invited to rate the System Usability Scale (SUS) to evaluate the dashboard (i.e., the visual presentation of the data) and conduct the semi-structural interview

Feasibility Measures

Feasibility is evaluated along the outcome domains of implementation research that are commonly used in feasibility studies (Arain et al., 2010; Proctor et al., 2011): Acceptability and appropriateness are assessed as part of the usability measures. Feasible adoption is reflected in participation (≥ 50% offered using the SCED infrastructure will enroll in the study and ≥ 70% of the participants who enrolled will complete participation; Frumkin et al., 2021). Reasons for drop-out will be assessed if participants consent to provide information. In addition, penetration, compliance and retention will be monitored.

Study 5: Piloting the Implementation of the SCED Infrastructure

The study aims (1) to determine how users accept, perceive, adopt, and integrate the SCED infrastructure into routine clinical care and (2) to use results to initiate strategies for further improvement. For that reason, the SCED infrastructure will be offered in routine clinical care to generate first evidence regarding perceptual and behavioral implementation domains. We intend to offer all new admissions to our clinic to use the SCED infrastructure for a 1-year period. Both patients (irrespective of their clinical diagnosis except for acute suicidal tendencies and psychotic symptoms) and psychotherapists will be invited to participate in the implementation study. We expect to have approximately 250 new admissions in a 1-year period. On average, during the last five years we had 295 new admissions per year, but due to the change of the system we expect a slight decrease. In 2021 and 2022, we had 77 active psychotherapists and 72 respectively. The number of patients and psychotherapists who decide to use the infrastructure will be used to estimate the adoption rate as one implementation outcome (see Section “Implementation Outcome Measures” and Table 3 for more information).Table 3 Implementation Outcome Domains and Measures (Study 5)

Domain	Definition	Concrete Operationalization	Measure	Criteria	
Acceptability and Appropriateness	Acceptability is defined as the perception among different stakeholders that an innovation is agreeable, palatable, or satisfactory

Appropriateness refers to the perceived fit, relevance, or compatibility of an innovation for a given practice or setting

	Degree that patients and psychotherapists think the SCED infrastructure is acceptable and helpful to improve clinical practice in an outpatient setting	(1) The tool (i.e., the handling of the m-path app) and dashboard (i.e., the visual presentation of the data) are rated independently on a 5-point Likert scale (1 = ‘strongly disagree’ to 5 = ‘strongly agree’):

• ‘The tool is acceptable for clinical practice’

• ‘The tool is helpful for clinical practice’

• ‘The dashboard is acceptable for clinical practice’

• ‘The dashboard is helpful for clinical practice’

(2) Three subscales of an adapted version of the Adoption of Information Technology Innovation Measure (Moore & Benbasat, 1991) are rated on a 7-point Likert Scale (1 = ‘extremely disagree’ to 7 = ‘extremely agree’):

• Relative advantages (e.g., ‘Using the tool improves the quality of the treatment’),

• Compatibility (e.g., ‘Using the tool is completely compatible with my current situation’)

• Ease of use (e.g., ‘Overall, I believe that the tool is easy to use’)

(3) Adapted version of the Evidence-based Practice Attitudes Scale (EBPAS; Aarons, 2004) together with its extension (Aarons et al., 2012)

	(1) Mean scores of > 3.5/5 on each item are considered sufficiently acceptable and appropriate

(2) Mean scores of > 4.5/7 on each subscale are considered sufficiently advantageous, compatible, and easy to use

(3) N/A

	
Adoption	Adoption comprises the intention, intentional decision, or action to try or employ an innovation	Number of individuals who are willing to use the SCED infrastructure (i.e., who consent to test the infrastructure as part of the study)	(4) Number of patients and psychotherapists who decided to use the infrastructure divided by the number of invitations (i.e., new admissions to the clinic)	(4) A rate ≥ 50% is considered substantial demand	
Penetration	Penetration refers to the integration of a practice within a setting	Number of patients and psychotherapists who actually use the SCED infrastructure (e.g., in terms of responding to the ESM surveys; retrieving and discussing the dashboard during treatment or supervision)	(5) ESM data (including signal- and event-contingent surveys of completed ESM surveys (i.e., the last question has been answered)) is used to derive an estimate for the penetration:

• Exploration of arithmetic values (e.g., median, range, mean standard deviation) for the number of completed ESM surveys for each SCED phase, especially for the open-ended intervention phase

• RoBiNT scale requires each phase to consist of at least five, but not less than three data points for the analysis of SCED data (Tate et al., 2013). We will evaluate each phase separately, with ≥ 5 data points as an acceptable criterion. For each phase, we divide the number of patients who fulfill the acceptable (≥ 5 data points) criterion by the number of participating patients

(6) Indication how often patients and psychotherapists retrieved the dashboards either to prepare for treatment or discuss the dashboard during treatment on a 5-point Likert scale (1 = never to 5 = every session) after each SCED phase. Psychotherapists will also be asked to indicate how often they discussed the dashboard with their supervisors

• ‘How often did you retrieve the dashboard (e.g., to prepare for treatment or discuss the dashboard during treatment)?’

• ‘How often did you discuss the dashboard together with your supervisor?’

	(5) A rate ≥ 75 is considered substantial use

(6) Mean scores of > 3.5/5 on each item are considered substantial use

	
Implementation costs	All costs needed for an implementation effort	All monetary resources needed for the implementation of the SCED infrastructure. These costs could either be anticipated (e.g., for the m-path software) or unexpected (i.e., they only become clear in the course of implementation)	Implementation costs are also difficult to calculate because they vary substantially in the different settings

(7) A list of the costs we have incurred (e.g., required working time of an employee to coordinate the single-case clinic)

		
Compliance and Retention Rates	Compliance refers to the ratio of completed surveys over the theoretical maximum number of surveys

Retention refers to the proportion of participants included in the final analyses

	Compliance and retention rates to the ESM surveys intend to evaluate to what extent the data generated in a routine clinical setting could be used for future data analyses to address overarching research questions	(8) Ratio of completed signal-contingent surveys divided by the number of theoretically possible surveys (as an indicator for compliance)

(9) Proportion of participants who responded to at least five of the signal-contingent surveys after data cleaning (as an indicator for retention)

	(7) Rates of 70–80% are considered satisfactory compliance

(8) Rates of 90% are considered satisfactory retention

	
Note. Definitions are derived by Proctor et al. (2011). Implementation outcome measures will be administered to both patients and psychotherapists after each SCED phase; only the Evidence-based Practice Attitudes Scale (EBPAS) will be administered once at the study beginning (i.e., when patients and psychotherapists decide whether to use the SCED infrastructure); this will allow to compare individuals who are willing to use the SCED infrastructure with those who are not. Criteria for a successful implementation were chosen according to previous research (Schleider et al., 2020); criteria for compliance and retention rates were formulated based on systematic reviews (Rintala et al., 2019; Vachon et al., 2019; Wrzus & Neubauer, 2023); it should be noted, however, that these values were estimated in the context of research studies (as opposed to a routine clinical care setting)

To evaluate our implementation process, we assess several perceptual and behavioral outcome domains (Proctor et al., 2011). We hope to obtain some outcome measures from people who are willing to use the SCED infrastructure, as well as those who are not interested. This will allow us to conduct some exploratory analyzes about how these groups differ. For patients, we expect to collect a relatively diverse sample which will probably allow such a comparison.

Results of the initial implementation study will be a first indicator for the expected sustainability of the SCED infrastructure in our clinic. In case, results do not meet our predefined criteria for a successful implementation, the research team will develop a proposal for another implementation strategy plan (Kirchner et al., 2020b; Powell et al., 2015) to maintain and further improve the uptake of the SCED infrastructure. Other implementation strategies could comprise the provision of interactive assistance or the training of stakeholders. Again, the implementation plan will be optimized and finalized together with different stakeholders. In either case, we plan to replicate the pilot implementation study for another year, with the goal to either test whether implementation outcomes substantially improve compared to the first implementation round or to evaluate its sustainability.

Implementation Outcome Measures

Perceptual and behavioral outcome measures were selected in accordance with a generally recognized taxonomy for outcome domains in implementation research (see Table 3; Proctor et al., 2011). For our study purposes, we considered the outcome domains acceptability, appropriateness, adoption, and penetration. In addition, we plan to report on compliance and retention rates as indicators inherently linked to ESM studies. Criteria for a successful implementation were chosen according to previous research (Schleider et al., 2020). Criteria for compliance and retention rates were formulated based on systematic reviews (Rintala et al., 2019; Vachon et al., 2019; Wrzus & Neubauer, 2023). We decided to omit the domains of feasibility, fidelity, and sustainability as these issues are addressed in the other phases of the agile research framework. For example, feasibility will be ameliorated and tested in phase 3, while sustainability will be addressed in phase 5. Likewise, it is difficult to include fidelity in an agile research framework as methodological decisions can change based on outcomes from previous phases. We have chosen behavioral measures whenever possible. Given the general lack of unified implementation measures with mostly unknown psychometric quality, questionnaires to assess perceptual outcomes were selected based on a recent systematic review evaluating the psychometric properties of implementation measures (Mettert et al., 2020). Perceptual implementation outcome measures will be collected from both patients and psychotherapists when they decide whether they want to use the SCED infrastructure prior to treatment. Perceptional implementation outcomes will also be administered after each SCED phase to record their development over time. In the intervention phase, implementation measures will be administered 6 weeks after the intervention started. If the post-treatment and follow-up phases are not yet available for all patients at the time of data analysis, patients will be invited to take part in another 14 ESM assessment days and to provide concluding information on perceptual implementation outcome measures.

Study 6: Implementation of the SCED Infrastructure

The aim of the study during the implementation phase is to evaluate the SCED infrastructure compared to routine diagnostic procedures prior implementation. A quasi-experimental single case AB-design with the outpatient clinic as entity will be conducted. The design has the advantages that it reduces confounders (e.g., secular trends, seasonality) and allows to measure long-term effects (Bernal et al., 2017). We will use a continuous sequence of outcome data taken repeatedly over time from the outpatient clinic prior implementation and compare it to outcome data post implementation. This time series is used to estimate an interrupted time series regression. An underlying trend is established, which is ‘interrupted’ by the implementation of the SCED infrastructure at a known point in time (Bernal et al., 2017). Therefore, pre- and post-implementation need to be clearly distinguished (Miller et al., 2020). A graphical example of an interrupted time series is presented as Supplemental Material (S1 Design illustrations, https://osf.io/dkytg).

In our case, data are considered for the pre-implementation phase up to the start of Study 5 and for the post-implementation phase after the completion of Study 5, when the SCED infrastructure is introduced for everyone into routine clinical care. Two items assessing satisfaction and burden of the diagnostic procedure (that includes the ESM assessments after the implementation of the SCED infrastructure) will be used to evaluate the new diagnostic procedure. In addition, we will take an exploratory look at clinical effectiveness using the Symptom Checklist – 90 (Derogatis & Unger, 2010; Franke, 2014). We expect a slight level and slope change regarding clinical effectiveness because research has shown that self-monitoring already effects outcomes positively leading to level change (Bartels et al., 2019; Guo & Albright, 2018; McBain et al., 2015; Simons et al., 2015). In addition, the continuous feedback might reduce treatment failure and improve the positive effects of psychotherapy for example through enhanced tailoring (Lambert & Harmon, 2018; Lutz et al., 2019), leading to a temporary slope change resulting in a level change.

Discussion

SCEDs are prospective idiographic research designs aiming to estimate meaningful treatment effects for individual patients (Vlaeyen et al., 2020). We argue that the method is highly suitable to foster the development and evaluation of personalized psychological treatments for two main reasons: First, current data-driven personalization efforts are mainly based on nomothetic data that come with limitations due to power (Lorenzo-Luaces et al., 2021) and due to invalid inferences to the individual. In contrast, the SCED infrastructure offers a truly idiographic approach to research and practice. Nevertheless, generalization is enabled through replication creating data that can also be accumulated to answer nomothetic research questions. Second, since the SCED infrastructure allows systematic manipulation, it is valid to draw causal inferences and estimate treatment effects for the individual patient which extends the observational character of current routine outcome monitoring systems. For those reasons, we outlined the step-by-step implementation with stakeholder involvement of a SCED infrastructure including ESM into routine clinical care of a German outpatient research and training clinic. While focusing on routine clinical care in this protocol, the infrastructure can also serve future research studies using SCED methodology.

To our knowledge, such a SCED infrastructure has not been implemented in the context of mental health up till now. Yet, such an infrastructure could be associated with several advantages: The SCED infrastructure could link everyday life to the psychotherapy process more closely and brings the psychotherapy process back into daily life. For example, it could empower patients through self-monitoring and feedback and enhance clinical and collaborative decision-making through continuous evaluation (Herzog et al., 2022; Schemer et al., 2022), also overcoming biases of clinical judgements (Kaiser et al., 2022a, 2022b). Therapists will probably find this feedback highly credible because it results from multiple assessments in the daily life of patients. The feedback provides new information about the patients' progress with high precision. This accurate information from a credible source facilitates the integration of feedback and update of beliefs about patients’ therapy progress—a necessary prerequisite for the feedback to effect treatment outcome (Herzog et al., 2023).

Practitioners in routine clinical care could also use systematic manipulation in an ongoing treatment to evaluate their own clinical practice for their particular patients (Piccirillo & Rodebaugh, 2019). Thus, the infrastructure could bring science into practice allowing a truly individualized and at the same time evidence-based treatment to deliver the most effective treatment for the respective patient. Above that, it provides an infrastructure for a range of idiographic and/or nomothetic research questions that need intensive longitudinal data assessment and/or systematic manipulation: What specific emotional regulation difficulties does the individual patient have and how can they best be addressed in therapy? How effective is a particular treatment strategy such as high-intensity exposure for an individual patient or specific patient groups? To what extent does a change in therapeutic strategy lead to an improvement in the process outcome if there was no improvement or even a deterioration in the previous phase? Asked differently, how does changing the therapeutic strategy too quickly affect the therapeutic process? How valid are inferences from group data to the individual vs. from individual data to the group regarding the psychotherapy process (ergodicity problem; see Adolf & Fried, 2019; Fisher et al., 2018)? Methodologically, the SCED infrastructure allows to focus on patterns within a person using intensive longitudinal data that can be used to inform decisions about personalized treatment strategies for this specific person (Delgadillo & Lutz, 2020). For example, it might be interesting to take into account the dynamic nature of psychological mechanisms. Once data has accumulated, personalized advantage indexes could be developed to guide personalized treatment selection on a very fine-grained level (DeRubeis et al., 2014).

The step-by-step implementation of a SCED infrastructure including EMA with stakeholder involvement comes with the following limitations: First, the step-by-step implementation is initiated expecting that the SCED infrastructure will enable studies that will push personalization forward leading to more effective treatments for individual patients. However, evidence that will support this expectation can only be generated after implementation. Therefore, we did not describe a stopping rule for the implementation process, even though there could be a high patient burden, lacking penetration, or it might not lead to better treatment in the long run.

Second, it must be kept in mind that evidence for the effectiveness of stakeholder engagement from experimental or empirical studies is lacking (Slattery et al., 2020). At the same time, it comes with an increase in research resources such as time, costs, and training efforts, changes that might not be feasibly and uncertainty to resolve conflicts (Holzer et al., 2022; Tindall et al., 2021). Further barriers are the use of technical jargon, power imbalances between the researcher and stakeholders, difficulty for stakeholders to understand how their input is reflected in the final research (Holzer et al., 2022). Therefore, it is important to introduce the research purpose, the questions being addressed and the requirements for the studies carefully (Holzer et al., 2022). Role descriptions and responsibilities should be clearly defined and the stakeholders’ time and input should be valued to build trust and rapport between researchers and stakeholders (Slattery et al., 2020).

Third, there are many obstacles when implementing a new infrastructure into an existing routine. As the consolidated framework for implementation research outlines, many domains need to be taken into consideration including the inner and outer setting, the individuals involved in the implementation, and the implementation process itself (Damschroder et al., 2009, 2022). While this protocol outlines the implementation process and some aspects of the other domains, the adaptation to the inner and outer setting will present numerous challenges including the accounting of the service with insurances, patient data protection and handling policies, or the maintenance of the service with existing staff. Even though we try to address these challenges from early on, most of them will come up throughout the process. The implementation strategies were chosen to respond to those upcoming issues flexibly on the way.

Despite these limitations, we are optimistic that challenges can be faced and overall, the potential of the SCED infrastructure will outweigh the burden. We will propose a business model for a Outpatient Clinic for Personalized Psychotherapy—a learning mental health care system that can reduce the research-practice gap. We will implement this SCED infrastructure in combination with EMA into routine clinical care step-by-step with stakeholder involvement to leverage the potential of person-specific data-based and data-driven methods (Schemer et al., 2022).

Supplementary Information

Below is the link to the electronic supplementary material.Supplementary file1 (PDF 118 kb)

Supplementary file2 (PDF 62 kb)

Supplementary file3 (PDF 70 kb)

Acknowledgements

We would like to thank Johan W. S. Vlaeyen for his continuous reflections regarding the implementation of a SCED infrastructure.

Author Contributions

All authors contributed to the manuscript conception and design. The original draft of the manuscript was written by Saskia Scholten and Lea Schemer. Methodological considerations were developed by Saskia Scholten, Lea Schemer, and Philipp Herzog. Visualization was realized by Saskia Scholten and Julia W. Haas. Supervision was provided by Julia Anna Glombiewski. All authors reviewed and edited previous versions of the manuscript. All authors read and approved the final manuscript.

Funding

Open Access funding enabled and organized by Projekt DEAL. No funding was received to assist with the preparation of this manuscript.

Declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Ethical Approval

All procedures performed in studies involving human participants will be in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. We considered the Single-Case Reporting Guideline in Behavioral Intervenions (SCRIBE) 2016 Statement (Tate et al., 2016) and Standards for Reporting Implementation Studies (StaRI) Statement (Pinnock et al., 2017) to draft this manuscript.

1 Different terms are used in the context of SCEDs such as Single-Case Experimental Design (SCED), Single-Subject Research Design, Single-Case Observational Design (SCOD), N-of-1 trial, Single Patient Open Trial (SPOT) and Single-Case Design (SCD) (Krasny-Pacini & Evans, 2018; Nikles et al., 2021). Although SCEDs is widely used, we use from here on SCD as an umbrella term, allowing also observational and open designs and thus fitting better with the intended use in our routine clinical care setting.

2 Experience Sampling Methodology (ESM), Ecological Momentary Assessments, and Ambulatory Assessments comprise data collection methods used to study individuals’ daily life experiences and environmental features in naturalistic settings (see Myin-Germeys & Kuppens, 2021). In particular, ESM and Ecological Momentary Assessment prompt individuals to respond to surveys assessing psychological constructs as they unfold in daily life (Schemer et al., 2022). For consistency reasons, we use the term “ESM”.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

Abelson J Li K Wilson G Shields K Schneider C Boesveld S Supporting quality public and patient engagement in health system organizations: Development and usability testing of the Public and Patient Engagement Evaluation Tool Health Expectations 2016 19 4 817 827 10.1111/hex.12378 26113295
Abelson, J., Li, K., Wilson, G., Shields, K., Schneider, C., & Boesveld, S. (2016). Supporting quality public and patient engagement in health system organizations: Development and usability testing of the Public and Patient Engagement Evaluation Tool. Health Expectations, 19(4), 817–827. 10.1111/hex.1237826113295 10.1111/hex.12378
Adolf JK Fried EI Ergodicity is sufficient but not necessary for group-to-individual generalizability Proceedings of the National Academy of Sciences 2019 116 14 6540 6541 10.1073/pnas.1818675116
Adolf, J. K., & Fried, E. I. (2019). Ergodicity is sufficient but not necessary for group-to-individual generalizability. Proceedings of the National Academy of Sciences, 116(14), 6540–6541. 10.1073/pnas.181867511610.1073/pnas.1818675116
Aldao A Sheppes G Gross JJ Emotion regulation flexibility Cognitive Therapy and Research 2015 39 3 263 278 10.1007/s10608-014-9662-4
Aldao, A., Sheppes, G., & Gross, J. J. (2015). Emotion regulation flexibility. Cognitive Therapy and Research, 39(3), 263–278. 10.1007/s10608-014-9662-410.1007/s10608-014-9662-4
Allport, G. W. (1937). Personality: A psychological interpretation (pp. xiv, 588). Holt.
Altman AD Shapiro LA Fisher AJ Why does therapy work? An idiographic approach to explore mechanisms of change over the course of psychotherapy using digital assessments Frontiers in Psychology 2020 11 782 10.3389/fpsyg.2020.00782 32390922
Altman, A. D., Shapiro, L. A., & Fisher, A. J. (2020). Why does therapy work? An idiographic approach to explore mechanisms of change over the course of psychotherapy using digital assessments. Frontiers in Psychology, 11, 782. 10.3389/fpsyg.2020.0078232390922 10.3389/fpsyg.2020.00782
Anampa-Guzmán A Freeman-Daily J Fisch M Lou E Pennell NA Painter CA Sparacio D Lewis MA Karmo M Anderson PF Graff SL For the Collaboration for Outcomes using Social Media in Oncology The rise of the expert patient in cancer: From backseat passenger to co-navigator JCO Oncology Practice 2022 18 8 578 583 10.1200/OP.21.00763 35344398
Anampa-Guzmán, A., Freeman-Daily, J., Fisch, M., Lou, E., Pennell, N. A., Painter, C. A., Sparacio, D., Lewis, M. A., Karmo, M., Anderson, P. F., Graff, S. L., For the Collaboration for Outcomes using Social Media in Oncology. (2022). The rise of the expert patient in cancer: From backseat passenger to co-navigator. JCO Oncology Practice, 18(8), 578–583. 10.1200/OP.21.0076335344398 10.1200/OP.21.00763
Arain M Campbell MJ Cooper CL Lancaster GA What is a pilot or feasibility study? A review of current practice and editorial policy BMC Medical Research Methodology 2010 10 1 67 10.1186/1471-2288-10-67 20637084
Arain, M., Campbell, M. J., Cooper, C. L., & Lancaster, G. A. (2010). What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology, 10(1), 67. 10.1186/1471-2288-10-6720637084 10.1186/1471-2288-10-67
Ashworth M Robinson SI Godfrey E Shepherd M Evans C Seed P Parmentier H Tylee A Measuring mental health outcomes in primary care: The psychometric properties of a new patient-generated outcome measure’, PSYCHLOPS’('psychological outcome profiles’) Primary Care Mental Health 2005 3 4
Ashworth, M., Robinson, S. I., Godfrey, E., Shepherd, M., Evans, C., Seed, P., Parmentier, H., & Tylee, A. (2005). Measuring mental health outcomes in primary care: The psychometric properties of a new patient-generated outcome measure’, PSYCHLOPS’('psychological outcome profiles’). Primary Care Mental Health, 3, 4.
Bartels SL Johnsson SI Boersma K Flink I McCracken LM Petersson S Christie HL Feldman I Simons LE Onghena P Vlaeyen JWS Wicksell RK Development, evaluation and implementation of a digital behavioural health treatment for chronic pain: Study protocol of the multiphase DAHLIA project British Medical Journal Open 2022 12 4 e059152 10.1136/bmjopen-2021-059152
Bartels, S. L., Johnsson, S. I., Boersma, K., Flink, I., McCracken, L. M., Petersson, S., Christie, H. L., Feldman, I., Simons, L. E., Onghena, P., Vlaeyen, J. W. S., & Wicksell, R. K. (2022). Development, evaluation and implementation of a digital behavioural health treatment for chronic pain: Study protocol of the multiphase DAHLIA project. British Medical Journal Open, 12(4), e059152. 10.1136/bmjopen-2021-05915210.1136/bmjopen-2021-059152
Bartels SL van Knippenberg RJM Dassen FCM Asaba E Patomella A-H Malinowsky C Verhey FRJ de Vugt ME A narrative synthesis systematic review of digital self-monitoring interventions for middle-aged and older adults Internet Interventions 2019 18 100283 10.1016/j.invent.2019.100283 31890630
Bartels, S. L., van Knippenberg, R. J. M., Dassen, F. C. M., Asaba, E., Patomella, A.-H., Malinowsky, C., Verhey, F. R. J., & de Vugt, M. E. (2019). A narrative synthesis systematic review of digital self-monitoring interventions for middle-aged and older adults. Internet Interventions, 18, 100283. 10.1016/j.invent.2019.10028331890630 10.1016/j.invent.2019.100283
Bauer MS Kirchner J Implementation science: What is it and why should I care? Psychiatry Research 2020 283 112376 10.1016/j.psychres.2019.04.025 31036287
Bauer, M. S., & Kirchner, J. (2020). Implementation science: What is it and why should I care? Psychiatry Research, 283, 112376. 10.1016/j.psychres.2019.04.02531036287 10.1016/j.psychres.2019.04.025
Beatty PC Willis GB Research synthesis: The practice of Cognitive Interviewing Public Opinion Quarterly 2007 71 2 287 311 10.1093/poq/nfm006
Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of Cognitive Interviewing. Public Opinion Quarterly, 71(2), 287–311. 10.1093/poq/nfm00610.1093/poq/nfm006
Bentley KH Kleiman EM Elliott G Huffman JC Nock MK Real-time monitoring technology in single-case experimental design research: Opportunities and challenges Behaviour Research and Therapy 2019 117 87 96 10.1016/j.brat.2018.11.017 30579623
Bentley, K. H., Kleiman, E. M., Elliott, G., Huffman, J. C., & Nock, M. K. (2019). Real-time monitoring technology in single-case experimental design research: Opportunities and challenges. Behaviour Research and Therapy, 117, 87–96. 10.1016/j.brat.2018.11.01730579623 10.1016/j.brat.2018.11.017
Berg M Schemer L Kirchner L Scholten S Mind the gap—Ideas for making clinical research more relevant for practitioners and patients PsyArXiv 2023 10.31234/osf.io/2qvhy
Berg, M., Schemer, L., Kirchner, L., & Scholten, S. (2023). Mind the gap—Ideas for making clinical research more relevant for practitioners and patients. PsyArXiv. 10.31234/osf.io/2qvhy10.31234/osf.io/2qvhy
Bernal JL Cummins S Gasparrini A Interrupted time series regression for the evaluation of public health interventions: A tutorial International Journal of Epidemiology 2017 46 1 348 355 10.1093/ije/dyw098 27283160
Bernal, J. L., Cummins, S., & Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology, 46(1), 348–355. 10.1093/ije/dyw09827283160 10.1093/ije/dyw098
Boateng GO Neilands TB Frongillo EA Melgar-Quiñonez HR Young SL Best practices for developing and validating scales for health, social, and behavioral research: A primer Frontiers in Public Health 2018 6 149 10.3389/fpubh.2018.00149 29942800
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. 10.3389/fpubh.2018.0014929942800 10.3389/fpubh.2018.00149
Boemo T Nieto I Vazquez C Sanchez-Lopez A Relations between emotion regulation strategies and affect in daily life: A systematic review and meta-analysis of studies using ecological momentary assessments Neuroscience & Biobehavioral Reviews 2022 139 104747 10.1016/j.neubiorev.2022.104747 35716875
Boemo, T., Nieto, I., Vazquez, C., & Sanchez-Lopez, A. (2022). Relations between emotion regulation strategies and affect in daily life: A systematic review and meta-analysis of studies using ecological momentary assessments. Neuroscience & Biobehavioral Reviews, 139, 104747. 10.1016/j.neubiorev.2022.10474735716875 10.1016/j.neubiorev.2022.104747
Brooke, J. (1996). SUS: A quick and dirty usability scale. In Usability evaluation in industry (pp. 189–194).
Brown J Isaacs D The World Café: Shaping our futures through conversations that matter 2005 1 Berrett-Koehler
Brown, J., & Isaacs, D. (2005). The World Café: Shaping our futures through conversations that matter (1st ed.). Berrett-Koehler.
Chekroud AM Bondar J Delgadillo J Doherty G Wasil A Fokkema M Cohen Z Belgrave D DeRubeis R Iniesta R Dwyer D Choi K The promise of machine learning in predicting treatment outcomes in psychiatry World Psychiatry 2021 20 2 154 170 10.1002/wps.20882 34002503
Chekroud, A. M., Bondar, J., Delgadillo, J., Doherty, G., Wasil, A., Fokkema, M., Cohen, Z., Belgrave, D., DeRubeis, R., Iniesta, R., Dwyer, D., & Choi, K. (2021). The promise of machine learning in predicting treatment outcomes in psychiatry. World Psychiatry, 20(2), 154–170. 10.1002/wps.2088234002503 10.1002/wps.20882
Cloos L Ceulemans E Kuppens P Development, validation, and comparison of self-report measures for positive and negative affect in intensive longitudinal research Psychological Assessment 2022 10.1037/pas0001200 36480406
Cloos, L., Ceulemans, E., & Kuppens, P. (2022). Development, validation, and comparison of self-report measures for positive and negative affect in intensive longitudinal research. Psychological Assessment. 10.1037/pas000120036480406 10.1037/pas0001200
Cohen ZD Delgadillo J DeRubeis RJ Barkham M Lutz W Castonguay LG Personalized treatment approaches Bergin and Garfield’s handbook of psychotherapy and behavior change 2021 Wiley
Cohen, Z. D., Delgadillo, J., & DeRubeis, R. J. (2021). Personalized treatment approaches. In M. Barkham, W. Lutz, & L. G. Castonguay (Eds.), Bergin and Garfield’s handbook of psychotherapy and behavior change. Wiley.
Cohen ZD DeRubeis RJ Treatment Selection in Depression Annual Review of Clinical Psychology 2018 14 1 28 10.1146/annurev-clinpsy-050817
Cohen, Z. D., & DeRubeis, R. J. (2018). Treatment Selection in Depression. Annual Review of Clinical Psychology, 14, 1–28. 10.1146/annurev-clinpsy-05081710.1146/annurev-clinpsy-050817
Cooper M Xu D Goals Form: Reliability, validity, and clinical utility of an idiographic goal-focused measure for routine outcome monitoring in psychotherapy Journal of Clinical Psychology 2023 79 3 641 666 10.1002/jclp.23344 35366375
Cooper, M., & Xu, D. (2023). Goals Form: Reliability, validity, and clinical utility of an idiographic goal-focused measure for routine outcome monitoring in psychotherapy. Journal of Clinical Psychology, 79(3), 641–666. 10.1002/jclp.2334435366375 10.1002/jclp.23344
Damschroder LJ Aron DC Keith RE Kirsh SR Alexander JA Lowery JC Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science Implementation Science 2009 4 1 50 10.1186/1748-5908-4-50 19664226
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 50. 10.1186/1748-5908-4-5019664226 10.1186/1748-5908-4-50
Damschroder LJ Reardon CM Widerquist MAO Lowery J The updated consolidated framework for implementation research based on user feedback Implementation Science 2022 17 1 75 10.1186/s13012-022-01245-0 36309746
Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022). The updated consolidated framework for implementation research based on user feedback. Implementation Science, 17(1), 75. 10.1186/s13012-022-01245-036309746 10.1186/s13012-022-01245-0
Darnall BD Sturgeon JA Cook KF Taub CJ Roy A Burns JW Sullivan M Mackey SC Development and validation of a daily Pain Catastrophizing Scale The Journal of Pain 2017 18 9 1139 1149 10.1016/j.jpain.2017.05.003 28528981
Darnall, B. D., Sturgeon, J. A., Cook, K. F., Taub, C. J., Roy, A., Burns, J. W., Sullivan, M., & Mackey, S. C. (2017). Development and validation of a daily Pain Catastrophizing Scale. The Journal of Pain, 18(9), 1139–1149. 10.1016/j.jpain.2017.05.00328528981 10.1016/j.jpain.2017.05.003
Delgadillo J Lutz W A development pathway towards precision mental health care JAMA Psychiatry 2020 10.1001/jamapsychiatry.2020.1048 32459326
Delgadillo, J., & Lutz, W. (2020). A development pathway towards precision mental health care. JAMA Psychiatry. 10.1001/jamapsychiatry.2020.104832459326 10.1001/jamapsychiatry.2020.1048
Derogatis, L. R., & Unger, R. (2010). Symptom checklist-90-revised. In The Corsini Encyclopedia of Psychology (pp. 1–2). American Cancer Society. 10.1002/9780470479216.corpsy0970
DeRubeis RJ Cohen ZD Forand NR Fournier JC Gelfand LA Lorenzo-Luaces L The Personalized Advantage Index: Translating research on prediction into individualized treatment recommendations. A demonstration PLoS ONE 2014 9 1 e83875 10.1371/journal.pone.0083875 24416178
DeRubeis, R. J., Cohen, Z. D., Forand, N. R., Fournier, J. C., Gelfand, L. A., & Lorenzo-Luaces, L. (2014). The Personalized Advantage Index: Translating research on prediction into individualized treatment recommendations. A demonstration. PLoS ONE, 9(1), e83875. 10.1371/journal.pone.008387524416178 10.1371/journal.pone.0083875
Domecq JP Prutsky G Elraiyah T Wang Z Nabhan M Shippee N Brito JP Boehmer K Hasan R Firwana B Erwin P Eton D Sloan J Montori V Asi N Abu Dabrh AM Murad MH Patient engagement in research: A systematic review BMC Health Services Research 2014 14 1 89 10.1186/1472-6963-14-89 24568690
Domecq, J. P., Prutsky, G., Elraiyah, T., Wang, Z., Nabhan, M., Shippee, N., Brito, J. P., Boehmer, K., Hasan, R., Firwana, B., Erwin, P., Eton, D., Sloan, J., Montori, V., Asi, N., Abu Dabrh, A. M., & Murad, M. H. (2014). Patient engagement in research: A systematic review. BMC Health Services Research, 14(1), 89. 10.1186/1472-6963-14-8924568690 10.1186/1472-6963-14-89
Doré BP Silvers JA Ochsner KN Toward a personalized science of emotion regulation Social and Personality Psychology Compass 2016 10 4 171 187 10.1111/spc3.12240 29750085
Doré, B. P., Silvers, J. A., & Ochsner, K. N. (2016). Toward a personalized science of emotion regulation. Social and Personality Psychology Compass, 10(4), 171–187. 10.1111/spc3.1224029750085 10.1111/spc3.12240
Doria N Condran B Boulos L Curtis Maillet DG Dowling L Levy A Sharpening the focus: Differentiating between focus groups for patient engagement vs. qualitative research Research Involvement and Engagement 2018 4 1 19 10.1186/s40900-018-0102-6 29983994
Doria, N., Condran, B., Boulos, L., Curtis Maillet, D. G., Dowling, L., & Levy, A. (2018). Sharpening the focus: Differentiating between focus groups for patient engagement vs. qualitative research. Research Involvement and Engagement, 4(1), 19. 10.1186/s40900-018-0102-629983994 10.1186/s40900-018-0102-6
Duan N Kravitz RL Schmid CH Single-patient (n-of-1) trials: A pragmatic clinical decision methodology for patient-centered comparative effectiveness research Journal of Clinical Epidemiology 2013 66 8 S21 S28 10.1016/j.jclinepi.2013.04.006 23849149
Duan, N., Kravitz, R. L., & Schmid, C. H. (2013). Single-patient (n-of-1) trials: A pragmatic clinical decision methodology for patient-centered comparative effectiveness research. Journal of Clinical Epidemiology, 66(8), S21–S28. 10.1016/j.jclinepi.2013.04.00623849149 10.1016/j.jclinepi.2013.04.006
Eisele G Vachon H Lafit G Kuppens P Houben M Myin-Germeys I Viechtbauer W The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population Assessment 2020 29 2 136 151 10.1177/1073191120957102 32909448
Eisele, G., Vachon, H., Lafit, G., Kuppens, P., Houben, M., Myin-Germeys, I., & Viechtbauer, W. (2020). The effects of sampling frequency and questionnaire length on perceived burden, compliance, and careless responding in experience sampling data in a student population. Assessment, 29(2), 136–151. 10.1177/107319112095710232909448 10.1177/1073191120957102
Elliott R Wagner J Sales CMD Rodgers B Alves P Café MJ Psychometrics of the Personal Questionnaire: A client-generated outcome measure Psychological Assessment 2016 28 3 263 278 10.1037/pas0000174 26075406
Elliott, R., Wagner, J., Sales, C. M. D., Rodgers, B., Alves, P., & Café, M. J. (2016). Psychometrics of the Personal Questionnaire: A client-generated outcome measure. Psychological Assessment, 28(3), 263–278. 10.1037/pas000017426075406 10.1037/pas0000174
Epton T Currie S Armitage CJ Unique effects of setting goals on behavior change: Systematic review and meta-analysis Journal of Consulting and Clinical Psychology 2017 85 12 1182 1198 10.1037/ccp0000260 29189034
Epton, T., Currie, S., & Armitage, C. J. (2017). Unique effects of setting goals on behavior change: Systematic review and meta-analysis. Journal of Consulting and Clinical Psychology, 85(12), 1182–1198. 10.1037/ccp000026029189034 10.1037/ccp0000260
Eslick, I., & Sim, I. (2014). Information Technology (IT) infrastructure for N-of-1 trials. In R. L. Kravitz, N. Duan, & DEcIDE Methods Center N-of-1 Guidance Panel (Duan N, Eslick I, Gabler NB, Kaplan HC, Kravitz RL, Larson EB, Pace WD, Schmid CH, Sim I, Vohra S), Design and implementation of N-of-1 trials: A user’s guide: Vol. AHRQ Publication No. 13(14)-EHC122-EF (pp. 55–70). Agency for Healthcare Research and Quality.
Fernandez KC Jazaieri H Gross JJ Emotion regulation: A transdiagnostic perspective on a new RDoC domain Cognitive Therapy and Research 2016 40 3 426 440 10.1007/s10608-016-9772-2 27524846
Fernandez, K. C., Jazaieri, H., & Gross, J. J. (2016). Emotion regulation: A transdiagnostic perspective on a new RDoC domain. Cognitive Therapy and Research, 40(3), 426–440. 10.1007/s10608-016-9772-227524846 10.1007/s10608-016-9772-2
Fisher AJ Toward a dynamic model of psychological assessment: Implications for personalized care Journal of Consulting and Clinical Psychology 2015 83 4 825 836 10.1037/ccp0000026 26009781
Fisher, A. J. (2015). Toward a dynamic model of psychological assessment: Implications for personalized care. Journal of Consulting and Clinical Psychology, 83(4), 825–836. 10.1037/ccp000002626009781 10.1037/ccp0000026
Fisher AJ Medaglia JD Jeronimus BF Lack of group-to-individual generalizability is a threat to human subjects research Proceedings of the National Academy of Sciences 2018 115 27 E6106 E6115 10.1073/pnas.1711978115
Fisher, A. J., Medaglia, J. D., & Jeronimus, B. F. (2018). Lack of group-to-individual generalizability is a threat to human subjects research. Proceedings of the National Academy of Sciences, 115(27), E6106–E6115. 10.1073/pnas.171197811510.1073/pnas.1711978115
Franke GH Symptom-Checklist-90-Standard (1 2014 Hogrefe
Franke, G. H. (2014). Symptom-Checklist-90-Standard (1. Hogrefe.
Frumkin MR Piccirillo ML Beck ED Grossman JT Rodebaugh TL Feasibility and utility of idiographic models in the clinic: A pilot study Psychotherapy Research: Journal of the Society for Psychotherapy Research 2021 31 4 520 534 10.1080/10503307.2020.1805133 32838671
Frumkin, M. R., Piccirillo, M. L., Beck, E. D., Grossman, J. T., & Rodebaugh, T. L. (2021). Feasibility and utility of idiographic models in the clinic: A pilot study. Psychotherapy Research: Journal of the Society for Psychotherapy Research, 31(4), 520–534. 10.1080/10503307.2020.180513332838671 10.1080/10503307.2020.1805133
Gross JJ Emotion regulation: Current status and future prospects Psychological Inquiry 2015 26 1 1 26 10.1080/1047840X.2014.940781
Gross, J. J. (2015). Emotion regulation: Current status and future prospects. Psychological Inquiry, 26(1), 1–26. 10.1080/1047840X.2014.94078110.1080/1047840X.2014.940781
Gross JJ Jazaieri H Emotion, emotion regulation, and psychopathology: An affective science perspective Clinical Psychological Science 2014 2 4 387 401 10.1177/2167702614536164
Gross, J. J., & Jazaieri, H. (2014). Emotion, emotion regulation, and psychopathology: An affective science perspective. Clinical Psychological Science, 2(4), 387–401. 10.1177/216770261453616410.1177/2167702614536164
Gruber J Harvey AG Gross JJ When trying is not enough: Emotion regulation and the effort–success gap in bipolar disorder Emotion 2012 12 5 997 1003 10.1037/a0026822 22251049
Gruber, J., Harvey, A. G., & Gross, J. J. (2012). When trying is not enough: Emotion regulation and the effort–success gap in bipolar disorder. Emotion, 12(5), 997–1003. 10.1037/a002682222251049 10.1037/a0026822
Guest G Bunce A Johnson L How many interviews are enough? An experiment with data saturation and variability Field Methods 2006 18 1 59 82 10.1177/1525822X05279903
Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59–82. 10.1177/1525822X0527990310.1177/1525822X05279903
Guo Y Albright D The effectiveness of telehealth on self-management for older adults with a chronic condition: A comprehensive narrative review of the literature Journal of Telemedicine and Telecare 2018 24 6 392 403 10.1177/1357633X17706285 28449619
Guo, Y., & Albright, D. (2018). The effectiveness of telehealth on self-management for older adults with a chronic condition: A comprehensive narrative review of the literature. Journal of Telemedicine and Telecare, 24(6), 392–403. 10.1177/1357633X1770628528449619 10.1177/1357633X17706285
Harkin B Webb TL Chang BPI Prestwich A Conner M Kellar I Benn Y Sheeran P Does monitoring goal progress promote goal attainment? A meta-analysis of the experimental evidence Psychological Bulletin 2016 142 2 198 229 10.1037/bul0000025 26479070
Harkin, B., Webb, T. L., Chang, B. P. I., Prestwich, A., Conner, M., Kellar, I., Benn, Y., & Sheeran, P. (2016). Does monitoring goal progress promote goal attainment? A meta-analysis of the experimental evidence. Psychological Bulletin, 142(2), 198–229. 10.1037/bul000002526479070 10.1037/bul0000025
Hayes SC Single case experimental design and empirical clinical practice Journal of Consulting and Clinical Psychology 1981 49 2 193 211 10.1037/0022-006X.49.2.193 7217485
Hayes, S. C. (1981). Single case experimental design and empirical clinical practice. Journal of Consulting and Clinical Psychology, 49(2), 193–211. 10.1037/0022-006X.49.2.1937217485 10.1037/0022-006X.49.2.193
Herzog P Kaiser T Is it worth it to personalize the treatment of PTSD? - A variance-ratio meta-analysis and estimation of treatment effect heterogeneity in RCTs of PTSD Journal of Anxiety Disorders 2022 10.1016/j.janxdis.2022.102611 35963147
Herzog, P., & Kaiser, T. (2022). Is it worth it to personalize the treatment of PTSD? - A variance-ratio meta-analysis and estimation of treatment effect heterogeneity in RCTs of PTSD. Journal of Anxiety Disorders. 10.1016/j.janxdis.2022.10261135963147 10.1016/j.janxdis.2022.102611
Herzog P Kaiser T Brakemeier E-L Praxisorientierte Forschung in der Psychotherapie: Entwicklung, Gegenwart und Zukunft Zeitschrift Für Klinische Psychologie Und Psychotherapie 2022 51 2 127 148 10.1026/1616-3443/a000665
Herzog, P., Kaiser, T., & Brakemeier, E.-L. (2022). Praxisorientierte Forschung in der Psychotherapie: Entwicklung, Gegenwart und Zukunft. Zeitschrift Für Klinische Psychologie Und Psychotherapie, 51(2), 127–148. 10.1026/1616-3443/a00066510.1026/1616-3443/a000665
Herzog P Kube T Rubel J Why some psychotherapists benefit from feedback on treatment progress more than others: A belief updating perspective Clinical Psychology: Science and Practice 2023 30 4 468 479 10.1037/cps0000174
Herzog, P., Kube, T., & Rubel, J. (2023). Why some psychotherapists benefit from feedback on treatment progress more than others: A belief updating perspective. Clinical Psychology: Science and Practice, 30(4), 468–479. 10.1037/cps000017410.1037/cps0000174
Heyvaert M Onghena P Randomization tests for single-case experiments: State of the art, state of the science, and state of the application Journal of Contextual Behavioral Science 2014 3 1 51 64 10.1016/j.jcbs.2013.10.002
Heyvaert, M., & Onghena, P. (2014). Randomization tests for single-case experiments: State of the art, state of the science, and state of the application. Journal of Contextual Behavioral Science, 3(1), 51–64. 10.1016/j.jcbs.2013.10.00210.1016/j.jcbs.2013.10.002
Holzer KJ Veasley C Kerns RD Edwards RR Gewandter JS Langford DJ Yaeger LH McNicol E Ferguson M Turk DC Dworkin RH Haroutounian S Partnering with patients in clinical trials of pain treatments: A narrative review Pain 2022 163 10 1862 1873 10.1097/j.pain.0000000000002600 35297799
Holzer, K. J., Veasley, C., Kerns, R. D., Edwards, R. R., Gewandter, J. S., Langford, D. J., Yaeger, L. H., McNicol, E., Ferguson, M., Turk, D. C., Dworkin, R. H., & Haroutounian, S. (2022). Partnering with patients in clinical trials of pain treatments: A narrative review. Pain, 163(10), 1862–1873. 10.1097/j.pain.000000000000260035297799 10.1097/j.pain.0000000000002600
Howard KI Moras K Brill PL Martinovich Z Lutz W Evaluation of psychotherapy: Efficacy, effectiveness, and patient progress American Psychologist 1996 51 10 1059 1064 10.1037/0003-066X.51.10.1059 8870542
Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z., & Lutz, W. (1996). Evaluation of psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist, 51(10), 1059–1064. 10.1037/0003-066X.51.10.10598870542 10.1037/0003-066X.51.10.1059
Janssens KAM Bos EH Rosmalen JGM Wichers MC Riese H A qualitative approach to guide choices for designing a diary study BMC Medical Research Methodology 2018 18 1 140 10.1186/s12874-018-0579-6 30445926
Janssens, K. A. M., Bos, E. H., Rosmalen, J. G. M., Wichers, M. C., & Riese, H. (2018). A qualitative approach to guide choices for designing a diary study. BMC Medical Research Methodology, 18(1), 140. 10.1186/s12874-018-0579-630445926 10.1186/s12874-018-0579-6
Kahneman D Klein G Conditions for intuitive expertise: A failure to disagree American Psychologist 2009 64 6 515 526 10.1037/a0016755 19739881
Kahneman, D., & Klein, G. (2009). Conditions for intuitive expertise: A failure to disagree. American Psychologist, 64(6), 515–526. 10.1037/a001675519739881 10.1037/a0016755
Kaiser T Brakemeier E-L Herzog P What if we wait? Using synthetic waiting lists to estimate treatment effects in routine outcome data Psychotherapy Research 2023 10.1080/10503307.2023.2182241 36857510
Kaiser, T., Brakemeier, E.-L., & Herzog, P. (2023). What if we wait? Using synthetic waiting lists to estimate treatment effects in routine outcome data. Psychotherapy Research. 10.1080/10503307.2023.218224136857510 10.1080/10503307.2023.2182241
Kaiser T Herzog P Is personalized treatment selection a promising avenue in bpd research? A meta-regression estimating treatment effect heterogeneity in RCTs of BPD Journal of Consulting and Clinical Psychology 2023 10.1037/ccp0000803 36795433
Kaiser, T., & Herzog, P. (2023). Is personalized treatment selection a promising avenue in bpd research? A meta-regression estimating treatment effect heterogeneity in RCTs of BPD. Journal of Consulting and Clinical Psychology. 10.1037/ccp000080336795433 10.1037/ccp0000803
Kaiser T Herzog P Voderholzer U Brakemeier E-L Out of sight, out of mind? High discrepancy between observer- and patient-reported outcome after routine inpatient treatment for depression Journal of Affective Disorders 2022 300 322 325 10.1016/j.jad.2022.01.019 34995701
Kaiser, T., Herzog, P., Voderholzer, U., & Brakemeier, E.-L. (2022a). Out of sight, out of mind? High discrepancy between observer- and patient-reported outcome after routine inpatient treatment for depression. Journal of Affective Disorders, 300, 322–325. 10.1016/j.jad.2022.01.01934995701 10.1016/j.jad.2022.01.019
Kaiser T Volkmann C Volkmann A Karyotaki E Cuijpers P Brakemeier E-L Heterogeneity of treatment effects in trials on psychotherapy of depression Clinical Psychology: Science and Practice. 2022 10.1037/cps0000079
Kaiser, T., Volkmann, C., Volkmann, A., Karyotaki, E., Cuijpers, P., & Brakemeier, E.-L. (2022b). Heterogeneity of treatment effects in trials on psychotherapy of depression. Clinical Psychology: Science and Practice.10.1037/cps000007910.1037/cps0000079
Kaplan, H. C., & Gabler, N. B. (2014). User engagement, training, and support for conducting N-of-1 trials. In R. L. Kravitz, N. Duan, & DEcIDE Methods Center N-of-1 Guidance Panel (Duan N, Eslick I, Gabler NB, Kaplan HC, Kravitz RL, Larson EB, Pace WD, Schmid CH, Sim I, Vohra S), Design and implementation of N-of-1 trials: A user’s guide: Vol. AHRQ Publication No. 13(14)-EHC122-EF (pp. 71–81). Agency for Healthcare Research and Quality.
Kazdin AE Single-case experimental designs Evaluating interventions in research and clinical practice Behaviour Research and Therapy 2019 117 3 17 10.1016/j.brat.2018.11.015 30527785
Kazdin, A. E. (2019). Single-case experimental designs Evaluating interventions in research and clinical practice. Behaviour Research and Therapy, 117, 3–17. 10.1016/j.brat.2018.11.01530527785 10.1016/j.brat.2018.11.015
Kiesler DJ Some myths of psychotherapy research and the search for a paradigm Psychological Bulletin 1966 65 2 110 136 10.1037/h0022911
Kiesler, D. J. (1966). Some myths of psychotherapy research and the search for a paradigm. Psychological Bulletin, 65(2), 110–136. 10.1037/h002291110.1037/h0022911
Kirchner JE Smith JL Powell BJ Waltz TJ Proctor EK Getting a clinical innovation into practice: An introduction to implementation strategies Psychiatry Research 2020 283 112467 10.1016/j.psychres.2019.06.042 31488332
Kirchner, J. E., Smith, J. L., Powell, B. J., Waltz, T. J., & Proctor, E. K. (2020). Getting a clinical innovation into practice: An introduction to implementation strategies. Psychiatry Research, 283, 112467. 10.1016/j.psychres.2019.06.04231488332 10.1016/j.psychres.2019.06.042
Kiresuk TJ Sherman RE Goal attainment scaling: A general method for evaluating comprehensive community mental health programs Community Mental Health Journal 1968 4 6 443 453 10.1007/BF01530764 24185570
Kiresuk, T. J., & Sherman, R. E. (1968). Goal attainment scaling: A general method for evaluating comprehensive community mental health programs. Community Mental Health Journal, 4(6), 443–453. 10.1007/BF0153076424185570 10.1007/BF01530764
Krasny-Pacini A Evans J Single-case experimental designs to assess intervention effectiveness in rehabilitation: A practical guide Annals of Physical and Rehabilitation Medicine 2018 61 3 164 179 10.1016/j.rehab.2017.12.002 29253607
Krasny-Pacini, A., & Evans, J. (2018). Single-case experimental designs to assess intervention effectiveness in rehabilitation: A practical guide. Annals of Physical and Rehabilitation Medicine, 61(3), 164–179. 10.1016/j.rehab.2017.12.00229253607 10.1016/j.rehab.2017.12.002
Kravitz RL Aguilera A Chen EJ Choi YK Hekler E Karr C Kim KK Phatak S Sarkar S Schueller SM Sim I Yang J Schmid CH Feasibility, acceptability, and influence of mHealth-supported N-of-1 trials for enhanced cognitive and emotional well-being in US volunteers Frontiers in Public Health 2020 8 260 10.3389/fpubh.2020.00260 32695740
Kravitz, R. L., Aguilera, A., Chen, E. J., Choi, Y. K., Hekler, E., Karr, C., Kim, K. K., Phatak, S., Sarkar, S., Schueller, S. M., Sim, I., Yang, J., & Schmid, C. H. (2020). Feasibility, acceptability, and influence of mHealth-supported N-of-1 trials for enhanced cognitive and emotional well-being in US volunteers. Frontiers in Public Health, 8, 260. 10.3389/fpubh.2020.0026032695740 10.3389/fpubh.2020.00260
Kravitz, R. L., Duan, N., Vohra, S., & Li, J. (2014). Introduction to N-of-1 trials: Indications and barriers. In R. L. Kravitz, N. Duan, & DEcIDE Methods Center N-of-1 Guidance Panel (Duan N, Eslick I, Gabler NB, Kaplan HC, Kravitz RL, Larson EB, Pace WD, Schmid CH, Sim I, Vohra S), Design and implementation of N-of-1 Trials: A user’s guide: Vol. AHRQ Publication No. 13(14)-EHC122-EF (pp. 1–11). Agency for Healthcare Research and Quality.
Lambert MJ Harmon KL The merits of implementing routine outcome monitoring in clinical practice Clinical Psychology: Science and Practice 2018 25 4 e12268 10.1111/cpsp.12268
Lambert, M. J., & Harmon, K. L. (2018). The merits of implementing routine outcome monitoring in clinical practice. Clinical Psychology: Science and Practice, 25(4), e12268. 10.1111/cpsp.1226810.1111/cpsp.12268
Lavefjord A Sundström FTA Buhrman M McCracken LM Assessment methods in single case design studies of psychological treatments for chronic pain: A scoping review Journal of Contextual Behavioral Science 2021 21 121 135 10.1016/j.jcbs.2021.05.005
Lavefjord, A., Sundström, F. T. A., Buhrman, M., & McCracken, L. M. (2021). Assessment methods in single case design studies of psychological treatments for chronic pain: A scoping review. Journal of Contextual Behavioral Science, 21, 121–135. 10.1016/j.jcbs.2021.05.00510.1016/j.jcbs.2021.05.005
Lloyd CEM Duncan C Cooper M Goal measures for psychotherapy: A systematic review of self-report, idiographic instruments Clinical Psychology: Science and Practice 2019 26 3 e12281 10.1111/cpsp.12281
Lloyd, C. E. M., Duncan, C., & Cooper, M. (2019). Goal measures for psychotherapy: A systematic review of self-report, idiographic instruments. Clinical Psychology: Science and Practice, 26(3), e12281. 10.1111/cpsp.1228110.1111/cpsp.12281
Lorenzo-Luaces L Peipert A De Jesús Romero R Rutter LA Rodriguez-Quintana N Personalized medicine and cognitive behavioral therapies for depression: Small effects, big problems, and bigger data International Journal of Cognitive Therapy 2021 14 1 59 85 10.1007/s41811-020-00094-3
Lorenzo-Luaces, L., Peipert, A., De Jesús Romero, R., Rutter, L. A., & Rodriguez-Quintana, N. (2021). Personalized medicine and cognitive behavioral therapies for depression: Small effects, big problems, and bigger data. International Journal of Cognitive Therapy, 14(1), 59–85. 10.1007/s41811-020-00094-310.1007/s41811-020-00094-3
Lutz W Rubel JA Schwartz B Schilling V Deisenhofer A-K Towards integrating personalized feedback research into clinical practice: Development of the Trier Treatment Navigator (TTN) Behaviour Research and Therapy 2019 120 103438 10.1016/j.brat.2019.103438 31301550
Lutz, W., Rubel, J. A., Schwartz, B., Schilling, V., & Deisenhofer, A.-K. (2019). Towards integrating personalized feedback research into clinical practice: Development of the Trier Treatment Navigator (TTN). Behaviour Research and Therapy, 120, 103438. 10.1016/j.brat.2019.10343831301550 10.1016/j.brat.2019.103438
Lutz W Schwartz B Delgadillo J Measurement-based and data-informed psychological therapy Annual Review of Clinical Psychology 2022 18 1 071720 114821 10.1146/annurev-clinpsy-071720-014821
Lutz, W., Schwartz, B., & Delgadillo, J. (2022). Measurement-based and data-informed psychological therapy. Annual Review of Clinical Psychology, 18(1), 071720–114821. 10.1146/annurev-clinpsy-071720-01482110.1146/annurev-clinpsy-071720-014821
Lyon AR Koerner K Chung J Usability Evaluation for Evidence-Based Psychosocial Interventions (USE-EBPI): A methodology for assessing complex intervention implementability Implementation Research and Practice 2020 1 263348952093292 10.1177/2633489520932924
Lyon, A. R., Koerner, K., & Chung, J. (2020). Usability Evaluation for Evidence-Based Psychosocial Interventions (USE-EBPI): A methodology for assessing complex intervention implementability. Implementation Research and Practice, 1, 263348952093292. 10.1177/263348952093292410.1177/2633489520932924
McBain H Shipley M Newman S The impact of self-monitoring in chronic illness on healthcare utilisation: A systematic review of reviews BMC Health Services Research 2015 15 1 565 10.1186/s12913-015-1221-5 26684011
McBain, H., Shipley, M., & Newman, S. (2015). The impact of self-monitoring in chronic illness on healthcare utilisation: A systematic review of reviews. BMC Health Services Research, 15(1), 565. 10.1186/s12913-015-1221-526684011 10.1186/s12913-015-1221-5
McDonald AM Treweek S Shakur H Free C Knight R Speed C Campbell MK Using a business model approach and marketing techniques for recruitment to clinical trials Trials 2011 12 1 74 10.1186/1745-6215-12-74 21396088
McDonald, A. M., Treweek, S., Shakur, H., Free, C., Knight, R., Speed, C., & Campbell, M. K. (2011). Using a business model approach and marketing techniques for recruitment to clinical trials. Trials, 12(1), 74. 10.1186/1745-6215-12-7421396088 10.1186/1745-6215-12-74
Mestdagh M Verdonck S Piot M Niemeijer K Tuerlinckx F Kuppens P Dejonckheere E m-Path: An easy-to-use and flexible platform for ecological momentary assessment and intervention in behavioral research and clinical practice PsyArXiv 2022 10.31234/osf.io/uqdfs
Mestdagh, M., Verdonck, S., Piot, M., Niemeijer, K., Tuerlinckx, F., Kuppens, P., & Dejonckheere, E. (2022). m-Path: An easy-to-use and flexible platform for ecological momentary assessment and intervention in behavioral research and clinical practice. PsyArXiv. 10.31234/osf.io/uqdfs10.31234/osf.io/uqdfs
Mettert K Lewis C Dorsey C Halko H Weiner B Measuring implementation outcomes: An updated systematic review of measures’ psychometric properties Implementation Research and Practice 2020 1 263348952093664 10.1177/2633489520936644
Mettert, K., Lewis, C., Dorsey, C., Halko, H., & Weiner, B. (2020). Measuring implementation outcomes: An updated systematic review of measures’ psychometric properties. Implementation Research and Practice, 1, 263348952093664. 10.1177/263348952093664410.1177/2633489520936644
Miller CJ Smith SN Pugatch M Experimental and quasi-experimental designs in implementation research Psychiatry Research 2020 283 112452 10.1016/j.psychres.2019.06.027 31255320
Miller, C. J., Smith, S. N., & Pugatch, M. (2020). Experimental and quasi-experimental designs in implementation research. Psychiatry Research, 283, 112452. 10.1016/j.psychres.2019.06.02731255320 10.1016/j.psychres.2019.06.027
Molenaar PCM A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever Measurement: Interdisciplinary Research & Perspective 2004 2 4 201 218 10.1207/s15366359mea0204_1
Molenaar, P. C. M. (2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement: Interdisciplinary Research & Perspective, 2(4), 201–218. 10.1207/s15366359mea0204_110.1207/s15366359mea0204_1
Moore GC Benbasat I Development of an instrument to measure the perceptions of adopting aniInformation technology innovation Information Systems Research 1991 2 3 192 222 10.1287/isre.2.3.192
Moore, G. C., & Benbasat, I. (1991). Development of an instrument to measure the perceptions of adopting aniInformation technology innovation. Information Systems Research, 2(3), 192–222. 10.1287/isre.2.3.19210.1287/isre.2.3.192
Morris M Schindehutte M Allen J The entrepreneur’s business model: Toward a unified perspective Journal of Business Research 2005 58 6 726 735 10.1016/j.jbusres.2003.11.001
Morris, M., Schindehutte, M., & Allen, J. (2005). The entrepreneur’s business model: Toward a unified perspective. Journal of Business Research, 58(6), 726–735. 10.1016/j.jbusres.2003.11.00110.1016/j.jbusres.2003.11.001
Myin-Germeys I Kuppens P The open handbook of experience sampling methodology: A step-by-step guide to designing, conducting, and analyzing ESM studies 2021 Amazon
Myin-Germeys, I., & Kuppens, P. (Eds.). (2021). The open handbook of experience sampling methodology: A step-by-step guide to designing, conducting, and analyzing ESM studies. Amazon.
Nikles J Onghena P Vlaeyen JWS Wicksell RK Simons LE McGree JM McDonald S Establishment of an international collaborative network for n-of-1 trials and single-case designs Contemporary Clinical Trials Communications 2021 23 100826 10.1016/j.conctc.2021.100826 34401597
Nikles, J., Onghena, P., Vlaeyen, J. W. S., Wicksell, R. K., Simons, L. E., McGree, J. M., & McDonald, S. (2021). Establishment of an international collaborative network for n-of-1 trials and single-case designs. Contemporary Clinical Trials Communications, 23, 100826. 10.1016/j.conctc.2021.10082634401597 10.1016/j.conctc.2021.100826
Nikles J O’Sullivan JD Mitchell GK Smith SS McGree JM Senior H Dissanyaka N Ritchie A Protocol: Using N-of-1 tests to identify responders to melatonin for sleep disturbance in Parkinson’s disease Contemporary Clinical Trials Communications 2019 15 100397 10.1016/j.conctc.2019.100397 31338478
Nikles, J., O’Sullivan, J. D., Mitchell, G. K., Smith, S. S., McGree, J. M., Senior, H., Dissanyaka, N., & Ritchie, A. (2019). Protocol: Using N-of-1 tests to identify responders to melatonin for sleep disturbance in Parkinson’s disease. Contemporary Clinical Trials Communications, 15, 100397. 10.1016/j.conctc.2019.10039731338478 10.1016/j.conctc.2019.100397
Nosek BA Alter G Banks GC Borsboom D Bowman SD Breckler SJ Buck S Chambers CD Chin G Christensen G Contestabile M Dafoe A Eich E Freese J Glennerster R Goroff D Green DP Hesse B Humphreys M Yarkoni T Promoting an open research culture Science 2015 348 6242 1422 1425 10.1126/science.aab2374 26113702
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., & Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. 10.1126/science.aab237426113702 10.1126/science.aab2374
Osterwalder A Pigneur Y Clark T Business model generation: A handbook for visionaries, game changers, and challengers 2010 Wiley
Osterwalder, A., Pigneur, Y., & Clark, T. (2010). Business model generation: A handbook for visionaries, game changers, and challengers. Wiley.
Palmieri A Fernandez KC Cariolato Y Kleinbub JR Salvatore S Gross JJ Emotion regulation in psychodynamic and cognitive-behavioural therapy: An integrative perspective Clinical Neuropsychiatry. 2022 19 2 103 113 35601247
Palmieri, A., Fernandez, K. C., Cariolato, Y., Kleinbub, J. R., Salvatore, S., & Gross, J. J. (2022). Emotion regulation in psychodynamic and cognitive-behavioural therapy: An integrative perspective. Clinical Neuropsychiatry., 19(2), 103–113.35601247
Paul GL Strategy of outcome research in psychotherapy Journal of Consulting Psychology 1967 31 2 109 118 10.1037/h0024436 5342732
Paul, G. L. (1967). Strategy of outcome research in psychotherapy. Journal of Consulting Psychology, 31(2), 109–118. 10.1037/h00244365342732 10.1037/h0024436
Piccirillo ML Enkema MC Foster KT Using the experience sampling method to support clinical practice: An illustration with problematic cannabis use Psychotherapy: Research Journal of the Society for Psychotherapy Research 2023 10.1080/10503307.2023.2184284 36976153
Piccirillo, M. L., Enkema, M. C., & Foster, K. T. (2023). Using the experience sampling method to support clinical practice: An illustration with problematic cannabis use. Psychotherapy: Research Journal of the Society for Psychotherapy Research. 10.1080/10503307.2023.218428436976153 10.1080/10503307.2023.2184284
Piccirillo ML Rodebaugh TL Foundations of idiographic methods in psychology and applications for psychotherapy Clinical Psychology Review 2019 71 90 100 10.1016/j.cpr.2019.01.002 30665765
Piccirillo, M. L., & Rodebaugh, T. L. (2019). Foundations of idiographic methods in psychology and applications for psychotherapy. Clinical Psychology Review, 71, 90–100. 10.1016/j.cpr.2019.01.00230665765 10.1016/j.cpr.2019.01.002
Pinnock H Barwick M Carpenter CR Eldridge S Grandes G Griffiths CJ Rycroft-Malone J Meissner P Murray E Patel A Sheikh A Taylor SJC Standards for reporting implementation studies (StaRI) statement BMJ 2017 356 i6795 10.1136/bmj.i6795 28264797
Pinnock, H., Barwick, M., Carpenter, C. R., Eldridge, S., Grandes, G., Griffiths, C. J., Rycroft-Malone, J., Meissner, P., Murray, E., Patel, A., Sheikh, A., & Taylor, S. J. C. (2017). Standards for reporting implementation studies (StaRI) statement. BMJ, 356, i6795. 10.1136/bmj.i679528264797 10.1136/bmj.i6795
Piot M Mestdagh M Riese H Weermeijer J Brouwer JMA Kuppens P Dejonckheere E Bos FM Practitioner and researcher perspectives on the utility of ecological momentary assessment in mental health care: A survey study Internet Interventions 2022 30 100575 10.1016/j.invent.2022.100575 36193339
Piot, M., Mestdagh, M., Riese, H., Weermeijer, J., Brouwer, J. M. A., Kuppens, P., Dejonckheere, E., & Bos, F. M. (2022). Practitioner and researcher perspectives on the utility of ecological momentary assessment in mental health care: A survey study. Internet Interventions, 30, 100575. 10.1016/j.invent.2022.10057536193339 10.1016/j.invent.2022.100575
Powell BJ Waltz TJ Chinman MJ Damschroder LJ Smith JL Matthieu MM Proctor EK Kirchner JE A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project Implementation Science 2015 10 1 21 10.1186/s13012-015-0209-1 25889199
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., Proctor, E. K., & Kirchner, J. E. (2015). A refined compilation of implementation strategies: Results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), 21. 10.1186/s13012-015-0209-125889199 10.1186/s13012-015-0209-1
Proctor EK Powell BJ McMillen JC Implementation strategies: Recommendations for specifying and reporting Implementation Science 2013 8 1 139 10.1186/1748-5908-8-139 24289295
Proctor, E. K., Powell, B. J., & McMillen, J. C. (2013). Implementation strategies: Recommendations for specifying and reporting. Implementation Science, 8(1), 139. 10.1186/1748-5908-8-13924289295 10.1186/1748-5908-8-139
Proctor EK Silmere H Raghavan R Hovmand P Aarons G Bunger A Griffey R Hensley M Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda Administration and Policy in Mental Health 2011 38 2 65 76 10.1007/s10488-010-0319-7 20957426
Proctor, E. K., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38(2), 65–76. 10.1007/s10488-010-0319-720957426 10.1007/s10488-010-0319-7
Riese H von Klipstein L Schoevers RA van der Veen DC Servaas MN Personalized ESM monitoring and feedback to support psychological treatment for depression: A pragmatic randomized controlled trial (Therap-i) BMC Psychiatry 2021 21 1 143 10.1186/s12888-021-03123-3 33691647
Riese, H., von Klipstein, L., Schoevers, R. A., van der Veen, D. C., & Servaas, M. N. (2021). Personalized ESM monitoring and feedback to support psychological treatment for depression: A pragmatic randomized controlled trial (Therap-i). BMC Psychiatry, 21(1), 143. 10.1186/s12888-021-03123-333691647 10.1186/s12888-021-03123-3
Rintala A Wampers M Myin-Germeys I Viechtbauer W Response compliance and predictors thereof in studies using the experience sampling method Psychological Assessment 2019 31 2 2 10.1037/pas0000662
Rintala, A., Wampers, M., Myin-Germeys, I., & Viechtbauer, W. (2019). Response compliance and predictors thereof in studies using the experience sampling method. Psychological Assessment, 31(2), 2. 10.1037/pas000066210.1037/pas0000662
Ryan O Haslbeck JMB Waldorp L Non-stationarity in time-series analysis: Modeling stochastic and deterministic trends PsyArXiv 2023 10.31234/osf.io/z7ja2
Ryan, O., Haslbeck, J. M. B., & Waldorp, L. (2023). Non-stationarity in time-series analysis: Modeling stochastic and deterministic trends. PsyArXiv. 10.31234/osf.io/z7ja210.31234/osf.io/z7ja2
Sales CMD Alves PCG Patient-centered assessment in psychotherapy: A review of individualized tools Clinical Psychology: Science and Practice 2016 23 3 265 283 10.1037/h0101737
Sales, C. M. D., & Alves, P. C. G. (2016). Patient-centered assessment in psychotherapy: A review of individualized tools. Clinical Psychology: Science and Practice, 23(3), 265–283. 10.1037/h010173710.1037/h0101737
Sales CMD Ashworth M Ayis S Barkham M Edbrooke-Childs J Faísca L Jacob J Xu D Cooper M Idiographic patient reported outcome measures (I-PROMs) for routine outcome monitoring in psychological therapies: Position paper Journal of Clinical Psychology 2023 79 3 596 621 10.1002/jclp.23319 35194799
Sales, C. M. D., Ashworth, M., Ayis, S., Barkham, M., Edbrooke-Childs, J., Faísca, L., Jacob, J., Xu, D., & Cooper, M. (2023). Idiographic patient reported outcome measures (I-PROMs) for routine outcome monitoring in psychological therapies: Position paper. Journal of Clinical Psychology, 79(3), 596–621. 10.1002/jclp.2331935194799 10.1002/jclp.23319
Schemer L Glombiewski JA Scholten S All good things come in threes: A systematic review and Delphi study on the advances and challenges of ambulatory assessments, network analyses, and single-case experimental designs Clinical Psychology: Science and Practice 2022 10.1037/cps0000083
Schemer, L., Glombiewski, J. A., & Scholten, S. (2022). All good things come in threes: A systematic review and Delphi study on the advances and challenges of ambulatory assessments, network analyses, and single-case experimental designs. Clinical Psychology: Science and Practice. 10.1037/cps000008310.1037/cps0000083
Schemer L Vlaeyen JWS Doerr JM Skoluda N Nater UM Rief W Glombiewski JA Treatment processes during exposure and cognitive-behavioral therapy for chronic back pain: A single-case experimental design with multiple baselines Behaviour Research and Therapy 2018 108 58 67 10.1016/j.brat.2018.07.002 30031368
Schemer, L., Vlaeyen, J. W. S., Doerr, J. M., Skoluda, N., Nater, U. M., Rief, W., & Glombiewski, J. A. (2018). Treatment processes during exposure and cognitive-behavioral therapy for chronic back pain: A single-case experimental design with multiple baselines. Behaviour Research and Therapy, 108, 58–67. 10.1016/j.brat.2018.07.00230031368 10.1016/j.brat.2018.07.002
Schiele H Krummaker S Hoffmann P Kowalski R The “research world café” as method of scientific enquiry: Combining rigor with relevance and speed Journal of Business Research 2022 140 280 296 10.1016/j.jbusres.2021.10.075
Schiele, H., Krummaker, S., Hoffmann, P., & Kowalski, R. (2022). The “research world café” as method of scientific enquiry: Combining rigor with relevance and speed. Journal of Business Research, 140, 280–296. 10.1016/j.jbusres.2021.10.07510.1016/j.jbusres.2021.10.075
Schleider JL Sung J Bianco A Gonzalez A Vivian D Mullarkey MC Open pilot trial of a single-session consultation service for clients on psychotherapy wait-lists PsyArXiv 2020 10.31234/osf.io/fdwqk
Schleider, J. L., Sung, J., Bianco, A., Gonzalez, A., Vivian, D., & Mullarkey, M. C. (2020). Open pilot trial of a single-session consultation service for clients on psychotherapy wait-lists. PsyArXiv. 10.31234/osf.io/fdwqk10.31234/osf.io/fdwqk
Schrepp M Hinderks A Thomaschewski J Design and evaluation of a short version of the User Experience Questionnaire (UEQ-S) International Journal of Interactive Multimedia and Artificial Intelligence 2017 4 6 103 10.9781/ijimai.2017.09.001
Schrepp, M., Hinderks, A., & Thomaschewski, J. (2017). Design and evaluation of a short version of the User Experience Questionnaire (UEQ-S). International Journal of Interactive Multimedia and Artificial Intelligence, 4(6), 103. 10.9781/ijimai.2017.09.00110.9781/ijimai.2017.09.001
Selker HP Cohen T D’Agostino RB Dere WH Ghaemi SN Honig PK Kaitin KI Kaplan HC Kravitz RL Larholt K McElwee NE Oye KA Palm ME Perfetto E Ramanathan C Schmid CH Seyfert-Margolis V Trusheim M Eichler H A useful and sustainable role for N-of-1 trials in the healthcare ecosystem Clinical Pharmacology & Therapeutics 2022 112 2 224 232 10.1002/cpt.2425 34551122
Selker, H. P., Cohen, T., D’Agostino, R. B., Dere, W. H., Ghaemi, S. N., Honig, P. K., Kaitin, K. I., Kaplan, H. C., Kravitz, R. L., Larholt, K., McElwee, N. E., Oye, K. A., Palm, M. E., Perfetto, E., Ramanathan, C., Schmid, C. H., Seyfert-Margolis, V., Trusheim, M., & Eichler, H. (2022). A useful and sustainable role for N-of-1 trials in the healthcare ecosystem. Clinical Pharmacology & Therapeutics, 112(2), 224–232. 10.1002/cpt.242534551122 10.1002/cpt.2425
Shapiro MB A method of measuring psychological changes specific to the individual psychiatric patient* British Journal of Medical Psychology 1961 34 2 151 155 10.1111/j.2044-8341.1961.tb00940.x
Shapiro, M. B. (1961). A method of measuring psychological changes specific to the individual psychiatric patient*. British Journal of Medical Psychology, 34(2), 151–155. 10.1111/j.2044-8341.1961.tb00940.x10.1111/j.2044-8341.1961.tb00940.x
Sibalija J Barrett D Subasri M Bitacola L Kim RB Understanding value in a healthcare setting: An application of the business model canvas Methodological Innovations 2021 14 3 205979912110504 10.1177/20597991211050477
Sibalija, J., Barrett, D., Subasri, M., Bitacola, L., & Kim, R. B. (2021). Understanding value in a healthcare setting: An application of the business model canvas. Methodological Innovations, 14(3), 205979912110504. 10.1177/2059799121105047710.1177/20597991211050477
Simon GE Perlis RH Personalized medicine for depression: Can we match patients with treatments? The American Journal of Psychiatry 2010 167 12 1445 1455 10.1176/appi.ajp.2010.09111680 20843873
Simon, G. E., & Perlis, R. H. (2010). Personalized medicine for depression: Can we match patients with treatments? The American Journal of Psychiatry, 167(12), 1445–1455. 10.1176/appi.ajp.2010.0911168020843873 10.1176/appi.ajp.2010.09111680
Simons CJP Hartmann J , Kramer, I., Menne-Lothmann, C., Höhn, P., van Bemmel, A. L., Myin-Germeys, I., Delespaul, P., van Os, J., & Wichers, M. Effects of momentary self-monitoring on empowerment in a randomized controlled trial in patients with depression European Psychiatry 2015 30 8 8 10.1016/j.eurpsy.2015.09.004 25169443
Simons, C. J. P., Hartmann, J., Kramer, I., Menne-Lothmann, C., Höhn, P., van Bemmel, A. L., Myin-Germeys, I., Delespaul, P., van Os, J., & Wichers, M. (2015). Effects of momentary self-monitoring on empowerment in a randomized controlled trial in patients with depression. European Psychiatry, 30(8), 8. 10.1016/j.eurpsy.2015.09.00425169443 10.1016/j.eurpsy.2015.09.004
Slattery P Saeri AK Bragge P Research co-design in health: A rapid overview of reviews Health Research Policy and Systems 2020 18 1 17 10.1186/s12961-020-0528-9 32046728
Slattery, P., Saeri, A. K., & Bragge, P. (2020). Research co-design in health: A rapid overview of reviews. Health Research Policy and Systems, 18(1), 17. 10.1186/s12961-020-0528-932046728 10.1186/s12961-020-0528-9
Staniszewska S Brett J Simera I Seers K Mockford C Goodlad S Altman DG Moher D Barber R Denegri S Entwistle A Littlejohns P Morris C Suleman R Thomas V Tysall C GRIPP2 reporting checklists: Tools to improve reporting of patient and public involvement in research Research Involvement and Engagement 2017 3 1 13 10.1186/s40900-017-0062-2 29062538
Staniszewska, S., Brett, J., Simera, I., Seers, K., Mockford, C., Goodlad, S., Altman, D. G., Moher, D., Barber, R., Denegri, S., Entwistle, A., Littlejohns, P., Morris, C., Suleman, R., Thomas, V., & Tysall, C. (2017). GRIPP2 reporting checklists: Tools to improve reporting of patient and public involvement in research. Research Involvement and Engagement, 3(1), 13. 10.1186/s40900-017-0062-229062538 10.1186/s40900-017-0062-2
Tamir M Why do people regulate their emotions? A taxonomy of motives in emotion regulation Personality and Social Psychology Review 2016 20 3 199 222 10.1177/1088868315586325 26015392
Tamir, M. (2016). Why do people regulate their emotions? A taxonomy of motives in emotion regulation. Personality and Social Psychology Review, 20(3), 199–222. 10.1177/108886831558632526015392 10.1177/1088868315586325
Tanious R Onghena P Randomized single-case experimental designs in healthcare research: What, why, and how? Healthcare 2019 7 143 10.3390/healthcare7040143 31766188
Tanious, R., & Onghena, P. (2019). Randomized single-case experimental designs in healthcare research: What, why, and how? Healthcare, 7, 143.31766188 10.3390/healthcare7040143
Tate RL Perdices M Rosenkoetter U Wakim D Godbee K Togher L McDonald S Revision of a method quality rating scale for single-case experimental designs and n-of-1 trials: The 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale Neuropsychological Rehabilitation 2013 23 5 619 638 10.1080/09602011.2013.824383 24050810
Tate, R. L., Perdices, M., Rosenkoetter, U., Wakim, D., Godbee, K., Togher, L., & McDonald, S. (2013). Revision of a method quality rating scale for single-case experimental designs and n-of-1 trials: The 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale. Neuropsychological Rehabilitation, 23(5), 619–638.24050810 10.1080/09602011.2013.824383
Tindall RM Ferris M Townsend M Boschert G Moylan S A first-hand experience of co-design in mental health service design: Opportunities, challenges, and lessons International Journal of Mental Health Nursing 2021 30 6 1693 1702 10.1111/inm.12925 34390117
Tindall, R. M., Ferris, M., Townsend, M., Boschert, G., & Moylan, S. (2021). A first-hand experience of co-design in mental health service design: Opportunities, challenges, and lessons. International Journal of Mental Health Nursing, 30(6), 1693–1702. 10.1111/inm.1292534390117 10.1111/inm.12925
Turner-Bowker DM Lamoureux RE Stokes J Litcher-Kelly L Galipeau N Yaworsky A Solomon J Shields AL Informing a priori sample size estimation in qualitative concept elicitation interview studies for clinical outcome assessment instrument development Value in Health 2018 21 7 839 842 10.1016/j.jval.2017.11.014 30005756
Turner-Bowker, D. M., Lamoureux, R. E., Stokes, J., Litcher-Kelly, L., Galipeau, N., Yaworsky, A., Solomon, J., & Shields, A. L. (2018). Informing a priori sample size estimation in qualitative concept elicitation interview studies for clinical outcome assessment instrument development. Value in Health, 21(7), 839–842. 10.1016/j.jval.2017.11.01430005756 10.1016/j.jval.2017.11.014
Vachon H Viechtbauer W Rintala A Myin-Germeys I Compliance and retention with the experience sampling method over the continuum of severe mental disorders: Meta-analysis and recommendations Journal of Medical Internet Research 2019 21 12 12 10.2196/14475
Vachon, H., Viechtbauer, W., Rintala, A., & Myin-Germeys, I. (2019). Compliance and retention with the experience sampling method over the continuum of severe mental disorders: Meta-analysis and recommendations. Journal of Medical Internet Research, 21(12), 12. 10.2196/1447510.2196/14475
Varadhan R Segal JB Boyd CM Wu AW Weiss CO A framework for the analysis of heterogeneity of treatment effect in patient-centered outcomes research Journal of Clinical Epidemiology 2013 66 8 818 825 10.1016/j.jclinepi.2013.02.009 23651763
Varadhan, R., Segal, J. B., Boyd, C. M., Wu, A. W., & Weiss, C. O. (2013). A framework for the analysis of heterogeneity of treatment effect in patient-centered outcomes research. Journal of Clinical Epidemiology, 66(8), 818–825. 10.1016/j.jclinepi.2013.02.00923651763 10.1016/j.jclinepi.2013.02.009
Velten J Margraf J Benecke C Berking M In-Albon T Tania L Lutz W Schlarb A Schöttke H Willutzki U J, ürgen H. Methodenpapier zur Koordination der Datenerhebung und -auswertung an Hochschul- und Ausbildungsambulanzen für Psychotherapie (KODAP) Zeitschrift Für Klinische Psychologie Und Psychotherapie 2017 46 3 169 175 10.1026/1616-3443/a000431
Velten, J., Margraf, J., Benecke, C., Berking, M., In-Albon, T., Tania, L., Lutz, W., Schlarb, A., Schöttke, H., Willutzki, U., & Jürgen H. (2017). Methodenpapier zur Koordination der Datenerhebung und -auswertung an Hochschul- und Ausbildungsambulanzen für Psychotherapie (KODAP). Zeitschrift Für Klinische Psychologie Und Psychotherapie, 46(3), 169–175. 10.1026/1616-3443/a00043110.1026/1616-3443/a000431
Vlaeyen, J. W. S., Onghena, P., Vannest, K. J., & Kratochwill, T. R. (2022). Single-case experimental designs: Clinical research and practice. In Comprehensive clinical psychology (pp. 1–28). Elsevier. 10.1016/B978-0-12-818697-8.00191-6
Vlaeyen JWS Wicksell RK Simons LE Gentili C De TK Tate RL Vohra S Punja S Linton SJ Sniehotta FF Onghena P From Boulder to Stockholm in 70 years: Single case experimental designs in clinical research The Psychological Record 2020 10.1007/s40732-020-00402-5
Vlaeyen, J. W. S., Wicksell, R. K., Simons, L. E., Gentili, C., De, T. K., Tate, R. L., Vohra, S., Punja, S., Linton, S. J., Sniehotta, F. F., & Onghena, P. (2020). From Boulder to Stockholm in 70 years: Single case experimental designs in clinical research. The Psychological Record. 10.1007/s40732-020-00402-510.1007/s40732-020-00402-5
Weisz, J. R., Chorpita, B. F., Frye, A., Ng, M. Y., Lau, N., Bearman, S. K., Ugueto, A. M., Langer, D. A., Hoagwood, K. E., & The Research Network on Youth Mental Health Youth top problems: Using idiographic, consumer-guided assessment to identify treatment needs and to track change during psychotherapy Journal of Consulting and Clinical Psychology 2011 79 3 369 380 10.1037/a0023307 21500888
Weisz, J. R., Chorpita, B. F., Frye, A., Ng, M. Y., Lau, N., Bearman, S. K., Ugueto, A. M., Langer, D. A., Hoagwood, K. E., & The Research Network on Youth Mental Health. (2011). Youth top problems: Using idiographic, consumer-guided assessment to identify treatment needs and to track change during psychotherapy. Journal of Consulting and Clinical Psychology, 79(3), 369–380. 10.1037/a002330721500888 10.1037/a0023307
Wilson K Bell C Wilson L Witteman H Agile research to complement agile development: A proposal for an mHealth research lifecycle Npj Digital Medicine 2018 1 1 46 10.1038/s41746-018-0053-1 31304326
Wilson, K., Bell, C., Wilson, L., & Witteman, H. (2018). Agile research to complement agile development: A proposal for an mHealth research lifecycle. Npj Digital Medicine, 1(1), 46. 10.1038/s41746-018-0053-131304326 10.1038/s41746-018-0053-1
Wright, A. G. C., & Woods, A. C. (2020). Personalized models of psychopathology. In T. Widiger & T. D. Cannon (Eds.), Annual review of clinical psychology, Vol 16, 2020 (Vol. 16, pp. 49–74). Annual Reviews. 10.1146/annurev-clinpsy-102419-125032
Wright AGC Zimmermann J Applied ambulatory assessment: Integrating idiographic and nomothetic principles of measurement Psychological Assessment 2019 31 12 1467 1480 10.1037/pas0000685 30896209
Wright, A. G. C., & Zimmermann, J. (2019). Applied ambulatory assessment: Integrating idiographic and nomothetic principles of measurement. Psychological Assessment, 31(12), 1467–1480. 10.1037/pas000068530896209 10.1037/pas0000685
Wrzus C Neubauer AB Ecological momentary assessment: A meta-analysis on designs, samples, and compliance across research fields Assessment 2023 10.1177/10731911211067 35016567
Wrzus, C., & Neubauer, A. B. (2023). Ecological momentary assessment: A meta-analysis on designs, samples, and compliance across research fields. Assessment. 10.1177/1073191121106735016567 10.1177/10731911211067
