
==== Front
Res Psychother
RIPPPO
Research in Psychotherapy : Psychopathology, Process, and Outcome
2499-7552
2239-8031
PAGEPress Publications, Pavia, Italy

39221903
10.4081/ripppo.2024.803
Review
Does technique matter? A multilevel meta-analysis on the association between psychotherapeutic techniques and outcome
Grevenhaus Christin Janine
Flückiger Christoph
Theimer Lea
Benecke Cord
Contributions: CG, conception, coordination, analysis, interpretation, writing; CF, CB, idea, conception, interpretation, writing, mentoring, LT, conception, analysis, interpretation. Department of Psychology, University of Kassel, Germany
Department of Psychology, Clinical Psychology, University of Kassel, Holländische Straße 36-38, DE-34127 Kassel, Germany. grevenhaus@uni-kassel.de
Conflict of interest: the authors declare no potential conflict of interest.

Ethics approval and consent to participate: not applicable.

Citation: Grevenhaus, C.J., Flückiger, C., Theimer, L. & Benecke, C., (2024). Does technique matter? A multilevel meta-analysis on the association between psychotherapeutic techniques and outcome. Research in Psychotherapy: Psychopathology, Process and Outcome, 27(2), 803.

Publisher’s note: all claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

29 8 2024
29 8 2024
27 2 80329 5 2024
27 8 2024
Copyright © 2024, the Author(s)
2024
Licensee PAGEPress, Italy
https://creativecommons.org/licenses/by-nc/4.0/ This work is licensed under a Creative Commons Attribution NonCommercial 4.0 License (CC BY-NC 4.0).
The extent to which psychotherapeutic techniques have an impact on outcome has been studied on a regular basis. To date, there are no meta-analytic attempts to clarify the association between techniques and outcome in multi-therapeutic approach measures. This study aims to conduct a meta-analysis of the described association. A three-level meta-analysis and moderator-analysis were used. The meta-analysis revealed 13 studies with a total of 177 effect sizes. There was a significant effect r=.193 (t[176]=4.77, p<.01) with higher use of psychotherapeutic techniques being associated with better outcome. Significant moderator was therapeutic approach-specific subscales. The mean effect of cognitive-behavioral techniques was r=.088 (t[147]=1.50, p=.14, d=0.18; s=11, k=79), and the mean effect of psychodynamic techniques was r=.286 (t[147]=5.06, p<.01, d=0.60; s=11, k=70). The measurements for psychotherapeutic technique (Comparative Psychotherapy Process Scale and Psychotherapy-Process Q-Sort) showed no significant difference related to the association between technique and outcome (F[1, 175]=0.38, p=.54). This meta-analysis showed a positive relation between psychotherapeutic techniques and outcome. This leads to the assumption that specific psychotherapeutic techniques have positive effects on post-treatment outcome.

Key words

techniques
psychotherapy outcome
meta-analysis
technique-outcome relation
Funding: none.
==== Body
pmcIntroduction

Psychotherapeutic techniques, which describes the exact procedure in the frame of therapeutic approaches, are central components in psychotherapies (e.g., Bergin & Strupp, 2009). Nonetheless, the potential relevance of psychotherapeutic techniques for the success of treatment is still controversial (e.g., Power et al., 2022; Wampold & Imel, 2015; Webb et al., 2010). The role of psychotherapeutic techniques for treatment progress is a rather complex endeavor. The outcome of a psychotherapy may not exclusively be determined by the therapist’s very specific techniques. Multiple factors on the outcome could be impacted by a particular patient and its collaborative qualities with his therapist (Wampold & Flückiger, 2023; Wampold & Imel, 2015). For example, the interpersonal qualities from therapists, which may root in their personal lives and attachment history, revealed critical factors on the outcome (e.g., Schöttke et al., 2017). Moreover, relationship characteristics such as the alliance may robustly predict treatment outcomes (Flückiger et al., 2018).

Traditionally, particular orientations developed particular measures to investigate the therapist adherence and competence to these particular treatments. For example, the meta-analytic review from Webb et al. (2010) investigated the relation between adherence and competence to outcomes. Adherence describes the extent to which therapists apply the techniques and methods theoretically driven by the treatment model. Meanwhile, competence describes the skill with which therapists uses techniques and methods theoretically driven by the treatment model (Barber et al., 2006). A total of 49 effect sizes within 32 studies were found with no significant relation (Webb et al. 2010), suggesting that adherence and competence from therapists have only little relevance in determining symptom change. One recent systematic review and meta-analysis from Power et al. (2022) investigating the relation between psychological treatment adherence, competence or integrity and clinical outcomes, analyzed 91 effect sizes within 62 studies. Integrity describes a broader concept with therapeutic adherence and competence as components and includes the extent to which a treatment is implemented as intended (Perepletchikova & Kazdin, 2005). In line with Webb et al. (2010) there was no significant relation between adherence and outcomes. In contrast, competence and integrity were significantly associated with clinical outcomes, showing that skilled implementation of therapeutic approach specific strategies is associated with better clinical outcomes. These studies focused on particular therapeutic approach (e.g., cognitive behavior therapy, cognitive therapy, emotion focused therapy, emotion-focused trauma therapy, interpersonal psychotherapy, motivational interviewing/motivational enhancement therapy and psychodynamic, Power et al., 2022; Webb et al., 2010) but did not investigate any techniques across the different orientations.

Instruments for recording psychotherapeutic techniques

In recent years, different instruments have been developed to measure psychotherapeutic techniques across particular therapeutic approaches, i.e., Comparative Psychotherapy Process Scale (CPPS, Hilsenroth et al., 2005); Psychotherapy-Process Q-Sort (PQS, Jones, 1985) and Multitheoretical List of Therapeutic Interventions (MULTI, McCarthy & Barber, 2009). These measures assess a broad range of techniques that can be assessed across particular therapeutic approaches.

First the Comparative Psychotherapy Process Scale (CPPS) consisting of 20 items and focusing on characteristic psychodynamic-interpersonal and cognitive-behavioral techniques. The 20 randomly ordered items are completed by either therapist, patients, or an external observer on a 7-point Likert scale ranging from 0 (not at all characteristics for this session) over 2 (somewhat characteristic), 4 (characteristic) to 6 (extremely characteristic). The instrument shows excellent interrater reliability and internal consistency (Hilsenroth et al., 2005). Reliability and clinical validity have been well established in various contexts and samples (e.g., Gentile et al., 2020). For the CPPS, the prototypes from Hilsenroth et al., (2005) be used as baseline and were compared by the rater to how strongly the therapists adhered to the typical interventions and techniques (e.g., Sell et al., 2017).

Second the Psychotherapy-Process Q-Sort (PQS) with 100 items describing the patient's attitude, behavior, or experience; the actions and attitudes of the therapist; and the interaction of the dyad or the atmosphere of the exchange. Items are classified into one of nine categories from extremely characteristic to extremely uncharacteristic. Prototypes of specific therapeutic approaches are created on basis of these items. The rating profile of a real therapy session is then compared with these prototypes to determine the indices of agreement with the therapeutic approach-specific prototypes. Reliability and validity have been confirmed in several studies across different treatment samples (e.g., Ablon & Jones, 2002; Jones & Pulos, 1993).

Third the Multitheoretical List of Therapeutic Interventions (MULTI) consisting of 60 items about interventions from eight different therapeutic approaches (behavioral, cognitive, dialectical-behavioral, interpersonal, person-centered, psychodynamic, process-experimental, and common factors). Each of the items represents the use of a particular technique and is rated on a 5-point Likert scale from 1 (not at all) to 5 (very much), according to quality and intensity. Three perspectives are considered: a therapist self-report, a patient self-report, and an observer rating scale (McCarthy & Barber, 2009).

Empirical literature on psychotherapeutic techniques and outcomes

When investigating techniques across different therapeutic approaches, Jones and Pulos (1993) for example found that typically psychodynamic techniques were positively related to treatment outcome, whereas Ablon and Jones (2002) found a positive correlation of the use of typically cognitive-behavioral techniques with outcome. More recently, the single-case-study from Roussos et al. (2018) found a positive association between cognitive-behavioral techniques and the reduction of a broad range of symptoms, whereas the single-case study from Laskoski et al. (2021) illustrated that the use of psychodynamic techniques was the best predictor for the therapeutic process. Most studies especially concentrate on specific patient groups (e.g., trauma survivors, Taylor, 2010) or specific diagnoses (e.g., borderline personality disorder and substance use disorder, Philips et al., 2018). Taken together, already thirty years ago studies investigated the effectiveness of techniques on outcomes, but results are mixed. Cross-therapeutic approach instruments have not yet been systematically studied enough to answer the question about the relevance of psychotherapeutic techniques for the success of treatments.

To summarize the literature, there are two main arguments of how to contextualize technique-outcome relations (TOR). First: The value of a particular technique. A particular technique like Socratic questions or validation showed an impact on outcomes at the end of a therapy across all orientations (Hill & Norcross, 2023). Second: The use of particular techniques associated with outcomes is or is not quite independent of the underlying orientation. On one hand there are studies (e.g., Power et al., 2022) which highlight the differences in techniques according to outcomes, whereas on the other hand some studies (e.g., Hill & Norcross, 2023) highlight the equivalent potential of all orientations; but what these studies have in common is that they did not use multitherapeutic approach measures to investigate techniques. There is a lack of meta-analytic attempt to investigate multi-approach measures that assess techniques from different therapeutic approaches (e.g., Laskoski et al., 2021; Mullin et al., 2018). Some studies showed the same effectiveness for both, psychodynamic and cognitive-behavioral techniques (e.g., Glock et al., 2018), and some studies showed significant differences between them (e.g., Goodman et al., 2015; Solomonov et al., 2020). Taken together it is not clear whether the therapeutic approach-specific subscales differ or not in their relation to treatment outcome.

Present study

Like described above, the previous studies focused on the relationship of psychotherapeutic technique to outcome within a single therapeutic approach. We did not find any previous metaanalytic attempt to synthesize the effect sizes of the relation between psychotherapeutic techniques and outcomes within the multi-therapeutic approach measures CPPS, PQS or MULTI, since this meta-analytic synthesis is the first one focusing on multi-therapeutic approach measures.

Regarding the definition of outcomes and under consideration of the complexity of functioning, the authors decided to define outcomes with a wider range. An outcome can be any result of psychotherapy as reported in the respective studies, like the improvement of specific symptoms (e.g., anxiety disorders) as well as non-specific symptoms (e.g., reduction of guilt feelings) or that interpersonal and social areas have improved (e.g., psychosocial functioning). The major aim of our investigation was to examine this possible relation by using meta-analytic methods. In contrast to Power et al. (2022) we focused on multi-therapeutic approach measures for psychotherapeutic techniques. We are interested in the question of how the use of psychotherapeutic techniques and outcomes are related. Additionally, we are interested in whether the TOR differs in relation to therapeutic approach-specific subscales and whether the TOR differs in relation to the measures (CPPS, MULTI and PQS).

Methods

Meta-analytic search strategy, study selection and data collection

To identify studies that report the association of psychotherapeutic techniques assessed by CPPS, PQS or MULTI, and post-treatment outcomes we conducted a systematic search via EBSCO on 15th April 2023. The databases APA PsycArticles, APA PsycInfo, PSYNDEX Literature with PSYNDEX Tests, and Medline were used. The search was limited to papers published in English and German. We used the following search terms: (“comparative psychotherapy process scale” OR “vergleichende Psychotherapie Prozess Skalen” OR “CPPS” OR “psychotherapy process q-set” OR “Psychotherapie Prozess Q-sort” OR “PQS” OR “the multitheoretical list of therapeutic interventions” OR “multitheoretische Liste der therapeutischen Interventionen”) AND (“psychotherapy” OR “Psychotherapie”). "MULTI" is an unspecific term and therefore was not included as a search term. In total, the search yielded 741 results, of which 398 were for the PQS instrument, 292 for CPPS, and 51 for MULTI. 350 duplicates were excluded.

We used the following inclusion criteria: (a) instruments to measure CPPS, PQS or MULTI, (b) focus of an adult population, (c) measuring post-treatment outcomes, and (d) relationship between a and c. Exclusion criteria were: (a) not all scales of CPPS, PQS or MULTI are measured, (b) children or adolescents, (c) outcomes missing, (d), no correlational association between technique and outcome, (e) no psychotherapy, (f) subclinical population, (g) session outcomes, (h) no full-text available, (i) same sample, (j) only significant items reported, (k) sample size < 5, (l) no association between technique and outcome reported and (m) post-treatment outcomes with self-developed, not validated measures by researchers. The study selection followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA, Moher et al., 2009) guideline. The PRISMA flow chart is displayed in Figure 1 and provides the included and excluded studies. From the total of 741 studies, 118 studies fulfilled the inclusion criteria after abstract screening and were integrated for full-text screening. From these studies, 45 studies were checked for inclusion criteria, whereas from these 45 studies, 32 studies had to be excluded based on detailed full-text reviewing. Overall, a total of 13 studies was included, whereas 8 studies for PQS and 5 studies for CPPS.

MULTI was eliminated as instrument, because every available study meets at least one exclusion criteria.

Screenings of abstracts and full text were conducted by CG and LT split in half, whereas an assessment of a subsample of 100 studies indicated considerable consensus for the abstract screening (percent agreement >94%; Cohen’s Kappa >.85; Landis & Koch, 1977). Full-text screening was coded by both raters and again indicated considerable consensus (Cohen’s Kappa =.87). Six cases with disagreement were discussed and decided based on consensus of all authors.

Statistical analyses

We used a multilevel meta-analytic model with a three-level structure with restricted maximum likelihood estimates (Assink & Wibbelink, 2016). The analyses were conducted via the R-package metafor (Viechtbauer, 2010). In the three-level model, the sampling variance (Level 1) was nested within effect sizes (k=177; Level 2), which were nested within studies (s=13; Level 3; van den Noortgate et al., 2013; Viechtbauer, 2010). Overall heterogeneity was assessed with the Q-statistics. Additionally, we tested whether within-study and between-study heterogeneity had a significant proportion of the overall heterogeneity. Following Assink and Wibbelink (2016), we compared the full model with a reduced model, where the variance (within- or between study) was fixed as zero. For the analysis, correlation coefficients were transformed into Fisher’s z. To enhance interpretability the z was transformed back into correlation coefficients. Additionally, Cohen’s d and pseudo R² were reported. To identify publication bias, a funnel plot with trim and fill method was created and the Begg and Mazumdar's rank correlations test was examined.

The following moderators for the technique-outcome relation were determined. Therapeutic approach-specific subscales. Cognitive Behavioral Therapy (CBT) and Psychodynamic Therapy (PDT). Instruments. (0) CPPS and (1) PQS.1

Results

Descriptive sample characteristics

For the meta-analysis we considered 13 studies (N=1073 patients), including a total of 177 effect sizes. Eleven (84.62%) English-language studies and two (15.38%) German-language studies were included. Ten (76.92%) studies were published as journal articles and three (23.08%) were dissertations. There is considerable variation in the investigated treatments and treatment outcomes (up to eight different instruments to measure treatment outcome, e.g., Jones and Pulos 1993). Additionally, there is a broad range of diagnoses, ranging from studies which focused only on one diagnosis (e.g., panic disorder, Ablon et al., 2006) and studies with up to seven diagnoses (somatic symptom disorder, mood disorder, anxiety disorder, eating disorder, probable alcohol abuse, cluster A, B and C personality disorder, chronic somatic illness, Sell et al., 2018). The majority of ratings for the psychotherapeutic techniques using the CPPS and PQS were conducted by therapists and master/doctoral students (n=3), followed by external independent raters and patients (n=2). In one study each, the authors themselves or employees from the department conducted the ratings. Descriptive data are documented in Tables 1 and 2.

Association between technique and outcome

Overall effect

For CPPS and PQS, there were a total of k=177 effect sizes within s=13 studies. Overall association between use of psychotherapeutic technique and outcome was r=.193 (95% CI [.12; .27], pseudo R2=3.7, d=0.39). This effect was significantly different from zero (t[176]=4.77, p<.01).

Heterogeneity

There was considerable heterogeneity (Q[176]=3319.85, p<.01) whereas 75.22% of the heterogeneity could be attributed to within-study variability (at Level 2, F[2, 3]=1587.14, p<.01) and 20.34% between-study variability (at Level 3, F[2, 3]=25.09, p<.01).

Publication bias

The visual funnel plot showed no indication of publication bias (Figure 2), whereas the Begg and Mazumdar's rank correlations test was significant (p<.01), which indicates a publication bias.

Moderator analyses

To explore the high level of heterogeneity, we investigated two moderators namely the instruments which measure technique (CPPS and PQS), and the therapeutic approach-specific subscales of CPPS and PQS.

Figure 1. Flow diagram of the studies included in the meta-analysis based on preferred reporting items for systematic reviews and meta-analyses criteria (Page et al., 2021).

Figure 2. Funnel plot of all included effect sizes (k=177). Observed outcomes were reported in fisher’s z transformed correlation coefficients.

Table 1. Descriptive sample data about patients and therapists.

Authors	N. patients	Mean age patients	Female, %	Patients’ population	Main diagnoses N. t	herapists	
Ablon & Jones (1998)	100	51	67	PD-office, PD- treatments for PTSD, CBT	PTSD, mood, anxiety, obsessive-compulsive, conversion, dissociative psychogenic pain disorder, hypochondriasis, psychosexual dysfunctions	15	
Ablon & Jones (2002)	239	35	70	IPT, CBT	Major depressive disorder	18	
Ablon et al. (2006)	17	35	88.2 O	utpatient psychiatry servic	e Panic disorder	7	
DeFife et al. (2008)	55	29.55	73	PD training clinic	Adjustment disorder, anxiety disorder,
eating disorder, mood disorder,
substance-related disorder, V code
relational problems	19	
Goodman et al. (2015)	5	32.4	100	Psychiatric inpatient PDT	Borderline personality disorder	4	
Haugen et al. (2016)	29	48.55	17	Exposure of 9/11	Partial or full PTSD	6	
Hofmann (2008)	42	44.4	79	PDT, CBT	Generalized anxiety disorder	9	
Jones & Pulos (1993)	62	41.64	73	PD-office, CBT	Neurotic disorder, major depressive disorder	19	
Mullin et al. (2018)	75	29.8	73	PD training clinic	Adjustment disorder, anxiety disorder,
eating disorder, mood disorder,
substance-related disorder, V code
relational problems	26	
Owen & Hilsenroth (2014)	70	29.8	74	PD training clinic	Mood disorders	28	
Schweyer (2008)	52	43.58	48	CBT, EST	Bipolar-I, personality disorders	n.a.	
Sell et al. (2018)	300	37.24	70	Integrative PDT	Somatic symptom disorder, mood disorder,
anxiety disorder, eating disorder, probable alcohol abuse, cluster A, B and C personality disorder, chronic somatic illness	72	
Taylor (2010)	27	38.08	100	Women withviolence	Dissociative Identity Disorder; PTSD	22	
PD, psychodynamic; PTSD, posttraumatic stress disorder; CBT, cognitive behavioral therapy; IPT, interpersonal therapy; EST, emotional supportive therapy; n.a., not available; PDT, psychodynamic therapy.

Therapeutic approach-specific subscales

In this moderator analysis, we analyzed CPPS and PQS together, for those subscales that were available for both instruments, CBT- and PDT-techniques (Table 3). The subscale CBT-techniques showed a TOR of r=.088 (95% CI [-.03; .20], t[147]=1.50, p=.14, pseudo R2=0.8%, d=0.18), and the subscale PDT-techniques showed a TOR of r=.286 (95% CI [.18; .38], t[147]=5.06, p<.01, pseudo R2=8.2%, d=0.60) The difference between these two correlational effect sizes was significant (F[1, 147]=31.69, p<.01). This suggests a difference between CBT- and PDT-techniques in the relation of outcome. For CPPS the subscale CBT-technique indicated a TOR of r=.054 (95% CI [-.15; .26], d=0.11) and the subscale PDT-technique showed a TOR of r=.337 (95% CI [.15; .49], pseudo R2=11.4%, d=0.72). The difference between these two correlational effect sizes was significant (F[1, 24]=44.42, p<.01). For PQS there were more therapeutic approach-specific subscales: CBT, Dialectical Behavior Therapy (DBT), Interpersonal Therapy (IPT), PDT and Transference-Focused Therapy (TFP). The subscale CBT-techniques showed a TOR of r=.095 (95% CI [-.03; .21], d=0.19); the subscale DBT-techniques showed a TOR or r=.061 (95% CI [-.63; .68], d=0.12), the subscale IPT-technique showed a TOR of r=.185 (95% CI [.04; .32], pseudo R2=3.4%, d=0.38); the subscale PDT-technique a TOR of r=.262 (95% CI [.15; .37], pseudo R2=6.9%, d=0.54) and the subscale TFP-techniques showed a TOR of r=.280 (95% CI [-.50; .76], d=0.58), whereas the difference between effect sizes was significant (F[4, 146]=4.08, p<.01). This suggests a difference between CBT-, DBT, IPD, PDT and TFP-techniques in the relation of outcome.

Table 2. Descriptive data about treatment, technique and outcome.

Authors	SUD	Treatment type	Treatment frequency	M Sessions	Design	Technique instrument	Rater	Time point measuremen	Outcome t instruments	
Ablon & Jones (1998)	Excl.	CBT; PDT	n.a.	15	Natura.	PQS	Advanced doctoral students	Selected sessions	ATQ, BPRS, BDI, HRSD-17, MMPI-D, RDS, SDC-R, TCS	
Ablon & Jones (2002)	N. excl.	CBT; IPT	n.a.	16.2	RCT	PQS	Expert therapists	4th and 12th session	BDI, DAS, GAS, HRSD, HSCL-90, MSAS	
Ablon et al. (2006)	Excl.	PDT	1	21	Natura.	PQS	Master’s gradu students	ate 12th session	ASI, PDSS, SCL-90-R	
DeFife et al. (2008)	N. excl.	PDT	1-2	26.55	Natura.	CPPS	Undergraduat participants + patients	e 3rd session, when 90% is over	BSI, SAS, SOS-10	
Goodman et al. (2015)	N. excl.	PDT	3	n.a.	Natura.	PQS	n.a.	n.a.	SCL-90-R	
Haugen et al. (2016)	Excl.	Integrative	1	18.56	Natura.	CPPS	Patients	Every session	OQ-45.2	
Hofmann (2008)	Excl.	CBT; PDT	n.a.	n.a.	Natura.	PQS	Employees fro the departmen	m 10th and 18th t session	BAI, BDI, HAMA, HAMD, IIP, PSWQ, STAI	
Jones & Pulos (1993)	Excl.	CBT; PDT	1-2	15.1	Natura.	PQS	Clinicians an graduate students	d 1st, 5th and 14th session	ATQ, BDI, BPRS, HRSD-17, MMPI-D, OCR, RDS, SCL-90-R	
Mullin et al. (2018)	N. excl.	STPP	1-2	29	Natura.	CPPS	Trained external rater	3rd and 9th s session	SCORS-G	
Owen & Hilsenroth (2014)	N. excl.	PDT	1-2	27.5	Natura.	CPPS	Independent raters	3rd, 9th, when 90% is ov	BSI, PEI er	
Schweyer (2008)	Excl.	CBT; EST	n.a.	19	RCT	PQS	Author	2nd, 8th, 14th and 17th sessio	ADS-L, ADMS, n YMRS, GAF, F-SOZU	
Sell et al. (2018)	N. excl.	GIP; HY	n.a.	n.a.	Natura.	CPPS	Therapist	6, 12, 18, 24, 30 months	CGI-S. GAF	
Taylor (2010)	N. excl.	CBT; ET, PDT; IPT	1	12	Natura.	PQS	Therapist	Before 12th session	BSI, DES, PCL	
SUD, substance use disorder; CBT, Cognitive Behavioral Therapy; PDT, Psychodynamic Therapy; ADMS, Erweiterung zur Erfassung manischer Symptome; ADS-L, Allgemeine Depressionsskala; ASI, Anxiety Sensitivity Index; ATQ, Automatic Thoughts Questionnaire; BAI, Becks Angst Inventar; BDI, Becks Depression Inventory; BSI, Brief Symptom Inventory; BPRS, Brief Psychiatric Rating Scale; CGI-S, Clinical Global Impression-Severity; DAS, Dysfunctional Attitudes Scale; DBT, Dialectical Behavior Therapy; DES, Dissociative Experiences Scale; EST, Emotional Supportive Therapy; ET, Electical Therapy; Excl., Excluded; F-SOZU, Fragebogen zur Sozialen Unterstützung; GAS, Global Assessment Scale; GIP, Guided Imagery Psychotherapy; HAMA, Hamilton Angst Skala; HAMD, Hamilton Depression Skala; HRSD, Hamilton Rating Scale for Depression; HSRD-17, Hamilton Rating Scale for Depression; HSCL-90, Hopkins Symptom Checklist; H Y, Hypnopsychotherapy; IIP, Inventar Interpersonaler Probleme; IPT, Interpersonal Therapy; MSAS, Modified Social Adjustment Scale; MMPI-D, Minnesota Multiphasic Personality Inventory – Depression scale; N. Excl., Not excluded; n.a., not available; Natura., naturalistic; OCR, Overall Change Rating; OQ-45.2, Outcome Questionnaire-45.2; PCL, Posttraumatic Stress Disorder Checklist; PDSS, Panic Disorder Severity Scale; PEI, Patient Estimate of Improvement; PSWQ, Penn State Worry Questionnair; RCT, randomized controlled trial; RDS, Raskin Depression Scale; SAS, Social Adjustment Scale; SCL-90-R, Symptom Checklist-90-Revised; SCORS-G, Social Cognition and Object Relations Scale – Global; SDC-R, Symptom Distress Checklist-Revised; SOS-10, Schwarz Outcome Scale; STAI, State Trait Angst Inventory; STPP, Short Term Psychodynamic Psychotherapy; TCS, Target Complain Scale; TFP, Tranfserence-Focused Therapy; YMRS, Young Mania Rating Scale.

Table 3. Summary of the moderator therapeutic approach-specific subscale.

Moderators	s	k	r	95% CI	p	
CBT vs. PDT	13	149				
CBT	11	79	.088	-.03; .20	ns	
PDT	11	70	.286	.18; .38	<.01	
s, number of studies; k, number of effect sizes, r, Pearson’s correlation coefficient; CI, confidence interval; CBT, cognitive behavioral therapy, PDT, psychodynamic therapy; ns, not significant.

Instruments

At least we checked whether there is a difference in the association between psychotherapeutic technique and outcome related to the technique instrument (CPPS or PQS). This moderatoranalysis showed no significant difference related to the association between technique and outcome (F[1, 175]=0.38, p=.54).

Discussion

The potential outcome-relevance of particular psychotherapeutic methods, techniques and skills has a lasting tradition in psychotherapy research (Hill & Norcross, 2023). This study aimed to estimate the relation between psychotherapeutic techniques and post-treatment outcomes by analyzing three-level meta-analytic models. We found a small positive relation between techniques and outcome (r=.193 across 177 reported effect sizes within 13 studies); that is the higher use of psychotherapeutic techniques being associated with better outcomes. Results showed significant heterogeneity across the three levels, with mostly within-study variability across particular techniques and outcomes (75.22%). Funnel plots showed no indication of publication bias, but the Begg and Mazumdar's rank correlations test does, so that the results must be interpreted with caution due to the possible publication bias.

This meta-analysis extends previous efforts in the process-outcome research by analyzing assessments that focus on multidimensional techniques independent of therapeutic approach. Previous meta-analytic attempts, for example Power et al. (2022), analyzed within-therapeutic approach associations and found small significant positive associations between competence and outcome (non-hierarchical studies: r=.17), and small to moderate significant associations between integrity and outcome (non-hierarchical studies: r=.15; hierarchical studies2: r=.23). Compared to our study (r=.193) the effect sizes are in a similar range. These findings contrast with the meta-analysis from Webb et al. (2010), where no significant association between adherence-outcome as well as competence-outcome could be observed.

Psychotherapeutic techniques may play a central role in treatment. Nevertheless, the relation between technique and outcome across therapeutic approaches and the possible impact of a particular technique on the outcome independent of therapeutic approach has been little studied. In our meta-analysis, taken all psychotherapeutic techniques together it can be stated that the relation between psychotherapeutic techniques and outcomes are observable across therapeutic approaches, i.e., even within each other's therapeutic approach-specific treatment, the techniques showed significant relations to outcome. Four studies (Ablon & Jones, 1998; DeFife et al., 2008; Goodman et al., 2015; Jones & Pulos, 1993) within this meta-analysis could find a positive TOR for CBT-techniques within PDT-treatments; additionally in two studies (Ablon & Jones, 1998; Jones & Pulos, 1993) there was a positive TOR for PDT-techniques within CBT-treatments. This suggests that transtheoretical therapeutic approaches might have benefits compared to rigid boundaries of traditional manualized treatments. This assumption is in line with studies from Hilsenroth and Slavin-Mulford (2008), Hilsenroth et al. (2007) and Richards et al. (2016). These studies showed that the use of transtheoretical therapeutic approaches was beneficial for patients with depression, borderline, or bulimia nervosa. A major value of transtheoretical psychotherapy is the flexibility to adapt the technique to the potential needs and preferences of the patient (Norcross & Goldfried, 2005).

Our results showed a positive relationship between technique and outcome. The use of particular techniques could be a therapist factor, where orientation-specific competence is corresponded with the therapists’ approach. To examine this, we used a moderator-analysis for the moderator therapeutic approach-specific subscale, consisting of different therapeutic approach-specific techniques. Results showed significant differences between these techniques. For CPPS and PQS together the CBT-techniques showed a non-significant mean effect (r=.088, p=.14), whereas the PDT-techniques showed a significant positive mean effect (r=.286, p<.01). This suggest that the higher use of PDT-techniques being associated with better outcomes, whereas this association could not be found for CBT-techniques. Results are not in line with Power et al. (2022), where CBT showed the strongest associations. The differences in these findings are not trivial as Power et al. (2022; k = 91) mostly investigated measures within one therapeutic approach, whereas the present meta-analysis was focused on measures that assess techniques across particular therapeutic approaches. Moreover, specific techniques within a therapeutic approach such as homework assignment may be particularly effective (Ryum et al., 2023), whereas for other techniques the evidence base might be less consistent (e.g., Metaphors; McMullen & Tay, 2023). Additionally, our study has quite more naturalistic studies than randomized-controlled-trials (RCT). According to this it can be assumed that our results are not in line with Power et al. (2022) because of differences in the design (naturalistic or RCT) and for a more nuanced understanding of specific techniques, future research should use multidimensional assessments within naturalistic designs. For this more studies with MULTI should be aimed, because of a more detailed analysis of techniques from eight distinct therapeutic approaches, along with transtheoretical factors. Overall, for future research there is need for future studies investigating association between technique and outcome by the MULTI. Furthermore, more detailed analysis of techniques from eight distinct therapeutic approaches, along with transtheoretical (common) factors and patient factors are needed (e.g., de Felice et al., 2019; McAleavy & Castonguay, 2015).

Limitations and future directions

The present study has several limitations that may be important to consider in future research. First, the number of studies with 13 studies that analyzed multidimensional scales across therapeutic approaches is limited, indicating the potential relevance for future research of these assessments and its direct comparisons with mono-theoretical assessments. Second, primary studies investigated a broad range of outcome instruments. Whereas this diversity of investigated outcome instruments may represent research interests to investigate a broad range of facets across particular outcomes, it also limits our knowledge for particular outcomes and outcome measures. Third, most of the primary studies included a broad range of diagnosis, which also limits our knowledge in relation to particular diagnosis. For example, when individuals that suffer from a personality disorder are included, disruptions in the treatment adherence are to expect and therefore may impact psychotherapy outcomes (Clarkin & Levy, 2004). The broad range of outcome instruments and diagnoses acts also as a strength of this meta-analysis, since there could be observed an effect even within such diversity. Fourth, most studies with CPPS involved data collection in the context of a psychodynamic training clinic. It would be needed to collect data likewise in other contexts. Fifth, the instruments measuring psychotherapeutic techniques focused the presence of a certain technique. But besides what technique a therapist use it is also important how competent a therapist uses it (Sharpless & Barber, 2009). Future research is needed better understand the potential interplay between adherence and competence issues across particular techniques. Sixth, the primary studies in this meta-analysis focused on between-therapists correlations. Besides the between-therapists correlations, the within-therapists correlations are considered as a critical factor to better understand therapist specific effects (e.g., Baldwin & Imel, 2013). Additionally, we just consider the therapists as a factor for the use of technique. But the use of technique can although be a patient factor, which technique is used may depend on the individual patient and their needs. To disentangle therapist and patient contributions, future primary studies would need report more the variability of effects at patient and therapist levels (Baldwin et al., 2007; Del Re et al., 2021; Wampold & Owen, 2021). Last, allegiance effects may or may not impact the TOR as e.g., indicated in the alliance-outcome association (e.g., Flückiger et al. 2018). However, it might be a particular strength of the investigated multidimensional assessments to investigate potential allegiances effects across therapeutic approaches simultaneously.

Conclusions

Besides these potential limitations, our study presented the first meta-analytic attempt synthesizing instruments that assess therapeutic techniques across different therapeutic approaches. The results suggest a small but meaningful relation between psychotherapeutic techniques and outcomes, where a higher use of techniques is associated with better outcomes at the end of therapy. Overall, the present meta-analysis supports the underlying research attempt that developing and investigating measures that assesses a diversity of techniques may indicate a promising research strategy to enhance our knowledge across particular therapeutic approaches and provide the potential to advance our transtheoretical knowledge. Future research is needed to have a closer look at the relation between technique and outcome within naturalistic studies using multi-therapeutic approach measures.

Availability of data and materials

Since the current study is a systematic review, no original data from any study is included.

1 Only CPPS and PQS are left since studies with MULTI had to be eliminated.

2 Hierarchical studies are controlled for between-therapist variability in treatment outcomes, i.e., patients nested within therapist (Power et al., 2022).
==== Refs
References

Ablon J. S. Jones E. E. (1998). How expert clinicians´ prototypes of an ideal treatment correlate with outcome in psychodynamic and cognitive-behavioral therapy. Psychotherapy Research, 8 (1 ), 71-83. doi: 10.1093/ptr/8.1.71
Ablon J. S. Jones E. E. (2002). Validity of controlled clinical trials of psychotherapy: findings from the NIMH treatment of depression collaborative research program. American Journal of Psychiatry, 159 , 775-783. doi: 10.1176/appi.ajp.159.5.77511986131
Ablon J. S. Levy R. A. Katzenstein T. (2006). Beyond brand names of psychotherapy: Identifying empirically supported change processes. Psychotherapy: Theory, Research, Practice, Training, 43 (2 ), 216-231. doi: 10.1037/0033-3204.43 .2.216
Assink M. Wibbelink C. J. M. (2016). Fitting three-level meta-analytic models in R: a step-by-step tutorial. The Quantitative Methods for Psychology, 12 (3 ), 154-174. doi: 10.20982/tqmp.12.3.p154
Baldwin S. A. Imel Z. E. (2013). Therapist Effects: Findings and Methods. In Lambert M. J. (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (4th ed., pp. 258-297). New York, NY: John Wiley & Sons.
Baldwin S. A. Wampold B. E. Imel Z. E. (2007). Untangling the alliance-outcome correlation: exploring the relative importance of therapist and patient variability in the alliance. Journal of Consulting and Clinical Psychology, 75 (6 ), 842-852. doi: 10.1037/0022-006X.75.6.84218085902
Bergin A. E. Strupp H. H. (2009). Changing Frontiers in the Science of Psychotherapy. Chicago, IL: Aldine-Atherton.
Clarkin J. F. Levy K . N. (2004). The influence of client variables on psychotherapy. In Lambert M. J. (Ed.). Bergin and Garfield’s handbook of psychotherapy and behavior change (4th ed.) (pp. 194-226). New York, NY: John Wiley & Sons.
Cohen J . (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.
de Felice G. Giuliani A. Halfon S. Andreassi S. Paoloni G. Orsucci F. F. (2019). The misleading Dodo Bird verdict. How much of the outcome variance is explained by common and specific factors? New Ideas in Psychology, 50-55. doi: 10.1016/j.newideapsych.2019.01.006
DeFife J. A. Hilsenroth M. J. Gold J. R. (2008). Patient ratings of psychodynamic psychotherapy session activities and their relation to outcome. Journal of Nervous and Mental Disease, 196 (7 ), 538-547. doi: 10.1097/NMD.0b013e31817 cf6d018626294
Del ReA. C. Flückiger C. Horvath A. O. Wampold B. E. (2021). Examining therapist effects in the alliance-outcome relationship: a multilevel meta-analysis. Journal of Consulting and Clinical Psychology, 89 (5 ), 371-378. doi: 10.1037/ccp 000063733829817
Flückiger C. Del ReA. C. Wampold B. E. Horvath A. O. (2018). The alliance in adult psychotherapy: a meta-analytic synthesis. Psychotherapy, 55 (4 ), 316–340. doi: 10.1037/ pst000017229792475
Gentile D. Colli A. Lingiardi V. (2020). An Italian validation of the comparative psychotherapy process scale: reliability and validity. Clinical psychology & psychotherapy, 27 (5 ), 779-789. doi: 10.1002/cpp.246232307829
Glock G. Hilsenroth M. Curtis R. (2018). Therapeutic interventions patients with anxiety problems find most helpful: A case for integration? Journal of Psychotherapy Integration, 28 (2 ), 233-241. doi: 10.1037/int0000075
Goodman G. Edwards K. Chung H. (2015). The relation between prototypical processes and psychological distress in psychodynamic therapy of five inpatients with borderline personality disorder. Clinical psychology & psychotherapy, 22 (1 ), 83-95. doi: 10.1002/cpp.187525625597
Haugen P. T. Werth A. S. Foster A. L. Owen J. (2016). The role of theory-specific techniques and therapeutic alliance in promoting positive outcomes: integrative psychotherapy for world trade center responders. The Journal of Nervous and Mental Disease, 204 (12 ). 955-959. doi: 10.1097/NMD. 0000000000000631
Hill C. E. Norcross J. C. (2023). Psychotherapy skills and methods that work. New York, NY: Oxford University Press
Hilsenroth M. Blagys M. D. Ackerman S. J. Bonge D. R. Blais M. A. (2005). Measuring psychodynamic-interpersonal and cognitive-behavioral techniques: development of the Comparative Psychotherapy Process Scale (CPPS). Psychotherapy, 42 , 340-356. 10.1037/0033-3204.42.3.340
Hilsenroth M. J. DeFife J. A. Blake M. M. Cromer T. D. (2007). The effects of borderline pathology on short-term psychodynamic therapy for depression. Psychotherapy Research, 17 , 172-184. 10.1080/10503300600786748
Hilsenroth M. Slavin-Mulford J. (2008). Integrative dynamic treatment for co-morbid depression and borderline conditions. Journal of Psychotherapy Integration, 18 , 377-409.
Hofmann N . (2008). Kognitiv-behaviorale und tiefenpsychologisch fundierte Therapie derGeneralisierten Angst: Ein Therapieprozessvergleich [Dissertation; d-nb.info]. https://dnb.info/988979411/34
Jones E. E. (1985). Psychotherapy Process Q-Set Coding Manual. Unpublished manuscript, University of California, Berkeley.
Jones E. E. Pulos S. M. (1993). Comparing the process in psychodynamic and cognitive-behavioral therapies. Journal of Consulting and Clinical Psychology, 61 (2 ), 306-16. doi: 10.1037//0022-006x.61.2.306.8473585
Landis J. Koch G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33 (1 ), 159-174. doi: 10.2307/2529310843571
Laskoski P. B. Hauck S. Orsucci F. Giuliani A. Serralta F. B. de Felice G. (2021). Psychodynamic factor as predictor of outcome in the treatment of a psychosomatic spectrum patient. Journal of Contemporary Psychotherapy, 51 (2 ), 145-153. doi: 10.1007/s10879-020-09486-0.
McAleavey A.A. Castonguay L.G. (2015). The process of change in psychotherapy: common and unique factors. In: Gelo O. Pritz A. Rieken B. (Eds.) Psychotherapy Research (pp. 293-310). Vienna: Springer. doi: 10.1007/978-3-7091-1382-0_15
McCarthy K. S. Barber J. P. (2009). The Multitheoretical List of Therapeutic Interventions (MULTI): initial report. Psychotherapy Research, 19 (1 ). 96-113. doi: 10.1080/105033 00802524343.19065285
McMullen L. M. Tay D. (2023). Metaphors In: Psychotherapy Skills and Methods That Work. In Hill C. E. Nor-cross J. C. (Eds.), Psychotherapy Skills and Methods That Work (pp. 138-164), Oxford: Oxford University Press. doi: 10.1093/oso/9780197611012.003.0006
Moher D. Liberati A. Tetzlaff J. Altman D. G. PRISMA Group (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS medicine, 6 (7 ), e1000097. doi: 10.1371/journal.pmed.100009719621072
Mullin A. S. J. Hilsenroth M. J. Gold J. Farber B. A. (2018). Facets of object representation: process and outcome over the course of psychodynamic psychotherapy. Journal of personality assessment, 100 (2 ), 145-155. doi: 10.1080/ 00223891.2016.121532027606942
Norcross J. C. Goldfried M. R. (2005). Handbook of Psychotherapy Integration (2nd Ed.). Oxford: Oxford University Press
Owen J. Hilsenroth M. J. (2014). Treatment adherence: the importance of therapist flexibility in relation to therapy outcomes. Journal of Counseling Psychology, 61 (2 ), 280-288. doi: 10.1037/a003575324635591
Page M. J. McKenzie J. E. Bossuyt P. M. Boutron I. Hoffmann T. C. Mulrow C. D. Shamseer L. Tetzlaff J. M. Akl E. A. Brennan S. E. Chou R. Glanville J. Grimshaw J. M. Hróbjartsson A. Lalu M. M. Li T. Loder E. W. Mayo-Wilson E. McDonald S. Moher D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. The BMJ, 372 , Article n71. doi: 10.1136/ bmj.n71
Perepletchikova F. Kazdin A. E. (2005). Treatment integrity and therapeutic change: issues and research recommendations. Clinical Psychology: Science and Practice, 12 (4 ), 365–383. doi: 10.1093/clipsy.bpi045
Philips B. Karlsson R. Nygren R. Rother-Schirren A. Werbart A. (2018). Early therapeutic process related to dropout in mentalization-based treatment with dual diagnosis patients. Psychoanalytic Psychology, 35 (2 ), 205-216. doi: 10.1037/pap 0000170
Power N. Noble L. Simmonds-Buckley M. Kellett S. Stockton C. Firth N. Delgadillo J. (2022). Associations between treatment adherence-competence-integrity (ACI) and adult psychotherapy outcomes: a systematic review and metaanalysis. Journal of Consulting and Clinical Psychology, 90 (5 ). 427-445. ISSN 0022-006X. doi: 10.1037/ccp000073635604747
Richards L.K. Shingleton R. Goldman R.E. Siegel D. Thompson-Brenner H. (2016). Integrative dynamic therapy for bulimia nervosa: an evidence-based case study. Psychotherapy, 53 (2 ), 195-205. doi: 10.1037/pst000005427267504
Roussos A. J. Gomez PenedoJ. M. Muiños R. (2018). A time-series analysis of therapeutic alliance, interventions, and client’s clinical status in an evidence-based single-case study: Evidence for establishing change mechanisms in psychotherapy. Psychotherapy Research, 28 (1 ), 137-149. doi: 10.1080/10503307.2016.117434627168133
Ryum T. Bennion M. Kazantzis N. (2023). Integrating between-session homework in psychotherapy: a systematic review of immediate in-session and intermediate outcomes. Psychotherapy, 60 (3 ), 306-319. doi: 10.1037/pst000048837104804
Schöttke H. Flückiger C. Goldberg S. B. Eversmann J. Lange J. (2017). Predicting psychotherapy outcome based on therapist interpersonal skills: a five-year longitudinal study of a therapist assessment protocol. Psychotherapy research: journal of the Society for Psychotherapy Research, 27 (6 ), 642-652. doi: 10.1080/10503307.2015.112554628277042
Schweyer D . (2008). Prozessforschung in der Psychotherapie bei manisch-depressiven Patienten [Dissertation, Eberhard-Karls-Universität Tübingen].
Sell C. Möller H. Taubner S. (2018). Effectiveness of integrative imagery- and trance-based psychodynamic therapies: guided imagery psychotherapy and hypnopsychotherapy. Journal of Psychotherapy Integration, 28 (1 ), 90-113. doi: 10.1037/int0000073
Sharpless B. A. Barber J. P. (2009). A conceptual and empirical review of the meaning, measurement, development, and teaching of intervention competence in clinical psychology. Clinical psychology review, 29 (1 ), 47-56. doi: 10.1016/j.cpr.2008.09.00818952334
Solomonov N. Falkenström F. Gorman B. S. McCarthy K. S. Milrod B. Rudden M. G. Chambless D. L. Barber J. P. (2020). Differential effects of alliance and techniques on Panic-Specific Reflective Function and misinterpretation of bodily sensations in two treatments for panic. Psychotherapy Research, 30 (1 ), 97-111. doi: 10.1080/10503307.2019. 158559130821630
Taylor W.-M. (2010). Traveling the Road to Healing: An Examination of the Relationships between Psychotherapeutic Technique and Cognitive, Emotional and Physiological Outcomes in Women Exposed to Violence (Dissertation, University of Michigan). Library University of Michigan. https://deepblue. lib.umich.edu/handle/2027.42/75898.
Van den Noortgate W. López-López J. A. Marín-Martínez F. Sánchez-Meca J. (2013). Three-level meta-analysis of dependent effect sizes. Behavior Research Methods, 45 (2 ), 576-594. doi: 10.3758/s13428-012-0261-623055166
Viechtbauer W . (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36 (3 ), 1-48. doi: 10.18637/jss.v036.i03
Wampold B. E. Flückiger C. (2023). The alliance in mental health care: conceptualization, evidence and clinical applications. World Psychiatry, 22 (1 ), 25–41. doi: 10.1002/wps. 2103536640398
Wampold B. E. Imel Z. E. (2015). The great psychotherapy debate: The evidence for what makes psychotherapy work. New York, NY: Routledge/Taylor & Francis Group.
Wampold B. E. Owen J. (2021). Therapist effects: History, methods, magnitude, and characteristics of effective therapists. In Barkham M. Lutz W. Castonguay L. G. (Eds.), Bergin and Garfield's handbook of psychotherapy and behavior change: 50th anniversary edition (pp. 297-326). New York, NY: John Wiley & Sons.
Webb C. A. Derubeis R. J. Barber J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of consulting and clinical psychology, 78 (2 ), 200-211. doi: 10.1037/a001891220350031
