The discrepancy between the review result of the GSBP and other international reviews on HPT (e.g, by Elliott et al., 2013; Elliott et al., 2021) can be explained through the different methodological approaches, e.g. using meta-analyses and pooling effects for a certain disorder (Elliott et al., 2013; Elliott et al., 2021) vs. defining a required minimum of evidence (at least three RCTs with sufficient internal and external validity) for a certain number of diagnostic groups (currently: 4 out of 12) in the GSBP approach. Moreover, the discrepant definition of what constitutes HPT as defined by the German Work Group Humanistic Psychotherapy from the international definition, e.g. as defined in the reviews by Elliot is likely to be relevant here. The assignment of subapproaches to HPT is different across author groups and also within type of publications. Elliott et al. (2021) use the term HPT to embrace 10 subapproaches (person-centered therapy, emotion-focused therapy, motivational interviewing, gestalt, existential, psychodrama, focusing-oriented, expressive and body-oriented as well as “supportive / nondirective”), while in Elliott et al. (2013) the respective analyses included 9 subapproaches, omitting motivational interviewing. On the basis of current discussions in psychotherapy research (e.g., Wampold, 2015), we hypothesized in addition that two key reasons may have an impact on the effectiveness on HPT subapproaches (Schünemann et al., 2019):

1) Researchers’ allegiance that may be associated with a decrease of the (possible) effects of the non-preferred treatment condition (Munder et al., 2011), in case of the review of the GSBP a decrease of the effects of subapproaches of HPT. Leykin and DeRubeis (2009, p. 55) define “Allegiance, in the context of treatment outcome research, …[as] a belief in the superiority of a treatment. It usually also entails a belief in the superior validity of the theory of change that is associated with the treatment”. Often, this belief is associated with therapy outcomes and may reflect a risk of bias. A meta-analysis on the allegiance bias hypothesis demonstrated a moderate association between allegiance and treatment outcome moderated by methodological quality (Munder et al., 2011), indicating that allegiance as well as methodological quality may need to be taken into account for treatment comparisons (Munder et al., 2011). In the context of the HPT evaluation of the GSBP, researchers’ allegiance may have had an effect because many studies in the pool used a study design in which a humanistic subapproach was designed as control group in comparison to other psychotherapeutic approaches (e.g., CBT).

2) We further assumed that treatment quality of HPT studies in the GSBP evaluation may have an impact on outcomes. In this context, the two aspects of bona fide psychotherapy and adherence can be differentiated. Whereas bona fide represents conceptual quality of a treatment, adherence is rather concerned with process quality. For the present study, bona fide psychotherapy will be defined as follows: mentioning or describing an established psychological approach, psychological treatment principles, a treatment manual or active treatment ingredients (Benish et al., 2008) as well as the requirement that the intervention is implemented by a trained therapist (Wampold et al., 1997). Further, we use a definition of adherence in accordance with Munder et al.’s (2013) definition of treatment integrity: “[…] conceptualized broadly including adherence to specific treatment procedures (e.g., the importance of exposure in psychotherapy for post-traumatic stress disorder), common factors (e.g., therapeutic alliance), and therapist effects (i.e., differences in the effects due to individual therapists)” (Munder et al., 2013, p. 8). Concerning treatment quality, Wampold (2015) summarizes that trainers’ competence has only a small (d = 0.14) and adherence to protocol an almost negligible effect (d = 0.04) on treatment outcome. The (non-) effect of adherence to protocol may be explained with associated decreased patient-therapist alliance, increased likelihood of resistance to treatment and a lack of flexibility by the therapist. We hypothesized that the GSBP-evaluation included studies implementing HPT subapproaches (often as control group) in potentially fuzzy and poor quality so that significant HPT mechanisms could not work properly resulting in potentially decreased effects of the respective HPT subapproach.

The aim of this meta-analysis is the re-examination of studies included in the GSBP evaluation of HPT (GSBP, 2018) (based on the application of the Work Group Humanistic Psychotherapy and the additional searches conducted by the GSBP) to examine potential moderating effects of allegiance and treatment quality (bona fide psychotherapy and adherence) on the comparative effectiveness of HPT (including both efficacy and effectiveness studies). We examine whether in the HPT subapproaches vs. others psychotherapeutic interventions included in the GSBP evaluation 1) allegiance to a particular psychotherapeutic intervention and 2) treatment quality (bona fide/adherence) significantly moderate effect sizes. We expected higher allegiance towards other psychotherapy approaches and lower treatment quality to be significant predictors of lower effect sizes in the HPT condition.

This study was registered in the PROSPERO International prospective register of systematic reviews (CRD42019128983). The present full report relates to the second objective in this protocol (comparison of humanistic subapproaches versus other psychotherapeutic interventions).

The study pool which builds the basis of the present analysis is taken over from the GSBP (2018). For their final sample, the GSBP screened abstracts of n = 481 studies from which N = 114 went into the final pool of evaluated studies. The abstract screening procedure was based on 1) a list of studies submitted by the Work Group Humanistic Psychotherapy (n = 313), and 2) results from an independent literature search conducted by an independent institution. According to pre-specified PICOS criteria, all study abstracts were screened; the ones included went into full-text screening review by two independent reviewers (members of the GSBP) to decide about their further inclusion.

The present systematic review used the identical study pool of the GSBP (N = 114) in order to compare the results of this systematic review directly with the conclusions of the GSBP and by this to be able to provide recommendations for future analyses on psychotherapeutic approaches or methods by the GSBP.

It was necessary to conduct a secondary study screening for eligibility because the present systematic review needs specific study data beyond the data relevant for the purpose of the GSBP. As result, we used N = 50 studies which compared HPT subapproaches to alternative evidence-based psychotherapeutic interventions (see flowchart Figure 1). The remaining studies were either excluded or relate to inactive control conditions.

Study characteristics for the present meta-analyses were extracted by one author (either OS or AJ). Study results, study quality, allegiance and treatment quality were rated by two of the authors independently (OS & NH or AJ & UW or NH & UW). Inter-rater reliability (Cohen`s Kappa) for adherence (only available for one of the two rater teams: NH, OS) across all intervention groups (IG and CG) was moderate (Altman, 1999): κ = .507 (p < .001); agreement = 76.7%. Inter-rater reliability (intra-class correlation, ICC) for allegiance (MARS index) was .91 (95% CI [.82, .95]; two-way mixed-effects ICC, absolute agreement, average for two coders, same rater team).

Discrepancies were first discussed within dyads and if discrepancies could not be clarified, were discussed among all four authors until a final decision was reached.

Inclusion criteria were based on those of the GSBP and complemented by the availability of data we needed for the meta-analysis. Inclusion criteria were: 1) RCTs or non-randomized controlled trials with active control group and 2) report of pre- and post-assessments regardless of follow-up assessments. These criteria were complemented with the following exclusion criteria: 1) an active control group was missing (only two or more HPT subapproach-groups with no additional active control group precluding the examination of allegiance); 2) a metric outcome measure or post mean were lacking or 3) an indication of data manipulation could be found. For more details see Schünemann et al. (2019) or Supplementary Materials (Additional inclusion and exclusion criteria).

We extracted information on participant characteristics, study characteristics, intervention characteristics, primary and secondary outcomes, risk of bias, allegiance, treatment quality (bona fide and adherence). Guidelines of the Cochrane Collaboration were used to estimate and substitute missing data for outcomes, e.g., calculating standard errors from exactly reported t-values. The primary effectiveness outcome was symptom severity at the end of treatment measured on a metric symptom specific scale. Outcomes on self-rating scales (e.g., BDI) were given priority over observer-rated scales (e.g., HDRS). As symptom reduction is not necessarily the primary target of change across different psychotherapy approaches, we extracted data for different outcome domains to analyse moderator effects specific to different outcome domains. Secondary outcomes were interpersonal outcomes (e.g., DAS), general assessment of functioning (e.g., GAF) and quality of life (e.g., WHO QOL). The primary outcome was extracted for short-term (end of intervention) and follow-up if available (6 months after end of intervention or the one closest to 6 months). As primary negative outcome drop-out until end of intervention was extracted.

Allegiance was assessed according to the multilevel allegiance rating-scale (MARS) provided by Steinert et al. (2017). This instrument combines information about 1) researchers’ allegiance either respective treatment development or contribution to an etiological understanding of the treated disorder; 2) therapists’ allegiance; 3) trainers’ allegiance and 4) supervisors’ allegiance to a total score (0-4; Steinert et al., 2017).

Treatment quality with bona fide psychotherapy and adherence were rated using the definition by Wampold et al. (1997) and Benish et al. (2008) according to the following items: using/citing an established psychotherapy manual; used intervention is based on psychological principles; author mentions HPT (subapproach)/CG on own his/her website; author has other relevant HPT (subapproach)/CG publications; intervention was carried out by trained therapists. To meet the criterion of bona fide two of the items needed to be fulfilled. In the moderator analyses “bona fide” was compared to “non or unclear bona fide”. To fulfill the criterion of treatment adherence, treatment conditions and therapeutic procedure needed to be described in detail and adherence must have been proved by external raters. In the moderator analyses “non-adherence” was compared to “adherence”. Study quality of the included studies was rated according to the second version of Cochrane’s risk of bias tool (RoB 2.0; Higgins et al., 2018) considering the adaptions by Munder and Barth (2018) for its use in psychotherapy outcome research. Thus, the methodological quality was assessed via: 1) bias arising from the randomization process; 2) bias due to missing outcome data; 3) bias in outcome measurement; and 4) bias in selection of the reported result (Cochrane’s risk of bias tool) as well as 5) effect of adhering to intervention (see Higgins et al., 2018; Munder & Barth, 2018). For non-randomized controlled trials, the ROBIN-I tool (Sterne, Hernán, et al., 2016) was used to rate risk of bias. For moderator analyses a “low risk of bias” in each category was compared to an “unclear or high risk of bias”.

Standardized mean difference for metric measures and odds ratios for rare outcomes between the intervention groups at end of intervention and follow-up were calculated using the intention-to-treat sample, if available. For all analyses, a random effects model with inverse variance weights was applied (DerSimonian & Laird, 2015). Cochran’s Q-test was used and quantified using the I²-statistic (Higgins et al., 2003) to test statistical heterogeneity between study results for significance. Visual examination of funnel plots and Egger’s test (Sterne, Egger, & Smith, 2016) were applied to examine possible publication bias.

A priori defined subgroup (in case of categorical predictors) or meta-regression (in case of metric predictors) analyses (univariate) were conducted concerning study quality, allegiance, treatment quality, type of non-active control (waitlist vs. all others including TAU), type of HPT subapproach (client centered vs. all others) and population (children/adolescents vs. adults). Differences between subgroups were tested formally (Bucher et al., 1997; Deeks et al., 2001; Song et al., 2003). A posteriori (explorative) meta-regression analyses (univariate) were performed in case of considerable heterogeneity between studies for number of sessions, length of intervention, percentage of women, affective disorder, anxiety disorder, post-traumatic stress disorder (PTSD), F 54, study design (no RCT) and comparator (CBT). All analyses were conducted using the metafor package in R (Viechtbauer, 2010).

Because of space limitations additional information and results are documented in the Supplementary Materials. Individual characteristics of all studies are shown in the Supplementary Materials (Supplementary Table 1). All ratings for Cochrane’s risk of bias and ROBIN-I tool (study quality) as well as for allegiance and treatment quality for each study can be found in the Supplementary Materials (Supplementary Table 2). Further, the overall main effect sizes are presented in Supplementary Table 3. In order to avoid misunderstandings when interpreting the results of the overall effects a more detailed explanation of the main effects can also be found in Appendix C – Supplementary Information (see Preliminary Results: Overall main effect sizes of HPT). The forest plot for the primary outcome is displayed in Figure 2.

We will first report on moderation effects (pre-specified moderators followed by exploratory moderators) for post-intervention outcomes and then move to reporting the results for the same moderators at follow up.

The results for the moderator analyses for the symptom severity (primary outcome) are presented in Table 1A. No significant moderating effects on the primary comparative effectiveness outcome symptom severity at end of intervention could be shown for the pre-specified moderators study quality, allegiance or treatment quality.

Exploratory subgroup analyses indicated that the difference between HPT and other psychotherapies regarding symptom severity increased significantly favouring other psychotherapies when CBT was used as comparator and in studies examining anxiety disorders. Sensitivity analysis for this comparison showed that results were not influenced by the study design.

For the outcome symptom severity at follow-up subgroup analyses indicated that the difference between HPT in comparison to other psychotherapies was reduced significantly in studies with no or unclear allegiance against HPT (β = -0.208, p = .044; for examples for interpretation, please see table note).

Results for secondary outcome domains are shown in Table 1B. For the outcome interpersonal problems subgroup analyses indicated that the difference between HPT in comparison to other psychotherapies was reduced significantly in studies with no or unclear allegiance against HPT. Also, the difference between HPT in comparison to other psychotherapies was reduced when bona fide HPT was used, in studies with a low risk of bias due to missing data and in studies not examining anxiety disorders. Yet, somewhat contradictory, the difference increased in favour of other psychotherapies when the control group was not adherent to intervention. No significant moderator effects were detected for the outcomes general functioning and quality of life.

The results for some of the outcome measures should be interpreted with caution because of the small number of studies (n = 9 for interpersonal problems, n = 12 for general functioning, n = 8 for quality of life) in comparison to the large number of moderator analyses.

To test whether primary moderators were sufficiently independent, we examined their associations post-hoc via Fisher’s Exact Test (FET). Results showed a significant association between allegiance and bona fide HPT (p = .01, FET). Moreover, bona fide HPT was significantly associated with adherence HPT (p = .007, FET) as well as with adherence to the control condition (p = .02, FET). There were no other significant associations between primary moderators.

The high proportion of allegiance against HPT as well as the high number of non-bona fide treatments in the HPT-treatment arm in the present meta-analysis has also been reported in other studies (Cuijpers et al., 2012; Elliott et al., 2013; Elliott et al., 2021). Cuijpers et al. (2012) concluded that non-directive supportive therapy (as a subapproach of HPT) for depression is equally effective as other psychological treatments after researcher allegiance was controlled. Similarly, Elliott et al. (2013) report a drop in (weighted) effects when comparing supportive therapies to other psychotherapy approaches. In addition, researcher allegiance has been demonstrated to have an impact on outcomes in psychotherapy studies not focusing on HPT alone (Munder et al., 2013).

When focusing on the impact of allegiance, it is important to keep in mind that it is not useful to interpret it as a purposeful attempt to skew results but rather take it into account as a human tendency to believe in one’s own ideas and practices in a way that objectivity is compromised (Lomangino, 2016; Yoder et al., 2019). This perspective is supported by research that shows that awareness and acceptance of its potential impact may reduce its effects (Munder et al., 2013). Another option to avoid a bias through allegiance are allegiance-controlled trials where the interventions in the different conditions are planned and supervised by proponents of the respective approaches (Barkham et al., 2021; Leichsenring et al., 2009). Similarly, more recently, bona fide is also explicitly considered in trial designs: In their recent RCT comparing person-centered experiential therapy (PCET as a HPT subapproach) and CBT, Barkham et al. (2021) carefully implemented bona fide treatment arms with a similar level of professional training with the help of trainers qualified in the respective interventions.

We did not find moderation effects of adherence on comparative effectiveness. A recent systematic review and meta-analysis of the role of adherence for outcome in child and adolescent psychotherapy found a small (statistically significant) (interventional, non-comparative) effect size for adherence of d = 0.096 (95% CI [.058, .124]. Considering the very small effect size, the authors conclude - despite the statistical significance - that other factors than adherence are much more relevant to consider for outcome. They included primary studies if “external or independent observer rated measures of adherence or competence, measures across multiple time points (…)., and interrater agreement established in the study or use of coders trained to this level (…) (ICC) > -0,60 or Kappa > -0,61 or percent agreement > - 90% or the score is based on agreement by multiple rates” (Collyer et al., 2020, p. 419). Our criteria to evaluate adherence were less demanding than in Collyer et al. (2020). When adherence as a construct is significantly linked to outcome, quite likely its actual impact upon explaining differences in outcomes remains very small with the presented mean effect size estimation of r < 0.10 (i.e. accounting for less than 1% of the variance in therapy outcome).

With respect to study quality (risk of bias, assessed via RoB), we found no clear evidence for moderation effects. These results are in line with the recent studies by Cuijpers, Quero, et al. (2021) as well as Hoppen and Morina (2020). In this context it has to be kept in mind that study quality is quite likely not fully independent from allegiance: Munder et al. (2011) found that allegiance is more strongly related to outcome in studies with lower methodological quality. The inclusion criteria of the GSBP define relatively high methodological standards. These standards may have prevented that allegiance effects unfold as strongly in our study as maybe in other meta-analyses where inclusion criteria are less methodologically strict.

The present study compasses a considerable number of trials, examined important predefined moderators and demonstrates the challenges in evaluating the significance of allegiance and treatment quality (bona fide and adherence). However, in conducting this meta-analysis, we also were faced with a number of problems:

The present results could not demonstrate effects of the examined moderators on treatment outcome. However, we mainly examined studies with high study quality. This is crucial to consider for generalizing our results and it is necessary to prove whether results can be replicated for studies with low study quality.