Development and Initial Validation of a Brief Questionnaire on the Patients’ View of the In-Session Realization of the Six Core Components of Acceptance and Commitment Therapy

Acceptance and Commitment Therapy (ACT; Hayes, 2004) is one of the third-wave cognitive-behavioral therapies (CBT). Several reviews and meta-analyses summarized the effectiveness of ACT for various clinically relevant problems (A-Tjak et al., 2015; Graham, Gouick, Krahé, & Gillanders, 2016; Öst, 2014; Powers, Zum Vörde Sive Vörding, & Emmelkamp, 2009; Swain, Hancock, Hainsworth, & Bowman, 2013). A central treatment strategy in ACT is reducing the patients’ psychological inflexibility and thereby increasing psychological flexibility. The ACT model of psychological flexibility consists of the following of six core components (see Table 1): acceptance, cognitive defusion, contact with the present moment, self-as-context, values, and committed action. These six core components of psychological flexibility can be described as mindfulness and acceptance processes (acceptance, cognitive defusion, contact with the present moment, self-as-context) as well as commitment and behavior change processes (contact with the present moment, self-as-context, values, and committed action). The counterparts of these six components of psychological flexibility are formulated in the ACT model of psychological inflexibility (see Table 1): experiential avoidance (vs. acceptance), cognitive fusion (vs. cognitive defusion), dominance of the conceptualized past and feared future (vs. contact with the present moment), attachment to the conceptualized self (vs. self-as-context), lack of values (vs. values), and inaction, impulsivity, or avoidant persistence (vs. committed action).

A meta-analysis on laboratory-based component studies revealed positive effects for treatment strategies on the six ACT core components (Levin, Hildebrandt, Lillis, & Hayes, 2012). Moreover, psychotherapy research has shown that patients who improve their skills in acceptance, cognitive defusion, contact with the present moment, and values-based actions during therapy show better treatment outcomes (e. g., Åkerblom, Perrin, Rivano Fischer, & McCracken, 2015; Arch, Wolitzky-Taylor, Eifert, & Craske, 2012b; Baranoff, Hanrahan, Kapur, & Connor, 2013; Forman, Herbert, Moitra, Yeomans, & Geller, 2007; Forman et al., 2012; Hesser, Westin, & Andersson, 2014; Niles et al., 2014; Vowles & McCracken, 2008; Zettle, Rains, & Hayes, 2011). Interestingly, some of these studies found improvements in ACT processes to be beneficial for the outcome not only in ACT but also in CBT as well as multidisciplinary treatments. ACT processes might therefore be change mechanisms in other psychotherapies than ACT as well, i. e. general change mechanisms. Some ACT processes were even more strongly associated with the outcome in CBT than in ACT, for example in the study by Arch et al. (2012b) in which cognitive defusion predicted worry reductions more in CBT than in ACT.

Several questionnaires have been published to measure a patient’s skill in the ACT components: e. g., Acceptance and Action Questionnaire II (Bond et al., 2011); Acceptance and Action Questionnaire for University Students (Levin, Krafft, Pistorello, & Seeley, 2019); Comprehensive assessment of Acceptance and Commitment Therapy processes (Francis, Dawson, & Golijani-Moghaddam, 2016); Chronic Pain Acceptance Questionnaire (McCracken, Vowles, & Eccleston, 2004), Cognitive Fusion Questionnaire (Gillanders et al., 2014), Multidimensional Experiential Avoidance Questionnaire (Gámez, Chmielewski, Kotov, Ruggero, Suzuki, & Watson, 2014), Tinnitus Acceptance Questionnaire (Weise, Kleinstäuber, Hesser, Westin, & Andersson, 2013), The Valued Living Questionnaire (Wilson, Sandoz, Kitchens, & Roberts, 2010). How strong patients improve their skills in ACT components might depend on the in-session realization of the ACT components. As far as we know, no study has yet explored this research question. This might be because only the observer-based Drexel University ACT/CBT Therapist Adherence Rating Scale (DUTARS; McGrath, 2012) is available to measure the degree the ACT components are realized in a psychotherapy session. The DUTARS was applied in previous clinical trials on ACT to assess treatment adherence (Arch et al., 2012a; Gloster et al., 2015). Although such observer-based measures provide valuable data, there are several barriers to apply observer-based ratings in psychotherapy, especially under the conditions of routine practice. For example, observers must be trained to provide reliable and valid data, financial or other compensations are necessary since observing sessions or session segments consumes a serious amount of time (Weck, Grikscheit, Höfling, & Stangier, 2014), and only certain consent to being observed in-session limiting the generalizability of the results.

Besides observer ratings, ratings given by patients are complementary data sources. Patient ratings on in-session processes are easier to obtain than observer ratings. Patients can fill out session questionnaires directly after the psychotherapy session to measure the degree therapeutic factors were realized in this given psychotherapy session. Patient ratings of in-session processes are especially relevant as they correlate most consistently with psychotherapy outcome (e. g, Horvath & Symonds, 1991; Mander et al., 2013, 2015; Ogrodniczuk, Piper, Joyce, & McCallum, 2000). Several session questionnaires were published on the in-session realization of the therapeutic alliance (Horvath & Greenberg, 1989) and the psychotherapeutic change processes according to Grawe (1997): problem actuation (activation of problems and related emotions), clarification of meaning (acquiring new insights and a deeper understanding of the problems), resource activation (recognizing potential, strengths, and positive facets), and mastery (the ability to cope with problems) (see Mander et al., 2013, 2015). Yet, no session report exists, to our knowledge, which captures the in-session realization of the six core components of ACT. A brief, time-economic and psychometrically sound ACT session report would have the potential to enrich psychotherapy research as well as clinical practice. Clinical implications would be that this measure could be applied in more settings than the observer-based DUTARS and that therapists could use this measure to obtain feedback on the patients’ perspective of the in-session realization of the ACT components.

In the present study, we developed and evaluated a brief ACT session questionnaire (ACT-SQ; see Supplementary Materials). The ACT-SQ was created to obtain patient ratings on the in-session realization of the ACT components of psychological flexibility. In this manuscript, we present two studies. Study 1 investigated the factor structure, the reliability, and the convergent validity. Study 2 analyzed the factor structure, the reliability, the convergent validity, and also criterion validity. The following research questions were evaluated:

Study 1

Study 2

Method

The methods of Study 2 were approved by the local ethics committee (Ethikkommission der Fakultät für Verhaltens- und Empirische Kulturwissenschaften der Universität Heidelberg) and written informed consent was obtained from the patients.

Measures

The ACT-SQ and the SACiP (see measures in Study 1) were administered to patients after the 15^th therapy session and at the end of psychotherapy. Furthermore, the German versions of the Brief Symptom Inventory (BSI; Franke, 2000) and the Beck Depression Inventory (BDI-II; Hautzinger, Keller, & Kühner, 2009) were administered as outcome measures at pre-treatment and post-treatment as well as after the 15^th psychotherapy session. The Global Severity Index (GSI) of the BSI and the total score of the BDI-II were used in the study at hand. These measures are reliable and valid (see for example, Franke, 2000 for the German version of BSI; Derogatis & Melisaratos, 1983 for the English version of BSI; Kühner et al., 2007 for the German version of BDI-II; Beck & Steer, 1998 for the English version of BDI-II). references. Cronbach’s alpha (α) values have been reported to be high: between .92 and .96 for the GSI of the German BSI and ≥ .84 for the German BDI-II.

Participants

Therapists and patients were different from the therapists and patients included in Study 1. Between November 2016 and November 2017, n = 77 CBT trainee therapists working at a large outpatient training center took part. These therapists treated the n = 254 patients who completed the ACT-SQ: n = 115 outpatients completed the ACT-SQ after the 15^th CBT session and n = 156 outpatients completing the ACT-SQ at the end of CBT (post-treatment). As the ACT-SQ was implemented for ongoing and new therapies, these two patient sample were independent from each other except for n = 17 patients who completed the ACT-SQ at both assessment points. A subset of patients filling in the ACT-SQ also provided data for the outcome measures (see flow-chart in Figure 1) and their data was used to evaluate associations between the ACT-SQ and pre-post outcome as well as early and late patient progress (Research Questions 5 and 6).

Click to enlarge

Figure 1

Flow-Chart

The patients answering the ACT-SQ at the end of CBT had on average M = 39.68 (SD = 14.98) individual therapy sessions. The description of the participating patients is given in Table 5. Structured clinical interviews (SCID) were used to make the diagnoses.

Table 5

Description of the Patients of Study 2

Variable	15th session sample		Post-treatment sample
	n	%	n	%
Gender
male	51	44.3	68	43.6
female	64	55.7	88	56.4
Diagnoses according to chapter V of the ICD-10 (all diagnoses, not only primary diagnosis)
F4	68	36.0	87	34.3
F3	72	38.1	102	40.2
F1	10	5.3	16	6.3
F6	22	11.6	23	9.1
others	17	9.0	26	10.2
	M	SD	M	SD
Comorbidity: Amount of diagnoses according to chapter V of the ICD-10
	1.64	.84	1.63	.87
Age at time of assessment
	36.50	13.03	35.73	13.60

Note. F4 = Neurotic, stress-related and somatoform disorders; F3 = Mood (affective) disorders; F1 = Mental and behavioural disorders due to psychoactive substance use; F6 = Disorders of adult personality and behavior. Number of diagnoses higher than number of patients since multiple diagnoses per patients are possible.

Analyses

SPSS 25 was used to perform the statistical analyses. Means (M), standard deviations (SD), frequencies (n), and percentages (%) were calculated for the sample description. An EFA with maximum likelihood estimation and oblique rotation (oblimin direct) was performed to investigate the factor structure of the ACT-SQ. The Kaiser criterion (factors with eigenvalues larger than 1 were retained), the Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO), and the Bartlett’s Test of Sphericity were applied. Cronbach’s alpha (α) was computed to measure reliability. Furthermore, Pearson correlation coefficients (r) were calculated to measure associations between the ACT-SQ and general change mechanisms (convergent validity). Moreover, associations between the ACT-SQ and treatment outcome were explored with linear regression analyses. To measure the pre-post outcome, the outcome measure (GSI, BDI-II) at post-treatment was the dependent variable and the ACT-SQ at post-treatment as well as the outcome measure (GSI, BDI-II) at pre-treatment were independent variables. We also investigated associations between the ACT-SQ and early as well as late patient progress. For early patient progress, the patient reported outcome measure (GSI, BDI-II) at the 15^th CBT session was the dependent variable and the ACT-SQ at the 15^th CBT session as well as the outcome measure (GSI, BDI-II) at pre-treatment were independent variables. For late patient progress, the patient reported outcome measure (GSI, BDI-II) at post-treatment was the dependent variable and the ACT-SQ at post-treatment as well as the outcome measure (GSI, BDI-II) at the 15^th CBT session were independent variables. We also performed these analyses without the ACT-SQ as independent variable to evaluate how the R2-squared values change when including the ACT-SQ as independent variable. All statistical tests were performed two-tailed and the significance value was set to p < .05. Results will be given with and without Bonferroni-correction for multiple comparisons.

General Discussion

A brief session questionnaire ACT-SQ was designed to obtain patient ratings on the in-session realization of the ACT components of psychological flexibility. The ACT-SQ was evaluated in ACT as well as CBT.

Results showed a one-factor solution (Research Question 1) and a good reliability (Research Question 2). All KMO values were good (.7 - .8) or great (.8 - .9) according to Hutcheson and Sofroniou (1999) or Field (2009). Moreover, all Bartlett’s tests were significant indicating that factor analysis was appropriate (Field, 2009). The loadings of all items were well above .45 as recommended in the literature (see for example, Bühner, 2010) and there were no cross-loadings. The one extracted factor could stand for the degree the in-session processes helped to increase the patient’s psychological flexibility. To further evaluate this hypothesis, a study is necessary investigating whether higher ACT-SQ session scores result in more improvements on established instruments measuring skills of psychologically flexibility (e. g., Acceptance and Action Questionnaire II; Bond et al., 2011).

Besides factor structure and reliability, we tested the convergent validity. Convergent validity was evaluated by correlating the ACT-SQ with the general change mechanisms proposed by Grawe (1997) since these mechanisms are considered to be relevant in all psychotherapies and because ACT processes might also be general change mechanisms as they mediated the outcome not only in ACT but also in CBT and multidisciplinary treatments (e. g., Åkerblom et al., 2015; Arch et al., 2012b). These analyses related to Research Question 3 revealed that the ACT-SQ is significantly associated with general change mechanisms (except for problem actuation in Study 1) according to Grawe (1997), most strongly with resource activation and mastery. A cautious clinical interpretation of these findings could be as follows: The content of the ACT-SQ items are associated with coping and self-efficacy as is the content of the items of the SACiP resource activation and mastery scales (Mander et al., 2013). Furthermore, the SACiP emotional bond and agreement on collaboration scales reflect the interaction processes between patient and therapist. The ACT-SQ items do not directly target this therapeutic relationship aspect. Hence, stronger associations of ACT-SQ and resource activation and mastery than with the alliance scales seem plausible. In summary, it is important to note that the ACT-SQ items are most strongly related to proximal items (resources and mastery) but also to items with more distanced but clinically relevant content (therapeutic alliance). This further underlines the validity of the measure. With regard to similarities and differences between ACT and CBT (Arch & Craske, 2008; Harley, 2015), most psychometric values were comparable between ACT and CBT, only a few differences emerged in the context of convergent validity (Research Question 4): associations between the ACT-SQ and problem actuation reached significance only in CBT. This could indicate more overlap between problem actuation and the ACT components in CBT than in ACT but it could also be related to the fact that the sample size of Study 1 (ACT) was not as large as the sample size of Study 2 (CBT). The same reasons might explain why the association between the ACT-SQ and the emotional bond was not significant anymore after controlling for multiple testing in Study 1 (ACT) but not in Study 2 (CBT).

In another step, we tested the criterion validity. This was related to Research Question 5 and the results showed significant associations between the ACT-SQ and outcome measures. It should be kept in mind, however, that relations with treatment outcomes were investigated only in CBT. Future research is necessary to evaluate whether the associations between the ACT-SQ and treatment outcomes are comparable or different between CBT and ACT. Finally, the factor structure, reliability, convergent validity, and criterion validity were comparable between earlier and later treatment phases (Research Question 6). Although differences in treatment phases have been highlighted (Ilardi & Craighead, 1994; Lambert, 2005), these differences did apparently no affect the psychometric values of the ACT-SQ.

A limitation of the current work is that only a patient version of the ACT-SQ was created and evaluated. A therapist version of the ACT-SQ would be an important tool that could be developed by future studies to get a more comprehensive picture of the therapeutic process. Other shortcomings of the studies at hand are that criterion validity was tested only in CBT but not in ACT. Moreover, contrasting the psychometric values in earlier vs. later treatment phases was possible only in CBT but not in ACT. Future studies on ACT are important to investigate criterion validity and similarities/differences between earlier and later ACT phases. A further limitation is that the mean of sessions attended was relatively high so that it remains unclear how well the results can be generalized to shorter psychotherapies. Moreover, we did not include other measures of ACT processes to correlate them with the ACT-SQ. Further validation studies should, therefore, compare ACT-SQ patient ratings with observer-based DUTARS ratings, since patient ratings are only one data source to rate in-session processes. Related to the factor analysis, setting the Kaiser criterion for determining the amount of factors at 1 is rather an arbitrary rule of thumb and an empirically founded way of determining the factors (i.e. Horn’s parallel analysis or Velicer’s MAP test) would have been a better method. In replication studies with larger samples, the factor structure needs to be tested with confirmatory factor analysis whether the instrument shows adequate model fit (Bühner, 2010). It is per se more probable for such a short questionnaire like the ACT-SQ to have a one-factor solution. Another suggestion for future research would be to enter additional predictors to the regression analyses to test interactions between patient characteristics (e. g., amount of diagnoses) and the impact the ACT-SQ has on the outcome. It would also be very interesting for future research to examine whether the factor structure of the ACT-SQ remains stable when patients are treated by specific ACT modules (open vs. engaged, see Villatte et al., 2016). The ACT-SQ might also be useful to measure adherence to ACT and to continuously track the ACT processes during psychotherapy. Parallel session-to-session assessments of the ACT processes and outcomes would allow investigating how the ACT processes are associated with patient progress on a between- and within-person level (Rubel et al., 2017). Such a systematic monitoring would also enable evaluating the ACT processes before and after sudden losses or sudden gains (Wucherpfennig, Rubel, Hofmann, & Lutz, 2017). Future research on group psychotherapy could also explore associations between group factors (see for example, Tasca et al., 2016, and Vogel, Blanck, Bents, & Mander, 2016) and ACT components.

In summary, the ACT-SQ has a clear factor structure, good reliability, shows strong associations to other validated psychotherapeutic change processes and is associated with treatment outcomes. Implications of this study are that the license-free ACT-SQ is a reliable and valid measure that can be used to measure how patients experience the in-session realization of ACT components.