==== Front Internet IntervInternet IntervInternet Interventions2214-7829Elsevier S2214-7829(17)30025-810.1016/j.invent.2017.10.005Full length ArticleMy care manager, my computer therapy and me: The relationship triangle in computerized cognitive behavioural therapy Cavanagh Kate kate.cavanagh@sussex.ac.uka⁎Herbeck Belnap Bea bRothenberger Scott D. cAbebe Kaleab Z. dRollman Bruce L. ba University of Sussex, School of Psychology, East Sussex, United Kingdomb Division of General Internal Medicine, Center for Behavioral Health and Smart Technology, University of Pittsburgh School of Medicine, Pittsburgh, PA, United Statesc Center for Research on Health Care Data Center, University of Pittsburgh, Pittsburgh, PA, United Statesd Division of General Internal Medicine, Center for Clinical Trials & Data Coordination, University of Pittsburgh School of Medicine, Pittsburgh, PA, United States⁎ Corresponding author at: School of Psychology, University of Sussex, Falmer, East Sussex BN1 9QH, United Kingdom. kate.cavanagh@sussex.ac.uk06 11 2017 3 2018 06 11 2017 11 11 19 24 2 2017 25 9 2017 28 10 2017 © 2017 The Authors2017This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).Previous research has reported mixed findings regarding the relationship between therapeutic alliance, engagement and outcomes in e-mental health. This study aims to overcome some of the methodological limitations of previous research and extend our understanding of alliance-outcome relationships in e-mental health by exploring the nature of the relationship triangle between the patient, their care manager and their computerized cognitive behavioural therapy (CCBT) program, accessed with or without an Internet Support Group (ISG). Positive patient-rated alliance with both their care manager and the CCBT program itself was found and these were significantly associated with measures of engagement and clinical outcome. The magnitude of this association was moderate, and within the range of that reported for traditional face-to-face psychotherapies in recent meta-analyses. Limitations of the study, including the reliance on completer data and a cross-sectional design, and directions for future research are presented. Our findings suggest that both the training and supervision of support staff and the optimization of CCBT interventions themselves to enhance alliance and experience may lead to improved engagement and outcomes. Trial Registration: Clinicaltrials.gov Identifier: NCT01482806https://www.clinicaltrials.gov/ct2/show/NCT01482806?term=rollman&rank=4 Highlights • First large clinical trial to measure working alliance and CCBT user experience delivered via a collaborative care strategy. • Patient rated working alliance and CCBT user experience rated positively. • Patient rated working alliance and CCBT user experience associated with engagement measures and clinical outcomes. Keywords Computerized CBTDepressionAnxietyTherapeutic relationshipWorking alliance, clinical outcomes ==== Body 1 Introduction There is growing evidence that guided computerized cognitive behavioural therapies (CCBT) can be an effective intervention for common mental health problems, including anxiety and depression (Andersson, 2016, Andersson et al., 2014, Andrews et al., 2010, Grist and Cavanagh, 2013, Newby et al., 2016, Richards and Richardson, 2012). Most of the studies to date have focused on evaluating the feasibility and clinical outcomes of such interventions, whilst limited research has begun to explore the change processes associated with their impact (Cavanagh and Millings, 2013). 1.1 Working alliance and guided CCBT In traditional psychological therapies the quality of ‘common factors’, including the therapeutic relationship, are widely held to be important for patient engagement and clinical outcomes (Horvath et al., 2011, Lambert and Barley, 2001). These include the ‘working alliance’ – a collaborative relationship between the patient and professional consisting of three elements: agreement on the goals of treatment, agreement of the tasks of treatment, and a positive personal/emotional bond (Bordin, 1979). Guided CCBT presents a challenge to the importance of these factors as therapeutic interactions are typically limited, remote, and often asynchronous; for example, communication may occur only by text message or email. CCBT is commonly offered as a ‘predominantly self-guided therapy’ or ‘minimal contact therapy’ (Newman et al., 2011), with contact time ranging from < 1.5 h in total (Andersson, 2009, Titov, 2011) to more active involvement by the clinician, but to a lesser degree than in a traditional therapy for the target problem (Newman et al., 2011). Despite the belief amongst many clinicians that extended face-to-face contact is essential for a meaningful working alliance to be established or maintained (Berger, 2015, Lopez, 2015), where measured, the client-rated relationship appears fairly robust to distance and limited contact. Perhaps surprisingly, given the more limited nature of this contact, where compared, no significant differences in patient-rated alliance have been found between guided CCBT and face-to-face CBT (Kiropoulos et al., 2008). However, the therapeutic relationship in guided CCBT may be less intimately associated with therapy outcomes than in traditional therapies (Berger, 2015, Cavanagh and Millings, 2013), and mixed findings have been reported on its relationship between the patient-rated therapeutic relationship and outcomes. Andersson et al. (2012) found that whilst working alliance ratings were high in the three samples of participants in their study (with depression (n = 49), generalized anxiety disorder (n = 35) and social anxiety disorder (n = 90)), these ratings were unrelated to self-reported measures of the primary outcome for each group. Similar findings have been reported elsewhere (e.g. Preschl et al., 2011, Richards and Richardson, 2012), whilst other studies have reported significant alliance-outcome relationships (e.g. Nordgren et al., 2013, Wagner et al., 2012). One explanation for this discrepancy in the literature may be that most studies in the field are not adequately designed to detect the relatively moderate correlation that might be expected between working alliance and clinical outcome. In a meta-analysis of > 200 studies of the alliance-outcome relationship in traditional face-to-face psychotherapies Horvath et al. (2011) identified an aggregate effect size of 0.275 (95% CI 0.25–0.30). A priori power calculations indicate that a sample size over 100 is required in order to be robustly designed to test this magnitude of association (G*power calculation, Power 0.9, Alpha 0.05, Faul et al., 2009) - a sample size well beyond most studies in this field. Indeed, Berger (2015) has noted that where the association between alliance and outcome is reported in published studies it is typically in a positive direction, if not statistically significant, supporting the notion that many studies reporting this relationship may simply have inadequate power to test such hypotheses. 1.2 The ‘relationship triangle’ in guided CCBT Most research in this area has focused on testing the reach and relevance of the working alliance between the patient and their ‘CCBT guide’ (i.e. their therapist or coach) in relation to its impact on outcomes for patients (Berger, 2015). In addition to this patient-professional relationship, there is some evidence that CCBT programs themselves can be designed to elicit relational experiences in program users (e.g. Barazzone et al., 2012), and it may be that this patient-program ‘relationship’ also contributes to engagement and outcomes in guided CCBT (Cavanagh, 2010, Cavanagh and Millings, 2013). A minority of studies have adapted measures of the therapeutic relationship to consider the user's experience of relating to the CCBT program itself, and at least some outcomes support the idea of an association between the patient-rated ‘relationship’ with the CCBT program and therapy outcomes (e.g. Berger et al., 2014, Meyer et al., 2015, Ormrod et al., 2010). It has been argued that this patient-program relationship may in fact be more closely associated with patient outcomes than therapist-alliance ratings (Berger et al., 2014, Berger, 2015). To date no studies have explored both elements of this ‘relationship triangle’ - that is both the nature of the patient-professional relationship and the patient-program relationship within one single study. One possibility is that relating experiences with both the guide and the CCBT program itself may contribute independently or interactively to the user's experience, engagement and outcomes associated with their care episode (Cavanagh, 2010). Furthermore, larger studies are needed to fully test the alliance-outcome relationship in CCBT and in order to draw conclusions about how it compares to the outcome-alliance relationship in traditional psychotherapies. 1.3 Aims of the study In the parent study, the randomized controlled trial “Online Treatments for Mood and Anxiety Disorders in Primary Care” (Online Treatment Trial), participants with anxiety and/or depression were offered access to a guided CCBT program, with or without access to a moderated Internet Support Group (ISG), supported by a care manager in the context of a 6-month collaborative care strategy. Working alliance, engagement, and clinical outcomes were measured in the context of the parent study that showed that participants randomized to guided CCBT report significant 6-month improvements in mood and anxiety symptoms, but not in mental-health related quality of life, compared to those receiving their physician's usual care (Rollman et al., 2017). This study aimed to explore the nature of the relationship between the participant, their care manager and their CCBT program, in a large study well powered to detect alliance-outcome relationships. 2 Methods 2.1 Study setting The Online Treatment Trial was conducted at 26 primary care offices that shared a common electronic medical record system (EMR; EpicCare, Madison, WI), implemented a protocol approved by the University of Pittsburgh's Institutional Review Board, and described in detail elsewhere (Rollman et al., 2017) (See Fig. 1). Briefly, primary care physicians (PCP) in participating practices received an EMR “Best Practice Alert” reminder about the trial study for all patients aged 18–75 whenever anxiety, generalized anxiety, panic, or depression was entered as an encounter diagnosis. If the patient agreed to a referral, then the PCP forwarded the patient's name and telephone number to a study recruiter who then telephoned the patient to confirm preliminary eligibility (Rollman et al., 2008).Fig. 1 Flowchart of participants. 1Working Alliance Inventory, ‘Beating the Blues’ Sessions Rating Scale, and Overall Treatment Satisfaction Scale. Abbreviations: CCBT, computerized cognitive behavioural therapy; GAD-7, 7-item Generalized Anxiety Disorder Scale; ISG, Internet support group; MH, mental health; PCP, primary care physician; PHQ-9, 9-item Patient Health Questionnaire; PROMIS Anxiety, Patient-Reported Outcomes Measurement Information System for Anxiety (fixed length, short form); PROMIS Depression, Patient-Reported Outcomes Measurement Information System for Depression (fixed length, short form), SF-12 MCS, Medical Outcomes Study Short Form Mental Component Scale. Fig. 1. Study recruiters screened referred patients for the presence of an anxiety disorder and/or depression using the Generalized Anxiety Disorder scale (GAD-7; Spitzer et al., 2006) and Patient Health Questionnaire (PHQ-9; Kroenke et al., 2001) respectively. If a patient scored ≥ 10 on either the PHQ-9 or GAD-7, signifying an at least moderately severe symptom level, met all other eligibility criteria (e.g., reliable Internet access, no severe mental health conditions, medically stable), and consented to the trial, recruiters administered the PRIME-MD Anxiety and Mood Modules (Spitzer et al., 1994) to establish a diagnosis and collected information on patients' self-reported race, gender, marital status, employment, and use of pharmacotherapy, followed by the clinical assessment battery. Protocol-eligible participants were randomized in a 3:3:1 ratio to either: (1) 24/7 access to a CCBT program provided under the guidance of a care manager (CCBT-alone; N = 301); (2) CCBT-alone plus access to a password-protected and moderated Internet Support Group (CCBT + ISG; N = 302); or (3) their PCP's “usual care” (UC; N = 101). 2.2 Interventions 2.2.1 Computerized cognitive behavioural therapy All participants randomized to either of the intervention groups received username and password for “Beating the Blues,” an evidence-based CCBT program (Proudfoot et al., 2004). It consisted of eight 50-min long interactive sessions and “homework” to complete between weekly sessions. Each session included simple and easily understood text, audio, and audiovisual clips to educate and teach basic CBT techniques (e.g., automatic thoughts, thinking errors, attributional styles) for up to three problems that the participant could choose during the first session. At the start of each session participants entered their mood symptoms using the PHQ-9 and their anxiety symptoms with the GAD-7, as well as their perceived burden by their problems. 2.2.2 Internet support group Participants randomized to the CCBT + ISG study arm, received also access to a password-protected and moderated study ISG and a unique username to preserve confidentiality. The ISG featured a variety of discussion boards on which participants posted and a collection of links to various resources (e.g., local $4 generic pharmacy programs, crisis hotlines, brief YouTube videos on such topics as stress management, sleep hygiene, and exercises). It also had a direct link to the CCBT program to further encourage participants to complete all sessions in a timely manner. Engagement in the ISG was promoted via email notifications of new ISG activities, invited member-guest moderators, and various contests and personalized home pages to encourage log-ins and posts. 2.2.3 Care manager support College graduates with mental health research experience and trained in a basic understanding of mood and anxiety disorders and the trial's interventions served as care managers. Following randomization, one of the care managers emailed the participant a web-link to the CCBT program, and, if applicable, the ISG, and telephoned the participant for an introductory telephone call to review the program(s) and establish rapport. They encouraged participants to complete a new session every 1–2 weeks, thus ideally finishing the program during the first three months of our interventions. Later, care managers monitored the progress of their participants using the reports generated by the “Beating the Blues” program (e.g., sessions completed, symptom ratings, stress ratings). They emailed participants a positive feedback after each completed session. If a participant did not complete a new session within two weeks, care managers sent an email reminder, and this was repeated once a week for one month, and then they telephoned participants to motivate them to complete the program or assist with technical problems. If a participant reported a worsening of symptoms or suicidal ideation, the care manager emailed or telephoned the at-risk individual to provide support, reinforce use of the CBT skills, and/or launch our suicide risk management protocol (Herbeck Belnap et al., 2015). 2.2.4 Case review and follow-up Each care manager presented their participants to the study PCP, psychiatrist, and psychologist-study coordinator (Clinical Team) in separate, weekly 60-min case review sessions using the trial's electronic patient registry that could sort patients by randomization date, last contact, and highest self-rated PHQ-9 or GAD-7 score. In addition to conveying general lifestyle adjustments including exercise and social engagement, the Clinical Team recommended antidepressant/anxiolytic pharmacotherapy based on participants' treatment preferences and response to CCBT; and referrals to mental health specialists (MHS) when they did not improve or had complex psychosocial issues. Given the trial's collaborative care framework, the care managers notified PCPs of their patients' progress and our treatment recommendations, as they prescribed and approved all adjustments to their patients' pharmacotherapy and the trial never dispersed medication to any participant (Rollman et al., 2003). Depending upon the participant's treatment choices, symptoms, and motivation, the care manager sent customized emails or telephoned the participant bi-weekly for the first two months of the intervention. These contacts lasted approximately 15 and 30 min, respectively. Afterwards, the care manager contacted the participant approximately monthly until the end of our 6-month intervention. We recorded number of emails, telephone calls, and total contacts to each participant. 2.3 Study participants Participants of the present study were those randomized to one of the intervention study arms as part of the parent trial and completed the ‘end of treatment experience questionnaires.’ Participants assigned to the UC condition are not reported on in this study, as they were not exposed to the CCBT program. 2.4 Assessments Research staff blinded as to the participant's randomization status administered via telephone the clinical outcome assessment battery at baseline, 3-, and 6-months. The battery included the 12-Item Short Form Health Survey (SF-12) to determine health-related quality of life (HRQoL) (Ware et al., 1995) and the fixed length Patient-Reported Outcomes Measurement Information System (PROMIS) measures to assess mood and anxiety symptoms (Pilkonis et al., 2011). At the end of the 6-month intervention, we emailed all participants from both intervention arms a link to complete measures of the end-of-intervention experience questionnaires. Alternatively, participants could ask to report their ratings via telephone at the conclusion of their assessment to a blinded research assistant who would directly enter the data into an electronic data system. 2.4.1 Working alliance Participants completed the 12-item client version of the short form of the Working Alliance Inventory (Horvath and Greenberg, 1989, Tracey and Kokotovic, 1989). This scale includes three 4-item subscales measuring i) bond (e.g. I believe that < name > liked me), ii) task (e.g. < name > and I agreed about the things that I should do to help my situation) and iii) goal (e.g. < name > did not understand what I was trying to accomplish in the interventions (reverse scored item)). The name of the participant's care manager was used to tailor these items (Table 4). Responses were rated on a 7 point scale from 0 = ‘never’ to 6 = ‘always’, with questions 4 and 10 reverse coded. Mean item scores are reported. Higher scores reflect more positive ratings. Reported internal consistency of subscales and total scores ranges from 0.90–0.98 (Hanson et al., 2002, Tracey and Kokotovic, 1989), strong convergent validity with the Agnew Relationship Measure has been reported (Stiles et al., 2002, Duncan et al., 2003). For this study, subscale reliability measured by Cronbach's alpha ranged from 0.78–0.89, and total score reliability was 0.90. 2.4.2 Patient-program alliance Patient-program alliance was measured using an adapted version of the Session Rating Scale (Duncan et al., 2000). This 4-item measure invites the participant to rate their therapeutic experience in terms of the i) relationship (bond), ii) goals and topics (goals), iii) approaches or methods (tasks), and iv) overall. For each item a positive anchor statement was presented with a 7-point response scale running from 0 = ‘absolutely agree’ to 6 = ‘absolutely disagree.’ The adapted version asked participants to rate their experience of the ‘Beating the Blues’ CCBT program (e.g., The computer therapy program worked on and talked about what I wanted to work on or talk about, Table 3). Internal consistency of the original Session Rating Scale has been reported as 0.88 (Duncan, Miller, Sparks & Johnson, 2003). Moderate concurrent validity for this measure with the Helping Alliance Questionnaire II (r = 0.48, p < 0.01) has been reported (Duncan et al., 2003). For this study, reliability measured by Cronbach's alpha was 0.92 for the “Beating the Blues” version. 2.4.3 Overall treatment satisfaction Overall treatment satisfaction was measured using an adapted version of the Session Rating Scale (Duncan et al., 2000; see above). This adapted version asked the patient to rate their overall experience of treatment received through the study intervention (Table 3). For this study, reliability measured by Cronbach's alpha was 0.92 for the Treatment Overall version. 2.4.4 Treatment engagement Treatment engagement was measured by number of log-ins to the “Beating the Blues” program, number of sessions completed, total number of care manager contacts (by email and telephone), and total number of care manager contacts pertaining to “Beating the Blues.” 2.5 Statistical analyses The reliability of Working Alliance Scale, Session Rating Scale for Beating the Blues, and Session Rating Scale for Overall Experience were assessed using Cronbach's Alpha. This was calculated overall as well as for the subscales of the Working Alliance Scale (i.e. Task, Bond, Goal). Based on the results of the parent study (Rollman et al., 2017) we combined the two intervention groups CCBT-alone and CCBT + ISG, as they did not differ on their baseline characteristics, nor did we find any significant differences in clinical outcomes at 6-months between the 2 groups. All further analyses were conducted using treatment measure completers of both intervention groups. Univariate associations between 6-month changes in clinical outcomes and the Beating the Blues Sessions Rating, Overall Treatment Satisfaction, and Working Alliance Inventory scales and subscales were assessed using Spearman correlation coefficients (rs). Survey completers were compared to non-completers on baseline characteristics, levels of CCBT engagement, care manager contacts at 6-months. Two-sample t-test and chi-square tests (or their nonparametric counterparts) were used to determine statistical significance for between-group comparisons of continuous and categorical measures, respectively. Linear mixed models were used to assess the 6-month changes in outcomes (SF-12 MCS, PROMIS Mood, PROMIS Anxiety) between survey completers and non-completers. We included predictors for age group and referral site size, as they were fixed by experimental design, as well as a random effect for subject. Additionally, we adjusted for gender, PHQ-9, and prior treatment due to between-group imbalances. Associations between 6-month changes in clinical outcomes and each of the WAI and Beating the Blues Sessions Rating scale total scores were assessed using linear mixed models with fixed effects for month, Beating the Blues Session Ratings, WAI, and all 2- and 3-way interactions. Of interest was whether there were additive and/or multiplicative effects of both rating scales on outcomes as well as on ‘Beating the Blues’ engagement (# of sessions completed). In addition, associations between 6-month changes in clinical outcomes and CCBT engagement and the Overall Treatment Satisfaction scale total score were examined using linear mixed models with fixed effects for month, Overall Treatment Satisfaction, and their 2-way interaction. All reported estimates, derived from linear combinations of model coefficients, are unstandardized. All analyses were conducted using SAS version 9.4 (SAS Institute, Cary, NC, USA) with a type I error rate of 5% with no adjustment for multiplicity. 3 Results 3.1 Description of treatment measures completers vs. non-completers Of the 603 participants randomized to the CCBT arms of the parent study 65.3% (394) completed the end of treatment questionnaires (treatment measures). They were 77.2% female, 82.0% white, and had a mean age of 43.0 years (SD 14.4 years). As shown in Table 1, participants who completed the treatment measures did not differ in age, level of education, diagnostic status GAD-7 score, and pharmacotherapy use from non-completers at baseline, but they were more likely to be male, less likely to have received treatment for depression in the past year and had a significantly lower baseline PHQ-9 score than the non-completers. During the 6-months intervention period, non-completers withdrew from the study significantly more often than completers (29% vs. 1%).Table 1 Baseline characteristics and engagement of completers and non-completers of the treatment measures. Table 1 Completers (N = 394) Non-completers (N = 209) P Baseline characteristics  Age, mean (SD) 43.0 (14.4) 42.5 (13.8) 0.74  Male, n (%) 90 (22.8) 33 (15.8) 0.04  White, n (%) 323 (82.0) 176 (84.2) 0.49 Education  ≥ High school, n (%) 380 (96.5) 202 (96.7) 0.90  ≥ 4 yr. college degree, n (%) 189 (48.0) 92 (44.0) 0.35 PHQ-9, mean (SD) 12.9 (4.9) 14.0 (5.0) 0.01 GAD-7, mean (SD) 12.9 (4.2) 12.7 (4.7) 0.60 Dx on PRIME-MD  Anxiety only, n (%) 33 (8.4) 12 (5.7) 0.24  Depression only, n (%) 71 (18.0) 39 (18.7) 0.85  Depression and anxiety, n (%) 272 (69.0) 154 (73.7) 0.23 Treatment in past 1 year, n (%) 269 (68.3) 161 (77.0) 0.02 Pharmacotherapy use, n (%) 304 (77.2) 168 (80.4) 0.36 Withdrawn, n (%) 2 (0.5) 61 (29.2) < 0.001 Treatment engagement CCBT program Logged in, n (%) 363 (92.1) 158 (75.6) < 0.001 Completed ≥ 1 session, n (%) 356 (90.4) 148 (70.1) < 0.001 Completed all 8 sessions, n (%) 192 (48.7) 29 (13.9) < 0.001 Care management Total contacts, median [IQR] 15 [13,19] 12 [9,16] < 0.001a Email contacts, median [IQR] 11 [8,14] 9 [5,12] < 0.001a Telephone contacts, median [IQR] 4 [3,6] 3 [2,5] < 0.001a Contacts pertaining to CCBT program, median [IQR] 7 [4,8] 2 [3,6] < 0.001a Abbreviations: CCBT, computerized cognitive behavioural therapy; GAD-7, 7-item Generalized Anxiety Disorder Scale; IQR, interquartile range; PRIME-MD, Primary Care Evaluation of Mental Disorders; PHQ-9, 9-item Patient Health Questionnaire. a Note: p-values for Contacts were obtained using Wilcoxon rank-sum tests. Compared to those who did not complete the treatment measures, participants who completed them were significantly more likely to have logged on to the CCBT program (92.1%), completed at least one session (90.4%), and were more likely to have completed all 8 sessions within the 6-months intervention period (48.7%). Completers had more care manager contacts (median 15, IQR 13–19), and more pertaining to the CCBT program (7, IQR 4–8) (Table 1). Participants who completed the treatment measures also reported significantly higher improvements in all clinical outcomes at the end of the 6-months intervention compared to those who did not complete the surveys (Table 2). Full analysis of clinical outcomes is reported in Rollman et al., 2017.Table 2 Improvement of clinical outcomes of treatment measures completers vs. non-completers. Table 2Estimated 6-month change Completers (N = 394) Non-completers (N = 209) Completers vs. non-completers mean difference [95% CI] P SF-12 MCS, mean (SE) 12.90 (0.55) 9.81 (0.91) 3.09 [1.00, 5.18] 0.004 PROMIS Mood, mean (SE) − 9.14 (0.38) − 7.22 (0.53) − 1.92 [− 3.35, − 0.48] 0.009 PROMIS Anxiety, mean (SE) − 9.76 (0.41) − 6.64 (0.67) − 3.12 [− 4.68, − 1.57] < 0.001 Abbreviations: PROMIS Anxiety, Patient-Reported Outcomes Measurement Information System for Anxiety (fixed length, short form); PROMIS Depression, Patient-Reported Outcomes Measurement Information System for Depression (fixed length, short form), SF-12 MCS, Medical Outcomes Study Short Form Mental Component Scale; 3.2 Treatment measures, patient engagement and clinical outcomes All subsequent analyses in this paper refer to participants who completed the treatment measures only. Survey completers reported a positive working alliance with their care manager on all subscales and overall (mean = 4.9, SD 1.0). Ratings of both the CCBT program and overall treatment satisfaction also indicated positive experiences in survey completers (Table 3).Table 3 Treatment Experience Ratings. Table 3 Survey completers (N = 394) Work Alliance Inventorya  a) Task scale, mean (SD) 4.8 (1.2)  b) Bond scale, mean (SD) 5.0 (1.1)  c) Goal scale, mean (SD) 5.0 (1.1)  d) Total working alliance scale, mean (SD) 4.9 (1.0) BtB Sessions Rating and Overall Satisfactionb a) I felt heard, understood and respected by the computer therapy program, mean (SD) 4.6 (1.5); n = 385 b) The computer program worked on and talked about what I wanted, mean (SD) 4.5 (1.4); n = 385 c) The computer therapy approach is a good fit for me, mean (SD) 4.0 (1.9); n = 385 d) Overall, the computer therapy program was right for me, mean (SD) 4.0 (1.9); n = 384 Sessions Rating BtB Total score (a-d), mean (SD) 17.1 (5.9); n = 384c e) I felt heard, understood and respected by the study intervention, mean (SD) 5.0 (1.2); n = 385 f) We worked on and talked about what I wanted, mean (SD) 4.8 (1.3); n = 384 g) The therapy approach is a good fit for me, mean (SD) 4.4 (1.7); n = 385 h) Overall, the study intervention was right for me, mean (SD) 4.4 (1.7); n = 385 Overall treatment satisfaction score (e-h), mean (SD) 18.6 (5.3); n = 384c Abbreviation: BtB, Beating the Blues. a All Working Alliance questions are coded here as: 0 = Never … 6 = Always. b All Sessions Rating questions are coded here as: 0 = Absolutely Disagree … 6 = Absolutely Agree. c When comparing the mean session ratings survey completers to a midpoint score of 12 using a one-sample t-test, the p-value < 0.0001 (for both a-d total score and e-h total score). 3.2.1 Treatment experience and engagement with CCBT program The number of ‘Beating the Blues’ sessions completed was significantly associated with the Working Alliance Inventory (WAI) total score (rs = 0.18 p ≤ 0.001), as well as all WAI subscale scores (Bond rs = 0.11, p = 0.03; Task rs = 0.17 p = 0.001; Goal rs = 0.18, p < 0.001). Number of sessions completed was also significantly associated with Beating the Blues Sessions Rating total score (rs = 0.32, p < 0.0001), and the Overall Treatment Satisfaction score (rs = 0.23, p < 0.0001). As presented in Table 4, increases in WAI total score were associated with a 0.45 (CI 0.24, 0.66; p < 0.001) point increase in sessions completed. Similarly, Beating the Blues Sessions Rating and Overall Treatment Satisfaction ratings showed increases in sessions completed, 0.55 (CI 0.41, 0.70; p < 0.001) and 0.36 (CI 0.20, 0.53; p < 0.001), respectively. However, in the main effects model of working alliance and program alliance, only the Beating the Blues Sessions Rating was significantly associated with CCBT sessions completed (Table 4, Model 3). Furthermore, no significant overall interaction effect was found (Table 4, Model 4).Table 4 Effects of treatment measures on 6-month changes in CCBT engagement and clinical outcomes. Table 4Engagement measure: 6-month change in number of BtB sessions completed Model Effecta Estimate (SE) (95% CI) P-value (1) WAI total 0.45 (0.11) (0.24, 0.66) < 0.001 (2) BTB session total 0.55 (0.07) (0.41, 0.70) < 0.001 (3) BTB session total 0.53 (0.08) (0.36, 0.69) < 0.001 (3) WAI total 0.08 (0.12) (− 0.16, 0.32) 0.523 (4) BTB session total 0.01 (0.32) (− 0.61, 0.63) 0.976 (4) WAI total − 0.32 (0.27) (− 0.85, 0.20) 0.224 (4) BTB * WAI interaction 0.11 (0.06) (− 0.02, 0.24) 0.090 (5) Overall treatment satisfaction 0.36 (0.08) (0.20, 0.53) < 0.001 Outcome: 6-month change in SF-12 MCS (1) WAI total 2.22 (0.54) (1.15, 3.28) < 0.001 (2) BTB session total 1.69 (0.37) (0.96, 2.42) < 0.001 (3) BTB session total 1.14 (0.43) (0.31, 1.98) 0.007 (3) WAI total 1.64 (0.63) (0.40, 2.87) 0.009 (4) BTB session total 0.46 (1.61) (− 2.70, 3.62) 0.776 (4) WAI total 1.09 (1.37) (− 1.60, 3.79) 0.427 (4) BTB * WAI interaction 0.15 (0.33) (− 0.50, 0.79) 0.659 (5) Overall treatment satisfaction 1.84 (0.42) (1.02, 2.66) < 0.001 Outcome: 6-month change in PROMIS depression (1) WAI Total − 2.08 (0.38) (− 2.81, − 1.34) < 0.001 (2) BTB session total − 1.49 (0.26) (− 2.00, − 0.98) < 0.001 (3) BTB session total − 0.99 (0.30) (− 1.57, − 0.41) 0.001 (3) WAI total − 1.49 (0.44) (− 2.35, − 0.64) 0.001 (4) BTB session total 0.02 (1.11) (− 2.17, 2.20) 0.987 (4) WAI total − 0.71 (0.95) (− 2.57, 1.16) 0.459 (4) BTB * WAI interaction − 0.21 (0.23) (− 0.66, 0.23) 0.350 (5) Overall Treatment Satisfaction − 1.53 (0.29) (− 2.10, − 0.95) < 0.001 Outcome: 6-month change in PROMIS anxiety (1) WAI total − 2.22 (0.41) (− 3.02, − 1.42) < 0.001 (2) BTB session total − 1.28 (0.28) (− 1.83, − 0.72) < 0.001 (3) BTB session total − 0.65 (0.32) (− 1.28, − 0.02) 0.043 (3) WAI total − 1.86 (0.47) (− 2.79, − 0.93) < 0.001 (4) BTB session total − 0.34 (1.21) (− 2.72, 2.04) 0.781 (4) WAI Total − 1.61 (1.04) (− 3.65, 0.42) 0.120 (4) BTB * WAI interaction − 0.07 (0.25) (− 0.55, 0.42) 0.789 (5) Overall treatment satisfaction − 1.60 (0.32) (− 2.21, − 0.98) < 0.001 Abbreviations: BtB, Beating the Blues; PROMIS Anxiety, Patient-Reported Outcomes Measurement Information System for Anxiety (fixed length, short form); PROMIS Depression, Patient-Reported Outcomes Measurement Information System for Depression (fixed length, short form); SF-12 MCS, Medical Outcomes Study Short Form Mental Component Scale; WAI, Working Alliance Inventory. a All estimates are derived from 6-month contrasts in the interaction between the listed Effect and Time. 3.2.2 Treatment experience and clinical outcomes Working Alliance Inventory total scores were significantly associated with symptom improvements at 6 months (depression, rs = 0.25, p < 0.0001; anxiety, rs = 0.25, p < 0.0001; SF-12 MCS, rs = 0.19, p < 0.001). Beating the Blues Session Ratings were significantly associated with symptom improvements at 6 months (depression, rs = 0.28, p < 0.0001; anxiety, rs = 0.26, p < 0.0001; SF-12 MCS, rs = 0.22, p < 0.001). Overall Treatment Satisfaction ratings were significantly associated with symptom improvements at 6 months (depression, rs = 0.25, p < 0.0001; anxiety, rs = 0.26, p < 0.0001; SF-12 MCS, rs = 0.21, p < 0.001). Increases in WAI total score were associated with a 2.22 (CI 1.15, 3.28; p < 0.001) point increase in SF-12 MCS improvement, and with a significant point decrease in mood (− 2.08; CI − 2.80, − 1.34; p < 0.001), and anxiety symptoms (− 2.22; CI − 3.02, − 1.42; p < 0.001). Similarly, increases in the Beating the Blues Sessions Ratings and Overall Treatment Satisfaction score were related with a significant SF-12 MCS improvement (1.69; and 1.84 p < 0.001), and a mood (− 1.49; and − 1.53; p < 0.001) and anxiety symptom (− 1.28; and − 1.60; p < 0.001) decease. For all three clinical outcomes measures, there were significant main effects of WAI and Beating the Blues Sessions Rating scores (Table 4, Model 3). We did not find any significant interaction effects for any of the clinical outcomes (Table 4, Model 4). 4 Discussion This study appears to be the first to directly measure the patient-rated therapeutic alliance, patient-program alliance and overall treatment satisfaction for study participants with anxiety and/or depression accessing a CCBT program, in the context of a 6-month collaborative care strategy. With > 600 participants randomized to receive CCBT and around two-thirds completing the end-of-intervention treatment experience measures it is also sufficiently powered to test our hypotheses about the association of treatment experience with participant engagement and clinical outcomes. Participants who completed the treatment experience measures were more likely to be male, have received a recent mental health treatment, and reported fewer depression symptoms at baseline. Not surprisingly, participants who did not complete the measures showed less engagement with the CCBT program or even withdrew from the study, had fewer interactions with the care managers, and reported worse clinical outcomes at 6-months. Interestingly, survey non-completers showed similar modest improvements as participants randomized to the ‘usual care’ group in the parent trial, although the majority of survey non-completers engaged in at least one CCBT session and they had numerous contacts with their care managers (Rollman et al., 2017). Based on our results we cannot discern whether participants who do not engage with a guided CCBT program may have disengaged because they did not agree with the treatment package or they felt that they had received enough care. 4.1 Relationship triangle Participants' ratings of their working alliance with the care manager, who supported their use of the CCBT program, indicated a positive experience of alliance. These findings echo previous research supporting the idea that a positive working alliance can be established despite the rather limited contact associated with supporting a CCBT program (e.g. Andersson et al., 2012, Kiropoulos et al., 2008). Ratings across task, bond and goal sub-scales were similar, contrasting with Berger's (2015) hypothesis that bond subscale ratings are necessarily lower for interventions using CCBT. In general, participants also rated positively the relationship, goals, tasks and overall experience with the CCBT program itself. They felt heard, understood and respected by the CCBT program, that the program ‘talked about’ what they wanted to, and was a good fit to their needs. This accords with previous research reporting positive patient-program relationships in the context of CCBT (e.g. Nordgren et al., 2013, Ormrod et al., 2010). Participants rated the overall treatment approach somewhat higher than their experience with the CCBT program alone, (18.6 vs. 17.1, Table 3) suggesting that interactions with the care-manager and additional components of the collaborative care intervention enhanced participants experience of their treatment overall. 4.2 Engagement We found statistically significant relationships between the number of CCBT sessions completed and measures of working alliance, patient-program alliance and overall treatment experience. This dose-response effect echoes previous research suggesting that patient-provider relationship is associated with mental health service engagement outcomes more broadly (Marsh et al., 2012, Thompson and McCabe, 2012). Furthermore, the patient-program alliance may be more closely associated with CCBT engagement than the patient-provider alliance. This supports previous commentary (Berger, 2015). However, the cross-sectional nature of this data means that it is not possible to make causal inferences about the direction of this relationship. Longitudinal research is needed to unpick the sequence of events. 4.3 Clinical outcomes We found statistically significant relationships between each treatment measure and improvements on our clinical outcome measures (SF-12 MCS, PROMIS depression and anxiety). Correlations between alliance and outcome measures were in the small-medium range and similar to those found for the alliance-outcome relationship in traditional psychotherapy studies (mean rs = 0.275, 95% CI 0.25–0.30; Horvath et al., 2011). This supports the idea that both a positive patient-provider and patient-program alliance may be typical of CCBT interventions and contribute to clinical improvements, but remain undetected in some smaller studies (Berger, 2015). Linear mixed model analysis indicated that patient-provider and patient-program alliance ratings may contribute independently and additively rather than interactively to clinical outcomes and program engagement. 4.4 Study limitations Only two thirds of participants randomized to the CCBT arms of the parent trial completed the treatment measures reported in this manuscript, had lower depression baseline scores, and were more likely to be generally more engaged with the trial's interventions than those who did not complete the survey. Given these differences between completers and non-completers caution in the interpretation of all further analysis is advised. In particular, it is possible that non-completers felt a poorer alliance and were less satisfied with the care package than completers. Second, treatment measures were collected at a single time-point at the end of the 6-months intervention which may inflate the correlation between the alliance and outcomes and limit confidence in our interpretation of these variables as predictors of engagement or outcome. Further research using a longitudinal design with multiple measurement points for treatment experience is needed to test the limits of our findings. Third, the study relied on adaptations of measures developed to capture relational aspects of ‘psychotherapy’ (WAI and SRS). As the nature of the relationship between a patient and computer program or remote care manager may be expected to differ from that of a traditional therapeutic relationship, research is needed to further understand and measure these constructs. As with any study measuring the therapeutic relationship in the context of an effective psychological intervention, a halo effect on measurement is possible. Finally, we combined the intervention group that had access to both the CCBT program and the ISG with the group who had access to the CCBT program alone. Although these two groups did not differ in baseline characteristics, program engagement, and improvements in clinical outcomes (Rollman et al., 2017), the availability of the ISG may have influenced the participants' working alliance with the care manager that may affect the generalizibility of our findings. 4.5 Future research implications This study highlights the need for future studies to be designed to adequately test questions relating to patient-professional and patient-program alliance. Like alliance-outcome relationships in traditional face-to-face psychotherapies, the association may be consistent but moderate in magnitude. Small N studies are unlikely to be sufficiently robust to test questions in this domain. Longitudinal research is required to untangle the relationship between patient-provider and patient-program relationships, engagement and outcomes. It may be the case that fuller engagement with the program leads to an improved therapeutic experience, or that a better therapeutic experience leads to greater engagement with the program or an interaction of these two effects best explains the patient journey. Finally, adding provider perspectives on the therapeutic/working alliance could augment future studies. 4.6 Clinical implications As patient-program alliance appears to be at least moderately associated with engagement and clinical outcomes for guided CCBT programs, ensuring that these programs are optimized to elicit a positive relational experience by tailoring them to a patient's specific needs (e.g., sociodemographic characteristics, medical conditions, cultural and linguistic preferences) that maximizes felt collaboration in terms of goals, tasks and bond may help to ensure the greatest patient benefit from these interventions. Furthermore, since the patient-provider alliance appears to be similar to face-to-face psychotherapy, the training and supervision of support staff to enhance alliance and experience may lead to improved engagement and outcomes. 5 Conclusions This study aimed to overcome some of the methodological limitations of previous research and extend our understanding of alliance-outcome relationships in e-mental health by exploring the nature of the relationship triangle between the patient, their care manager and their CCBT program. We found a positive participant-rated alliance with both their care manager and the CCBT program itself and these were each significantly associated with measures of engagement and clinical outcome. The magnitude of this association was moderate, and within the range of that reported for traditional face-to-face psychotherapies. These findings add importantly to the knowledge base in this research domain, and set the ground work for future research and practice. Further research is needed to ensure optimal service provision in e-mental health that acknowledges the role of both patient-professional but also patient-program alliance. Funding This work was supported by NIH grant R01 MH093501 from the National Institute of Mental Health. Declaration of interest The first author (KC) has acted as a research consultant to Ultrasis plc, the company that owned and marketed the British version of Beating the Blues at the time this study commenced. All work described was supported by a grant from the National Institute of Mental Health (R01 MH093501). The funding source had no role in the design, conduct, or reporting of our study or in the preparation, review, or decision to submit this manuscript for publication. ==== Refs References Andersson G. Using the internet to provide cognitive behaviour therapy Behav. Res. Ther. 47 3 2009 175 180 19230862 Andersson G. Internet-delivered psychological treatments Annu. Rev. Clin. Psychol. 12 2016 157 179 26652054 Andersson G. Paxling B. Wiwe M. Vernmark K. Felix C.B. Lundborg L. Furmark T. Cuijpers P. Carlbring P. Therapeutic alliance in guided internet-delivered cognitive behavioural treatment of depression, generalized anxiety disorder and social anxiety disorder Behav. Res. Ther. 50 9 2012 544 550 22728647 Andersson G. Cuijpers P. Carlbring P. Riper H. Hedman E. Guided internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis World Psychiatry 13 3 2014 288 295 25273302 Andrews G. Cuijpers P. Craske M.G. McEvoy P. Titov N. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis PLoS One 5 10 2010 e13196 Barazzone N. Cavanagh K. Richards D.A. Computerized cognitive behavioural therapy and the therapeutic alliance: a qualitative enquiry Br. J. Clin. Psychol. 51 4 2012 396 417 23078210 Berger T. The therapeutic alliance in internet interventions: a narrative review and suggestions for future research Psychother. Res. 2015 1 14 Berger T. Boettcher J. Caspar F. Internet-based guided self-help for several anxiety disorders: a randomized controlled trial comparing a tailored with a standardized disorder-specific approach Psychotherapy 51 2 2014 207 24041199 Bordin E.S. The generalizability of the psychoanalytic concept of the working alliance Psychother. Theory Res. Pract. 8 3 1979 252 Cavanagh K. Turn on, tune in and (don't) drop out: engagement, adherence, attrition, and alliance with internet-based interventions Oxford Guide to Low Intensity CBT Interventions 2010 227 233 Cavanagh K. Millings A. (Inter) personal computing: the role of the therapeutic relationship in e-mental health J. Contemp. Psychother. 43 4 2013 197 206 Duncan B.L. Miller S.D. Johnson L.D. The Session Rating Scale 3.0 2000 Authors Chicago, IL Duncan B.L. Miller S.D. Sparks J.A. Claud D.A. Reynolds L.R. Brown J. Johnson L.D. The session rating scale: preliminary psychometric properties of a “working” alliance measure Journal of Brief Therapy 3 1 2003 3 12 Faul F. Erdfelder E. Buchner A. Lang A.-G. Statistical power analyses using G*power 3.1: tests for correlation and regression analyses Behav. Res. Methods 41 2009 1149 1160 19897823 Grist R. Cavanagh K. Computerised cognitive behavioural therapy for common mental health disorders, what works, for whom under what circumstances? A systematic review and meta-analysis J. Contemp. Psychother. 43 4 2013 243 251 Hanson W.E. Curry K.T. Bandalos D.L. Reliability generalization of working alliance inventory scale scores Educ. Psychol. Meas. 62 4 2002 659 673 Herbeck Belnap B. Schulberg H.C. He F. Mazumdar S. Reynolds C.F. 3rd Rollman B.L. Electronic protocol for suicide risk management in research participants J. Psychosom. Res. 4 2015 340 345 Horvath A.O. Greenberg L.S. Development and validation of the working alliance inventory J. Couns. Psychol. 36 1989 223 233 Horvath A.O. Del Re A.C. Flückiger C. Symonds D. Alliance in individual psychotherapy Psychotherapy 48 1 2011 9 16 21401269 Kiropoulos L.A. Klein B. Austin D.W. Gilson K. Pier C. Mitchell J. Ciechomski L. Is internet-based CBT for panic disorder and agoraphobia as effective as face-to-face CBT? J. Anxiety Disord. 22 8 2008 1273 1284 18289829 Kroenke K. Spitzer R.L. Williams J.B.W. The PHQ-9: validity of a brief depression severity measure J. Gen. Intern. Med. 16 9 2001 606 613 11556941 Lambert M.J. Barley D.E. Research summary on the therapeutic relationship and psychotherapy outcome Psychother. Theory Res. Pract. Train. 38 4 2001 357 Lopez A. An investigation of the use of internet based resources in support of the therapeutic alliance Clin. Soc. Work. J. 43 2 2015 189 200 Marsh J.C. Angell B. Andrews C.M. Curry A. Client-provider relationship and treatment outcome: a systematic review of substance abuse, child welfare, and mental health services research Journal of the Society for Social Work and Research 3 4 2012 233 267 Meyer B. Bierbrodt J. Schröder J. Berger T. Beevers C.G. Weiss M. Jacob G. Späth C. Andersson G. Lutz W. Hautzinger M. Effects of an internet intervention (Deprexis) on severe depression symptoms: randomized controlled trial Internet Interventions 2 1 2015 48 59 Newby J.M. Twomey C. Yuan Li S.S. Andrews G. Transdiagnostic computerised cognitive behavioural therapy for depression and anxiety: a systematic review and meta-analysis J Affect. Disord. 15 199 2016 30 41 Newman M.G. Szkodny L.E. Llera S.J. Przeworski A. A review of technology-assisted self-help and minimal contact therapies for anxiety and depression: is human contact necessary for therapeutic efficacy? Clin. Psychol. Rev. 31 1 2011 89 103 21130939 Nordgren L.B. Carlbring P. Linna E. Andersson G. Role of the working alliance on treatment outcome in tailored internet-based cognitive behavioural therapy for anxiety disorders: randomized controlled pilot trial JMIR Research Protocols 2 1 2013 e4 23612437 Ormrod J.A. Kennedy L. Scott J. Cavanagh K. Computerised cognitive behavioural therapy in an adult mental health service: a pilot study of outcomes and alliance Cogn. Behav. Ther. 39 3 2010 188 192 20485996 Pilkonis P.A. Choi S.W. Reise S.P. Stover A.M. Riley W.T. Cella D. Item banks for measuring emotional distress from the patient-reported outcomes measurement information system (PROMIS®): depression, anxiety, and anger Assessment 18 2011 263 283 21697139 Preschl B. Maercker A. Wagner B. The working alliance in a randomized controlled trial comparing online with face-to-face cognitive-behavioral therapy for depression BMC Psychiatry 11 1 2011 189 22145768 Proudfoot J. Ryden C. Everitt B. Shapiro D.A. Goldberg D. Mann A. Tylee A. Marks I. Gray J.A. Clinical efficacy of computerised cognitive–behavioural therapy for anxiety and depression in primary care: randomised controlled trial Br. J. Psychiatry 185 1 2004 46 54 15231555 Richards D. Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis Clin. Psychol. Rev. 32 4 2012 329 342 22466510 Rollman B.L. Herbeck Belnap B. Reynolds C.F. Schulberg H.C. Shear M.K. A contemporary protocol for the treatment of panic and generalized anxiety in primary care Gen. Hosp. Psychiatry 25 2003 74 82 12676419 Rollman B.L. Fischer G.S. Zhu F. Herbeck Belnap B. Comparison of electronic physician prompts versus waitroom case-finding on clinical trial enrollment J. Gen. Intern. Med. 23 2008 447 450 18373143 Rollman B.L. Herbeck Belnap B. Abebe K.Z. Online collaborative care for treating mood and anxiety disorders in primary care: a randomized controlled trial JAMA Psychiat. 2017 (published online November 8, 2017) Spitzer R.L. Williams J.B.W. Kroenke K. Utility of a new procedure for diagnosing mental disorders in primary care: the PRIME-MD 1000 study J. Am. Med. Assoc. 272 1994 1749 1756 Spitzer R.L. Kroenke K. Williams J.B. Lowe B. A brief measure for assessing generalized anxiety disorder: the GAD-7 Arch. Intern. Med. 166 10 2006 1092 1097 16717171 Stiles W.B. Agnew-Davis R. Barkham M. Shapiro D.A. Convergent validy of the Agnew relationship measure and the working alliance inventory Psychol. Assess. 14 2 2002 209 220 12056083 Thompson L. McCabe R. The effect of clinician-patient alliance and communication on treatment adherence in mental health care: a systematic review BMC Psychiatry 12 1 2012 87 22828119 Titov N. Internet-delivered psychotherapy for depression in adults Curr. Opin. Psychiatry 24 1 2011 18 23 20827199 Tracey T.J. Kokotovic A.M. Factor structure of the working alliance inventory Psychological Assessment: A Journal of Consulting and Clinical Psychology 1 3 1989 207 Wagner B. Schulz W. Knaevelsrud C. Efficacy of an internet-based intervention for posttraumatic stress disorder in Iraq: a pilot study Psychiatry Res. 195 1 2012 85 88 21813187 Ware J.E. Kosinski M. Keller S.D. SF:12 How to Score the SF-12 Physical and Mental Health Summary Scales Second ed 1995 The Health Institute, New England Medical Center Boston, MA