Symposia
Dissemination & Implementation Science
Michael Tanana, Ph.D
Research Assistant Professor
University of Utah
Salt Lake City, Utah
Cognitive Behavior Therapy (CBT) is one of the most broadly studied and practiced evidence-based treatments used in psychotherapy today. While significant resources have been invested in moving CBT into routine practice, little is known about the quality of CBT delivered in community settings - in significant part due to the lack of scalable tools for assessing CBT fidelity. Gold-standard tools for assessing the fidelity of CBT in research settings, such as the Cognitive Therapy Rating Scale (CTRS), require trained expert raters, take hours to complete on every session, and are prohibitively expensive. In practice, this means that only a narrow sliver of CBT sessions are ever rated for fidelity. Fortunately, Natural Language Processing (NLP) has advanced dramatically over the past decade. Now, machine learning models can accurately answer natural language questions, rate interpersonal skills, and predict therapist empathy with similar accuracy to human reliability. This presentation explores the use of NLP models to predict CBT competence in a dataset collected as part of a large-scale CBT implementation program in a public mental health system. In this work, we train a machine learning model that is designed to rate the CTRS from just a simple transcript of a session. The model was first pre-trained on billions of words of written English, to develop a computational model of words and phrases. After this pre-training step, it was then fine-tuned on 220 sessions rated by expert human coders on the CTRS. We used a novel NLP method that combines a low-level deep learning transformer model for local context, and a higher level set of neural network layers that combine this local context into a session level prediction for CTRS codes. On a test set (i.e. a set of sessions not used in model development) our model can predict 9 of the 11 CTRS subscales as well as the total score at a level of performance that is indistinguishable from human reliability (Model Spearman’s Rho=.78, p< .001, Human Interrater Rho=.76). In this presentation we will discuss the deep learning techniques used to create the model, the machine learning methodology used to assess the model performance as well as the implications for the practice of evidence based treatments. More broadly, we will discuss the implementation of the CBT model within software to provide automated fidelity feedback at scale.