(PS5-B50) Development of a Vignette-based Measure of Practice Element and Treatment Target Knowledge for Training and Implementation in Youth Public Mental Health Services

Research in therapist training has highlighted the need for psychometrically-validated measures with an emphasis on therapist knowledge (Muse & McManus, 2013; McHugh & Barlow, 2010). Previous reviews of therapist knowledge indicate varying degrees from awareness (e.g., therapist knows exposure works) to procedural (e.g., therapist knows how to effectively deliver exposure), which are impacted through active learning strategies (Beidas & Kendall, 2010). Validated therapist knowledge measures can aid in developing standardized training efforts across the behavioral health landscape, where there is a critical need for reliable and pragmatic measures (Frank et al., 2020; Scott et al., 2016). Vignette-based knowledge measures can bridge the gap between traditional surveys and observational methods without the limitations associated with other methods (e.g., time intensiveness) and by offering both high internal validity and external validity to real life behavior (Evans et al., 2015). The aim of the current study is to develop a measure to ensure accurate practice reporting by therapists in public mental health services and explore the initial validity of the measure.
In a large statewide youth public mental health system, therapists report treatment targets and practice elements used after each therapy session as part of routine treatment progress monitoring (Child and Adolescent Mental Health Division, 2019). This monitoring approach was developed utilizing the distillation and matching model proposed by Chorpita, Daleiden, & Weisz (2005). Fifty vignettes were developed by five psychologists to examine therapists’ knowledge of treatment targets and associated practice elements. These vignettes reflect youth receiving services with the Hawaiʻi State Child and Adolescent Mental Health Division (CAMHD), and were refined for culturally relevant language, including regional colloquialisms. Content validation was completed using five expert raters with extensive experience coding treatment targets and practice elements. Raters were mostly female (80%, n = 4) and all held PhDs in clinical psychology (100%, n = 5). Raters indicated on a five-point Likert scale (1 = strongly disagree to 5 = strongly agree) the extent to which vignettes accurately described the treatment targets and practice elements. Intraclass correlations revealed that treatment targets were in the fair range (ICC = .57) and practice elements in the poor range (ICC = .39). Additionally, seven CAMHD staff served as pilot participants who provided feedback on the ease of answering vignette questions, their extent of referencing the codebook while completing the vignettes, and completion time. The pilot group was primarily female (85.7%, n = 6) and most held doctoral degrees in psychology (85.7%, n = 6). These results helped to inform further refinement to the vignettes prior to larger pilot administration. Results from these preliminary analyses and further measure development strengthens CAMHD’s ability to track youth progress. Additionally, our study integrates cognitive and behavioral science, public mental health, and culturally-appropriate language to create a pragmatic therapist knowledge measure.