Digital Innovations for Program Assessment

Overview: An innovative and anti-oppressive programmatic data collection method, utilizing the Canvas outcomes tool and embedded measures, will be presented. Discussion will demonstrate how resulting data can be utilized to inform student competency development and areas for explicit and implicit program renewal.Proposal text:

Background: Curriculum-embedded assessments (CEA) are an effective method to evaluate student learning outcomes (Gerretson & Golson, 2005). A literature review examined assessment practices in 116 social work programs nationwide and found that following field evaluations (90.8%), CEA is the second most commonly used instrument (58.8%) (Sellers & Neff, 2019). However, identified barriers for this method include time to administer, or the faculty/administrative burden of designing appropriate measures (Sellers & Neff, 2019). A digitally innovative outcomes project has been developed and implemented that addresses the challenges the literature highlights regarding CEA. The digital outcomes project is a programmatic data collection method, utilizing the Canvas Outcomes Tool (COT) to create multi-dimensional and integrated CEA. In partnership with the known advantage of CEA (Cummings et al., 2008), COT technology-enhanced assessment provides a comprehensive way to evaluate student progress (Kuh et al., 2015). By implementing the COT program-wide students' competency development is assessed each semester and across coursework and field education. As a result, a rich set of data allows for programmatic assessment of individual- and aggregate-student performance against program benchmarks for each competency. The resulting data is used to inform ongoing student competency development, retention, and explicit and implicit program enhancements. The COT supports a data-driven culture, that utilizes ongoing and real-time feedback towards positive change.

The purpose of this presentation is to provide an illustrative guide for leveraging the Canvas e-Learning System to create multi-dimensional and integrated curriculum-embedded assessments that support data-driven conversations for both program enhancements and student support needs. Our specific aims include: a) illustrating the digital assessment framework utilized (b) discussing how resulting data informs student support and retention and informs program renewal, and c) sharing the assessment framework for replication and customization by other teams.

Methods: Development of COT corresponded with a curriculum revision and the creation of a common assignment structure to implement embedded assessments and strengthen alignment between accreditation standards, program-level objectives, and course-level objectives (Bracy, 2018).The COT was piloted in 2018 and has been implemented programmatically since Fall 2018 (on-campus and online). 23 faculty were trained to implement the COT, which is an essential component of effective assessment (Cummings, Maddux, & Richmond, 2008; Lawson et al., 2015). Across the curriculum and in field education, all nine 2015 EPAS competencies were assessed utilizing a standardized five-point Likert scale [“1= Inadequate Competence” and “5 =Exceptional Competence”]. A detailed rubric for generalist and advanced generalist courses was developed to guide faculty assessment.

Data Collection: From Fall 2018 - Fall 2021, approximately 3,500 CEA items were collected towards competency assessment of 314 MSW students. Data analysis yields individual- and aggregate-student performance data against program benchmarks for each competency and across program offerings. Based on the data, explicit and implicit curricular enhancements were recommended as well as student support needs.

Implications: In conclusion, exploring digital innovations for programmatic assessments in social work education may offer layered benefits towards data-driven program enhancements and early identification of student support needs.