Measuring Student's Competencies: Identifying Sources of Unreliability and How to Reduce Them

Overview: Differences in the methods and assessor of the competencies can contribute to the reliability in the measure. In this workshop, participants will learn how to use Generalizability Theory to identify the multiple factors contributing to the reliability of a competency measure.Proposal text: Assessment is an integral component of competency-based education. At least two measures assess each competency (EP 4.0 Assessment of Student Learning Outcomes). In some cases, students’ competencies are assessed across different classes using different methods. Differences in the methods and assessor of the competencies can contribute to the reliability in the measure. Reliability refers to the degree to which the result of a measurement, calculation, or specification can be depended on to be accurate. Reliability Coefficients based on classical test theory (i.e., Cronbach’s alpha, Test-retest, and Inter-raters) does not allow for the simultaneous estimation of the multiple contributors to the reliability of a measure (i.e., method, assessor, etc.). It also does not allow for the examination of the interaction among them. In this workshop, participants will learn how to use Generalizability Theory to identify the multiple factors contributing to the reliability of a competency measure.

Key concepts and procedures will be illustrated using to assessment situation: 1) Students’ competency are assessed across different classes using different methods. 2) Multiple teachers providing an assessment of a competency using the same method.

Learning Objectives:
1. Participants will learn how to use existing data at their school to assess those factors that contribute to the reliability of a measure using Generalizability Theory.
2. Participants will learn how to reduce the contribution of the method and rater on the reliability of a measure.