Assessing Assessments: Are Your Questions Any Good?

A 2020 publication sought to answer: What is the benchmark for effectiveness in online CME?
Using effect size data from 40 accredited, Internet-based enduring materials produced between 2016 and 2018, it was suggested that a Cohen's d between 0.48 and 0.75 serve as preliminary benchmarks for "good" CME (results published here: J Eur CME 2020;9[1]:1832796).

Ideally, this report will encourage others to publish their CME effect size data - and collectively, produce more robust benchmarks. That may still happen, but in the interim, another group published (in the same journal) four criteria by which pre/post, multiple-choice questions (MCQs) can be quantitatively assessed for validity (ie, how well they measure what they intend to measure). Considering that the preliminary Cohen's d benchmarks referenced above were derived from pre/post MCQs, we had to wonder...what if the questions were not any good? How might our question's performance against these four validity criteria affect effect size calculations?

In this session, we will detail these four validity criteria and demonstrate how to apply them to pre/post MCQs using current data. How MCQ validity may impact our understanding of CME effectiveness will be discussed.

Participating learners will gain the following from this session:

1. Clarity of the strengths and limitations of effect size (ie, Cohen's d) in determining CME effectiveness


2. Quantitative tools for evaluating the quality of pre/post MCQs


3. Best practices for writing better MCQs

Innovation: The quality of our assessment tools matter. Outcomes data derived from insufficiently vetted pre/post, multiple-choice questions (MCQs) can lead to faulty conclusions about an activity's effectiveness. Such gaps in understanding reduce providers' ability to appropriately refine their CME practices, However, an understanding of available tools to improve MCQ quality can lead to more robust outcomes data, and correspondingly, encourage better alignment between the education produced and the needs of clinician learners.