1138 - Works Well Enough? Program Directors’ Perceptions of the Effectiveness and Transparency of Competency-Based Evaluations in Assessing Infectious Diseases Fellow Performance

Background: In July 2015, the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Internal Medicine (ABIM) jointly outlined an approach to assessing fellow performance using milestone-based core competencies for incorporation into standardized evaluation templates of trainee performance. Limited data exist regarding the clarity, effectiveness, and reproducibility of competency-based evaluations of infectious diseases fellows.

Methods: From March to May 2019, program directors of ACGME-accredited infectious diseases fellowship programs were invited to complete a Qualtrics-based survey of program characteristics and evaluation methods, including a trainee vignette to gauge evaluation reproducibility. Completed surveys were analyzed with descriptive statistics.

Results: Forty-three program directors initiated the survey, but 29 completed it. Seventeen (59%) were men, 19 (66%) were on a teaching service for over 8 weeks a year, and 19 (66%) had fewer than four first year fellows in their program. Most respondents agreed the competencies lacking the most clarity were systems-based practice (17/29, 58%), and practice based improvement (16/29, 55%). Eighteen (62%) were at least “somewhat satisfied” with their institution’s assessment tool, and 19 (66%) reported it was at least “moderately effective” in identifying academic deficiencies. Responses rating fellow performance from the vignette ranged from 1.5 to 4 on the standard milestone-based competency scale of 1-5 with 0.5 increments (median 3). For the same scenario using a qualitative ordinal scale, 66% (19/29) categorized the fellow as “early first year” and 34% (9/29) as “advanced first year.” Respondents offered a wide range of comments on milestone-based competencies, including “it works well enough” and “the process seems bloated and educratic.”

Conclusion: Clarity is needed on how to evaluate specific core competencies in infectious diseases, particularly systems-based practice and practice-based improvement. Describing anchoring milestones and evaluating fellows in accordance to stage in fellowship (i.e. early first year fellow) can help standardize responses. Further exploration on improving the evaluation process is warranted.