MP25-05: Program Directors' Perspectives on the Assessment of Residency Applicants in the Post-USMLE Step 1 Era: A strong case for Standardized Letters of Recommendation?
Introduction: The USMLE Step 1 exam evolved into a key metric utilized by program directors (PDs) in assessing candidates for residency. The transition to a USMLE Step 1 binary pass/fail scoring system has resulted in a loss of an important objective assessment. With national movements towards pass/fail systems for clerkship grading and trends towards abandonment of class ranking, assessing residency applications is becoming increasingly challenging. Methods: The Society for Academic Urologists (SAU) convened a task force to, in part, assess the perspectives of urology PDs regarding the importance of various aspects of a residency application for predicting clinical performance. An anonymous survey was disseminated to all urology program directors in the United States. Perspectives on 11 potential application predictors of clinical performance and demographics were recorded. Descriptive statistics characterized PD responses. Friedman’s Test and pairwise Wilcoxon tests were used to evaluate the relative ranks assigned to application elements by PD’s. Results: There was a 60.5% response rate (89/147). Letters of recommendation (LOR) was ranked as the most important predictor, with a mean of 2.39, median of 2 (IQR range 1- 3). Clerkship grades and USMLE Step 1 were comparable and ranked second with mean scores of 4.3, and 4.62 respectively. Medical school reputation ranked the lowest with a mean score of 8.47. There was significant heterogeneity among categories, however this was less so for LORs which predominated as the most important factor among application elements (p < 0.001). Conclusions: To our knowledge, this is the largest sample size assessing program director perspectives on application factors that predict clinical performance. The second (clerkship grades) and third (USLME Step 1) most important factors moving to binary pass/fail systems creates an opportunity for actionable change to improve assessment objectivity. Our data demonstrate LOR to be the most important factor of residency applications. Thus, there may be a strong argument for moving towards a standardized LOR to maximize this tool, mitigate bias, and improve inter-reviewer reliability. SOURCE OF Funding: N/A
