Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Residency program directors (PD) screen large volumes of applications each recruitment season. A significant portion of each application is subjective, leaving ambiguity in data interpretation. Additionally, the medical student performance evaluation (MSPE) includes selected quotations from clinical clerkships and may or may not make a summative comparison of students to their peers. Emergency medicine (EM) has attempted to standardize recommendation letters and clerkship-grading transparency through the Standardized Letter of Evaluation (SLOE).¹ Without standardization, letters of recommendation showed grade inflation, lack of meaningful comparison between applicants, and the inability to use them as discriminatory tools for success in residency.^2-4 Inconsistencies in grades and evaluations by gender have also been demonstrated in other specialties.⁵ Even with standardized letters in several specialties, the use of the full spectrum of global assessments has not been found consistently nor has the accurate prediction of an applicant’s position on the rank list.^1,6,7

The only fully objective data on the residency application are the licensing examinations (United States Medical Licensing Examination [USMLE] and the Comprehensive Osteopathic Medical Licensing Examination [COMLEX]). These exams allow for direct applicant comparison as opposed to grades, which vary between schools. USMLE Step 2 Clinical Skill (CS) and COMLEX Step 2 Performance Evaluation (PE) require medical students to perform a history and physical examination on standardized patients. Step 2 CS “uses standardized patients to test medical students on their ability to gather information from patients, perform physical exams and communicate their findings to patients and colleagues.”⁸ These exams also incorporate communication skills to patients and colleagues into the final pass/fail grade. In its rationale for the Step 2 CS portion, the USMLE reports that poor communication and interpersonal skills are a reason for complaints against physicians. The scores on this portion of the examination also predict the success of these skills in first-year residents.⁹

During the 2017–2018 application season, the Association of American Medical Colleges (AAMC) instituted the Standardized Video Interview (SVI) as part of the EM residency application process. Using six questions, the SVI sought to provide objective information related to interpersonal and communication skills, and knowledge of professional behaviors.¹⁰ Applicants answered each question for up to three minutes, and a trained rater scored each video on a scale of 1–5, yielding a summative score of 6–30. Trained raters used anchors based on behavioral examples defining the proficiency level for each competency. Additionally, in their training raters examined PDs’ ratings of video examples to understand their perspective and develop consistency with their thought processes.¹¹ On the Electronic Residency Application Service (ERAS®) application, the AAMC provided the numerical score and full video recordings for review. Developers of the SVI sought to provide a more holistic presentation of the applicants beyond traditional test scores.¹²

Given the new data available to PDs, we sought to identify whether a correlation exists between any component of the USMLE examinations and the summative SVI score. Of particular interest was Step 2 CS, which incorporates interpersonal and communication skills into its evaluation. If no correlation between USMLE and SVI scores exists, this suggests that the SVI provides a new piece of information not previously available on the residency application. We hypothesized that numeric Step 1 and Step 2 Clinical Knowledge (CK) scores would not correlate with the SVI score, but that performance on the Step 2 CS portion may correlate with the SVI since both test communication skills.

METHODS

This was a prospective, cross-sectional study during the 2017–2018 residency application cycle. The study included nine Accreditation Council of Graduate Medical Education (ACGME)-accredited EM residency programs. Each site exported data directly from the ERAS applications, including the SVI score and scores for each component of the USMLE. Unique applicants were identified by their AAMC identification numbers and only included once in the analysis. We included only applicants with an SVI score and at least one score on the USMLE. We studied correlations between USMLE scores and SVI scores. Predetermined subgroup analysis was performed based on applicants’ USMLE Step 1 and Step 2 CK scores as follows: </= 200, 201–220, 221–240, 241–260, >260. USMLE Step 2 CS is graded pass or fail.

We used linear regression to examine correlation between SVI score and USMLE Step 1 and Step 2 CK scores. The Kruskal-Wallis test was used to compare SVI scores with USMLE subcategory scores. We performed the Mann-Whitney U test to compare SVI scores and USMLE Step 2 CS scores.

The study was reviewed by the institutional review board at the primary site.

RESULTS

A total of 1,329 unique applicants had an SVI score and at least one USMLE step score and were included in the analysis (Table 1). Of these, 1,325 had USMLE Step 1, 1,275 had USMLE Step 2 CK, and 1,000 had USMLE Step 2 CS scores available. Mean scores were as follows: SVI 19.6 (+/− 3.0, range 9–28); USMLE Step 1 231 (+/− 16.0, range 191–273); USMLE Step 2 CK 244 (+/− 14.8, range 188–282).

Using linear regression we found no correlation between the SVI score and overall USMLE Step 1 (p=0.58) or Step 2 CK score (p=0.56, Figure). In subgroup analysis, there was no correlation with specific scores for either Step 1 (p=0.29) or Step 2 CK (p=0.10, Table 2).

Four of the 1,000 students who had a CS score failed the examination. This did not correlate with the SVI score (p=0.08, Table 2).

DISCUSSION

During the 2017–2018 application season, the SVI score provided an additional objective metric to the EM residency application. This score was intended to measure interpersonal and communication skills, and knowledge of professional behaviors, features not otherwise captured in an objective way on the application. The evaluations of students during their undergraduate medical education are difficult to compare, as schools have varied grading policies and distributions. Data suggest varied correlations between elements of the application and prediction of success in residency, including the USMLE and induction into honor societies such as Alpha Omega Alpha.^13,14

The USMLE provides PDs with a standardized metric as a result of a uniform grading system across all test-takers. In Step 1 and Step 2 CK, examinees answer multiple choice questions related to the basic sciences and then clinical medicine. In this analysis, we found that both overall score on the USMLE as well as individual ranges of score did not correlate with performance on the SVI. Given that the SVI was designed to specifically assess interpersonal and communication skills, as well as knowledge of professional behaviors, it is not surprising that we found no correlation between the USMLE Step 1 and Step 2 CK scores and the SVI score.

The CS portion of the USMLE Step 2, however, assesses communication skills using a standardized patient encounter. Since the SVI also focuses on components of communication, we hypothesized that a correlation could exist between these two scores. In this dataset, only four out of 1,000 (0.4%) applicants who had a CS score available failed the examination. This analysis suggested a trend towards a correlation more than the other analyses; however, it did not achieve statistical significance. Given the extremely low failure rate on the CS examination, it is difficult to assess this correlation. Additionally, the correlation is limited by our inability to break out the analysis by specific USMLE Step 2 CS subcomponent (Communication and Interpersonal Skills; Spoken English Proficiency; and Integrated Clinical Encounter) score ranges. Our results support the notion that the SVI may provide unique information on the residency application. At least in comparison with the USMLE – the only other standard score on the application – we found no correlation between the two scores.

Our results are consistent with those reported by the AAMC.¹⁵ The AAMC contends that Step 2 CS and the SVI measure related but different constructs.¹⁰ For example, Step 2 CS measures spoken English proficiency, which is not measured by the SVI. Similarly, the SVI measures teamwork, which is not measured by Step 2 CS. Over time, PDs will benefit from more SVI data including its ability to predict in-person interview scores, success during residency, chief resident selection, and professionalism or communication remediation.

LIMITATIONS

Given the study design with nine EM residency programs, only 1,329 of the 2,901 total applicants to United States EM residency programs in the 2017–2018 application season were included.¹⁶ This may limit the overall generalizability of the data set. Additionally, in the cohort of 1,000 applicants in which USMLE Step 2 CS scores were available, only four persons failed the examination, which may have impacted the ability to detect any correlation between this examination and the SVI score.

CONCLUSION

In this analysis, we found no correlation between the SVI score and any component of the USMLE. As a result, the SVI may provide a unique piece of data for PD interpretation. It is unclear how it will correlate long term with resident performance or success. Additionally, further investigation will help to determine whether the SVI score impacts the decision-making of PDs both in the interview offer and ultimate applicant selection.

Footnotes

Section Editor: Jonathan Fisher, MD, MPH

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Daniel J. Egan, MD, Columbia University College of Physicians and Surgeons, Department of Emergency Medicine, 622 West 168th Street, VC-260, New York, NY 10032. Email: danjegan@gmail.com. 1 / 2019; 20:87 – 91

Submission history: Revision received July 12, 2018; Submitted October 8, 2018; Accepted November 16, 2018

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Hegarty CB, Lane DR, Love JN, et al. Council of Emergency Medicine Residency Directors standardized letter of recommendation writers’ questionnaire. J Grad Med Educ. 2014;6(2):301-6.

2. Chole RA, Ogden MA. Predictors of future success in otolaryngology residency applicants. Arch Otolaryngol Head Neck Surg. 2012;138(8):707-12.

3. DeZee KJ, Thomas MR, Mintz M, et al. Letters of recommendation: rating, writing, and reading by clerkship directors of internal medicine. Teach Learn Med. 2009;21(2):153-8.

4. Shultz K, Mahabir RC, Song J, et al. Evaluation of the current perspectives on letters of recommendation for residency applicants among plastic surgery program directors. Plast Surg Int. 2012;2012:728981.

5. Messner AH, Shimahara E. Letters of recommendation to an otolaryngology/head and neck surgery residency program: their function and the role of gender. Laryngoscope. 2008;118(8):1335-44.

6. Oyama LC, Kwon M, Fernandez JA, et al. Inaccuracy of the global assessment score in the emergency medicine standard letter of recommendation. Acad Emerg Med. 2010;17(Suppl 2):S38-41.

7. Grall KH, Hiller KM, Stoneking LR. Analysis of the evaluative components on the Standard Letter of Recommendation (SLOR) in Emergency Medicine. West J Emerg Med. 2014;15(4):419-23.

8. USMLE Step 2 CS. Available at: https://www.usmle.org/step-2-cs. Accessed November 20, 2018.

9. FAQs. Available at: https://www.usmle.org/frequently-asked-questions/#step2cs. Accessed June 18, 2018.

10. About the SVI. Available at: https://students-residents.aamc.org/applying-residency/article/about-svi/. Accessed July 3, 2018.

11. The AAMC Standardized Video Interview: Essentials for the ERAS®2019 Season. Available at: https://aamc-orange.global.ssl.fastly.net/production/media/filer_public/75/19/75198c9f-29d1-4884-b932-1e7975e4262a/standardized-video-interview-essentials-eras-2019-20180720.pdf. Accessed September 19, 2018.

12. Bird SB, Blomkalns A, Deiorio NM, et al. Beyond test scores and medical knowledge: the Standardized Video Interview, an Innovative and ethical approach for holistic assessment of applicants. Acad Med. 2018;93(2):151.

13. Bhat R, Takenaka K, Levine B, et al. Predictors of a top performer during emergency medicine residency. J Emerg Med. 2015;49(4):505-12.

14. Van Meter M, Williams M, Banuelos R, et al. Does the National Resident Match Program rank list predict success in emergency medicine residency programs?. J Emerg Med. 2017;52(1):77-82.e1.

15. Dunleavy D, Overton R. Standardized Video Interview update. 2018.

16. The MATCH® National Resident Matching Program®. Results and Data 2018 Main Residency Match®. Available at: http://www.nrmp.org/wp-content/uploads/2018/04/Main-Match-Result-and-Data-2018.pdf. Accessed June 18, 2018.