Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Residencies in emergency medicine (EM) teach the fundamental skills, knowledge, and humanistic qualities that constitute the foundations of EM practice.¹ Certification by the American Board of Emergency Medicine (ABEM) is an important step in ensuring that graduates of EM residency programs are able to practice independently and to ensure the highest standards in the specialty.² Recognizing this, the Accreditation Council for Graduate Medical Education (ACGME) requires that 80% of graduates of an EM residency program from the preceding five years must pass the ABEM certification examination on their first attempt.¹ Passing the ABEM certification examinations is also a Level 5 educational milestone for all EM residents.³ Therefore, obtaining successful ABEM certification on the first attempt is an important goal for EM program directors, residents, oversight agencies, and the public.

EM program directors invest a significant amount of time and effort each year to select medical students who will become successful, ABEM-certified EM practitioners. In 2014, 4,101 medical students applied to 171 EM residency programs offering 1,821 positions, with each program receiving an average of 807 applications.^4,5 With increasing competition, program directors use a number of metrics to identify the best candidates for matriculation into their residency program. Scores on the United States Medical Licensing Examination (USMLE) are commonly used by EM program directors to select applicants.⁶ The USMLE is a three-step examination that assesses a physician’s ability to apply knowledge, concepts, and principles and to demonstrate fundamental patient-centered skills that are important in health and disease and that constitute the basis of safe and effective patient care.^6,7 The USMLE examinations are widely considered valid measures of a medical student’s medical knowledge.⁸

Certain specialties have studied the correlation between USMLE scores and board certification rates. Diagnostic radiology, obstetrics and gynecology, pathology, pediatrics, physical medicine and rehabilitation, and surgery have all shown that high USMLE Step 1 or Step 2 Clinical Knowledge (CK) scores may predict successful board certification.^9-14 Variable data have been reported by orthopedic surgery and internal medicine.^15-21 A search of the literature showed no reports examining for correlation between USMLE scores and ABEM certification rates for EM, despite the widespread use of USMLE scores in medical knowledge assessment and resident selection.^6,22 The aim of this study was to determine if USMLE Step 1 or Step 2 CK scores are associated with successful first-attempt ABEM initial certification.

METHODS

Study Design

This retrospective cohort study examined data from residents who graduated from nine ACGME-accredited EM residency programs between 2009 and 2011. Data were originally collected via a multi-institutional collaborative effort, details of which have been described previously.²³ Briefly, nine programs volunteered to provide performance data on all their residents graduating in the previous three years, along with pre-residency predictor data available at the time of resident matriculation into the program. Institutional review board approval was obtained at each participating program as required.

Residents’ three-digit USMLE Step 1 and Step 2 CK scores were obtained from their residency files. If a resident had multiple attempts at any examination, the first score was used. A “composite USMLE score” was defined as the sum of the USMLE Step 1 and Step 2 CK score. Pass or fail status of the resident’s first attempt on the ABEM qualifying examination (“written boards”) and the ABEM oral certification examination (“oral boards”) were collected and used as outcome variables. We excluded residents with no USMLE scores reported (e.g., osteopathic students), those who had not taken their written or oral boards, and those whose scores were not available at the time of data collection.

Population Health Research Capsule

What do we already know about this issue?

The USMLE examinations are widely considered valid measures of a medical student’s medical knowledge and may possibly correlate with ABEM certification.

What was the research question?

The aim of this study was to determine if there is any association between USMLE scores and first-time success on the ABEM Qualifying and Oral Certification Examinations.

What was the major finding of the study?

Higher USMLE Step 1, Step 2 CK and composite scores are associated with better performance on ABEM examinations with Step 2 CK being the strongest predictor.

Data Analysis

Wilcoxon rank sum test evaluated the association between USMLE scores and written and oral boards outcomes. P-values < .05 were considered significant. We used univariate logistic regression analysis to obtain receiver operating characteristic area under the curve when using the USMLE scores to predict boards outcomes. USMLE score cutoff points based on 90%, 95%, and 99% chance of passing the boards were defined as the optimal cutoff points for predicting boards outcomes. We performed data analysis using SAS 9.4 (SAS Institute, Cary, NC, USA).

RESULTS

The nine EM programs were mostly urban, academic centers with eight to 14 residents in each class and emergency department censuses ranging from 60,000 to 115,000 patients per year.²³ Data were available for 286 residents; eight did not have a USMLE Step 1 score, 60 did not have a USMLE Step 2 score, 21 did not have a written board status available, and 26 did not have oral board results. We included a total of 197 residents with complete data in the analysis. Of these, 187 (95%) residents passed the written boards on their first attempt, 193 (98%) passed the oral boards, and 183 (93%) passed both boards on their first attempt. USMLE Step 1, Step 2, and composite scores in addition to first-attempt board pass rate are listed in Table 1. Residents who passed written boards had significantly higher USMLE Step 1, Step 2, and composite USMLE scores (p < .001), while residents who passed both boards had significantly higher Step 2 and composite scores (p < .001) (Table 2).

We calculated receiver operating characteristic area under the curve value for each score and determined predictor cutoff points. Table 3 shows the association of Step 1, Step 2, and composite scores compared to passing written and both parts of the ABEM examination.

DISCUSSION

Our study of EM residents across nine centers showed that those who had better performance on the USMLE Step 1 and Step 2 CK examinations had a higher probability of obtaining ABEM initial certification at the first attempt. We found that a USMLE Step 1 score of 202 and Step 2 CK score of 209 or higher correlated with a 90% chance of passing the ABEM qualifying examination on the first attempt, and scores of 205 and 215, respectively, correlated with a 90% chance of achieving first-attempt ABEM initial certification.

Prior studies in EM have attempted to identify pre-residency variables that may predict success during residency. Election into the Alpha Omega Alpha honor society, grade assigned for EM rotation, USMLE Step 1 score, interview score, letters of recommendation, scholarly activity, quality of medical school, and distinctive talents have all demonstrated association with successful performance during residency or high placement on rank lists.^23-26 Many of these factors are used by EM program directors while selecting and ranking applicants.⁶ To identify USMLE scores that would identify candidates with a high likelihood of passing the ABEM examinations on first attempt, the 2014 pass rates of 90% on first-attempt written boards and 96% on first-attempt oral boards prompted our use of the 90%, 95%, and 99% cutoff points for the Step 1, Step 2, and composite scores, respectively.²⁷ While studies correlating USMLE and ABEM are lacking, the cutoff scores identified by our study appear to correlate well with previous reported scores among different specialties. USMLE scores below 200 correlated with higher chance of failure in the American Board of Obstetrics and Gynecology, while scores less than 204 and 207 correlated with higher chance of failure in orthopedic and general surgery board examinations.^13,16,20 In addition, we found that the Step 2 CK score was most useful in predicting which students had a higher likelihood of passing their board examinations. Again, these findings seem to be reproducible among different specialties, notably general surgery, orthopedics, and obstetrics and gynecology, where Step 2 scores were superior to Step 1 scores in determining board outcomes.^20,28,29

These results suggest that USMLE scores can play an important role in screening applicants to interview for EM programs. In addition, these results may help identify EM residents at risk for not achieving first-attempt ABEM initial certification and may be used to structure individualized learning plans during residency.

LIMITATIONS

Limitations of our study include the retrospective design leading to reporting bias and incomplete records. The study sample may not be representative of the entire population because of a lack of a systematic enrollment process. (All programs that volunteered to participate were enrolled.) Data for 89 residents were incomplete or missing, and some programs did not record their residents’ USMLE Step 2 CK scores after the match. We did assess excluded residents who showed a higher mean Step 1 score compared to study residents (225 vs. 219; p = .004), and no statistically significant difference was detected in mean Step 2 and composite scores between excluded and included residents (225 vs. 228 and 445 vs. 447; p = .4 and .3, respectively). The first-time pass rate for ABEM qualifying, oral, and both examinations for excluded residents was 95%, 98%, and 95%, respectively, compared to 95%, 98%, and 93% in included residents, respectively. Based on these observations, excluded residents seemed to follow similar trends compared to included residents; however, we do not expect that excluding these residents would affect internal validity of our presented data. There was only one resident with multiple attempts on any USMLE examinations; this resident had two attempts on USMLE Step 1 with scores of 178 (Fail) and 194 (Pass), and only the first score was included per study protocol.

ABEM calculates a numerical score for performance on their examinations, but these data are shared only with individual residents and not EM program directors. Therefore, our study treated performance on the ABEM examinations as a categorical variable (pass/fail) as opposed to a continuous one.

An additional limitation of our study is that the number of residents failing the board examination was low and our cohort had a higher pass rate in both boards compared to 2014 nationwide rates for ABEM.²⁷ Hence, there might be a selection bias as residents included may not accurately represent the entire population of EM residents nationwide. ABEM pass/fail data are only available for residents who successfully complete an EM residency program. Therefore, it is impossible to conduct this study in a more relevant sample, such as all applicants to an EM program.

All included residents took their USMLE Step 1 between 2004 and 2006 and Step 2 between 2006 and 2008. USMLE scores for a given step are comparable across years and across forms; however, the USMLE cautions against comparing scores that were obtained at dramatically different periods of time.³⁰ Therefore, it may be difficult to apply our calculated cutoff scores to medical students seeking EM residencies today. Mean scores on USMLE examinations taken by 2014 graduates from U.S./Canadian medical schools are slightly higher than those from preceding years. While this is the first study to include multiple ACGME-accredited EM residency programs, future studies should focus on a larger number of community and academic programs and aim to include more residents. Future areas of research should also focus on patient-centered outcomes, as there are limited data on how performance on USMLE or ABEM examinations relates to clinical outcomes.

CONCLUSION

In our multicenter study, higher USMLE Step 1, Step 2 CK, and composite scores were associated with better performance on ABEM examinations with Step 2 CK being the strongest predictor. Cutoff scores for USMLE Step 1, Step 2 CK, and composite score were established to predict first-time success on ABEM initial certification.

Footnotes

Section Editor: Jeffrey Druck, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Elie Harmouche, MD, Henry Ford Hospital, Department of Emergency Medicine, 2799 West Grand Blvd, Detroit, MI 48202. Email: elieharmouche@gmail.com. 4 / 2017; 18:544 – 549

Submission history: Revision received September 13, 2016; Submitted November 26, 2016; Accepted December 12, 2016

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. The authors disclosed none.

REFERENCES

1. ACGME program requirements for graduate medical education in emergency medicine. Accreditation Council for Graduate Medical Education Web site. 2015. Available at: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/110_emergency_medicine_07012015.pdf. Accessed Nov 10, 2015.

2. What are ABEM’s mission and purposes?. American Board of Emergency Medicine Web site. Available at: https://www.abem.org/public/general-information/what-are-abem’s-mission-and-purposes-. Accessed Nov 10, 2015.

3. The Emergency Medicine Milestone Project. 2015. Available at: https://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/EmergencyMedicineMilestones.pdf. Accessed Nov 10, 2015.

4. Historical Specialty Specific Data, ACGME Residency, 2014–2010, By Program, Emergency Medicine. Association of American Medical Colleges Web site. 2014. Available at: https://www.aamc.org/download/359222/data/emergencymed.pdf. Accessed Nov 10, 2015.

5. Results and data 2015 main residency match. National Resident Matching Program Web site. 2015. Available at: http://www.nrmp.org/wp-content/uploads/2015/05/Main-Match-Results-and-Data-2015_final.pdf. Accessed Nov 10, 2015.

6. Results of the 2014 NRMP program director survey. National Resident Matching Program Web site. 2014. Available at: http://www.nrmp.org/wp-content/uploads/2014/09/PD-Survey-Report-2014.pdf. Accessed Nov 10, 2015.

7. About the USMLE. United States Medical Licensing Examination Web site. 2015. Available at: http://www.usmle.org/about/. Accessed Nov 10, 2015.

8. Margolis MJ, Clauser BE, Winward M, et al. Validity evidence for USMLE examination cut scores: results of a large-scale survey. Acad Med. 2010;85(10 Suppl):S93-7.

9. McCaskill QE, Kirk JJ, Barata DM, et al. USMLE Step 1 scores as a significant predictor of future board passage in pediatrics. Ambul Pediatr. 2007;7(2):192-5.

10. Picarsic J, Raval JS, Macpherson T. United States Medical Licensing Examination Step 1 two-digit score: a correlation with the American Board of Pathology first-time test taker pass/fail rate at the University of Pittsburgh Medical Center. Arch Pathol Lab Med. 2011;135(10):1349-52.

11. de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting performance on the American Board of Surgery qualifying and certifying examinations: a multi-institutional study. Arch Surg. 2010;145(9):852-6.

12. Fish DE, Radfar-Baublitz L, Choi H, et al. Correlation of standardized testing results with success on the 2001 American Board of Physical Medicine and Rehabilitation Part 1 Board Certificate Examination. Am J Phys Med Rehabil. 2003;82(9):686-91.

13. Armstrong A, Alvero R, Nielsen P, et al. Do U.S. medical licensure examination Step 1 scores correlate with Council on Resident Education in Obstetrics and Gynecology in-training examination scores and American Board of Obstetrics and Gynecology written examination performance?. Mil Med. 2007;172(6):640-3.

14. Boyse TD, Patterson SK, Cohan RH, et al. Does medical school performance predict radiology resident performance?. Acad Radiol. 2002;9(4):437-45.

15. Kay C, Jackson JL, Frank M. The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 examination. Acad Med. 2015;90(1):100-4.

16. Crawford CH, Nyland J, Roberts CS, et al. Relationship among United States Medical Licensing Step I, orthopedic in-training, subjective clinical performance evaluations, and American Board of Orthopedic Surgery examination scores: a 12-year review of an orthopedic surgery residency program. J Surg Educ. 2010;67(2):71-8.

17. Dougherty PJ, Walter N, Schilling P, et al. Do scores of the USMLE Step 1 and OITE correlate with the ABOS Part I certifying examination?: a multicenter study. Clin Orthop Relat Res. 2010;468(10):2797-802.

18. Swanson DB, Sawhill A, Holtzman KZ, et al. Relationship between performance on part I of the American Board of Orthopaedic Surgery certifying examination and xcores on USMLE Steps 1 and 2. Acad Med. 2009;84(10 Suppl):S21-4.

19. Herndon JH, Allan BJ, Dyer G, et al. Predictors of success on the American Board of Orthopaedic Surgery examination. Clin Orthop Relat Res. 2009;467(9):2436-45.

20. Klein GR, Austin MS, Randolph S, et al. Passing the Boards: can USMLE and Orthopaedic in-training examination scores predict passage of the ABOS Part-I examination?. J Bone Joint Surg Am. 2004;86-A(5):1092-5.

21. FitzGerald JD, Wenger NS. Didactic teaching conferences for IM residents: who attends, and is attendance related to medical certifying examination scores?. Acad Med. 2003;78(1):84-9.

22. Goyal N, Aldeen A, Leone K, et al. Assessing medical knowledge of emergency medicine residents. Acad Emerg Med. 2012;19(12):1360-5.

23. Bhat R, Takenaka K, Levine B, et al. Predictors of a top performer during emergency medicine residency. J Emerg Med. 2015;49(4):505-12.

24. Crane JT, Ferraro CM. Selection criteria for emergency medicine residency applicants. Acad Emerg Med. 2000;7(1):54-60.

25. Breyer MJ, Sadosty A, Biros M. Factors affecting candidate placement on an emergency medicine residency program’s rank order list. West J Emerg Med. 2012;13(6):458-62.

26. Hayden SR, Hayden M, Gamst A. What characteristics of applicants to emergency medicine residency programs predict future success as an emergency medicine resident?. Acad Emerg Med. 2005;12(3):206-10.

27. Application and examination activity: certification. American Board of Emergency Medicine Web site. 2015. Available at: http://www.abem.org/public/docs/default-source/longitudinal-surveys/exam-app-info-updated052214.pdf?sfvrsn=2. Accessed Nov 11, 2015.

28. Spellacy WN, Downes KL. United states medical licensing examination scores as a predictor of performance on the annual Council of Resident Education in Obstetrics and Gynecology examinations. J Reprod Med. 2014;59(1–2):17-9.

29. Spurlock DR, Holden C, Hartranft T. Using United States Medical Licensing Examination (USMLE) examination results to predict later in-training examination performance among general surgery residents. J Surg Educ. 2010;67(6):452-6.

30. USMLE score interpretation guidelines. United States Medical Licensing Examination Web site. 2015. Available at: http://www.usmle.org/pdfs/transcripts/USMLE_Step_Examination_Score_Interpretation_Guidelines.pdf. Accessed Jan 10, 2016.

Share on Facebook

Tweet

Share via email