Author	Affiliation
Alexis Pelletier-Bui, MD	Cooper Medical School of Rowan University, Department of Emergency Medicine, Camden, New Jersey
Doug Franzen, MD	University of Washington School of Medicine, Department of Emergency Medicine, Seattle, Washington
Liza Smith, MD	University of Massachusetts Medical School Baystate Health, Department of Emergency Medicine, Springfield, Massachusetts
Laura Hopson, MD	University of Michigan Medical School, Department of Emergency Medicine, Ann Arbor, Michigan
Lucienne Lutfy-Clayton, MD	University of Massachusetts Medical School Baystate Health, Department of Emergency Medicine, Springfield, Massachusetts
Kendra Parekh, MD	Vanderbilt University School of Medicine, Department of Emergency Medicine, Nashville, Tennessee
Mark Olaf, DO	Geisinger Commonwealth School of Medicine, Department of Emergency Medicine, Scranton, Pennsylvania
Tom Morrissey, MD	University of Florida College of Medicine-Jacksonville, Department of Emergency Medicine, Jacksonville, Florida
David Gordon, MD	Duke University, Division of Emergency Medicine, Durham, North Carolina
Erin McDonough, MD	University of Cincinnati College of Medicine, Department of Emergency Medicine, Cincinnati, Ohio
Benjamin H. Schnapp, MD, MEd	University of Wisconsin School of Medicine and Public Health, BerbeeWalsh Department of Emergency Medicine, Madison, Wisconsin
Mary Ann Edens, MD	Louisiana State University Health Shreveport, Department of Emergency Medicine, Shreveport, Louisiana
Michael Kiemeney, MD	Loma Linda University School of Medicine, Department of Emergency Medicine, Loma Linda, California

Introduction
Application review
Interviews
Exploration of new methods/metrics for applicant assessment
Conclusion

INTRODUCTION

The coronavirus pandemic has substantially disrupted undergraduate medical education. While creative solutions have been implemented for classroom-based activities, suspension of patient contact¹ causes a disproportionate impact on students in clinical rotations. Rotational experiences are critical to the development of learners into independent practitioners and comprise an important component of the residency selection process.² Emergency medicine (EM) is particularly vulnerable to the impact of the altered clinical learning environment as emergency departments (ED) are dealing with large volumes of coronavirus disease (COVID-19) patients and often low volumes of non-COVID patients.³ Further, EM courses are frequently only available to senior medical students, limiting students’ opportunities for exposure to the specialty.⁴

Given these constraints, residency programs will need to adjust their expectations for residency application materials, including numbers and types of recommendation letters, clinical experiences, and United States Medical Licensing Examination (USMLE) scores. Considering forecasts for a second surge of coronavirus in the fall or winter, paired with ongoing or reinstituted travel restrictions, the Coalition for Physician Accountability (Coalition) has recommended that all residency programs commit to virtual interviews for the entire upcoming application cycle, and the EM community has released a statement supporting this recommendation.^5,6 While challenging, these changes do, however, provide an opportunity to explore alternative models for recruiting the next generation of emergency physicians while simultaneously identifying creative solutions to longstanding difficulties such as volume of applications and cost. We encourage programs to respond to the challenges presented by COVID-19 not just reactively, but with an eye toward transformative change of the application and interview process. We offer here suggestions on the application and interview process, many of which could be carried forward into future application cycles.

APPLICATION REVIEW

EM Rotations and Letters of Recommendation

Challenges

EM program directors (PD) cite the Standardized Letter of Evaluation (SLOE) as the most important component of the residency application when making interview invitation decisions.^2,7-9 Many residency programs expect two SLOEs prior to making interview offers.¹⁰ Academic group SLOEs from residency programs carry the most weight,^8,9 placing a greater burden on applicants from schools without a home EM residency (“orphan” students). Recognizing that evaluations from EM subspecialty rotations or EM faculty not affiliated with a residency program also have merit, the Council of Residency Directors in EM (CORD) has developed modified SLOEs (https://www.cordem.org/resources/residency-management/sloe/), which carry less weight.⁹ EM residency programs place lowest value on letters of recommendation from non-EM faculty.⁹

With clinical rotations suspended nationally, formerly predictable clerkship curricula are in flux, potentially delaying fourth-year rotations, including EM. Even those who are able to resume regularly scheduled fourth-year rotations are discouraged from performing away rotations to promote equity between applicants;^5,6 therefore, the majority of applicants will only have one traditional SLOE in their residency application. At an even greater disadvantage are “orphan” applicants, who may have difficulty obtaining even a single EM rotation. Applicant groups that are disproportionately affected by “orphan” status are osteopathic and international medical graduate (IMG) applicants. Restricted access to fourth-year EM rotations may also impact the number of applicants to EM, as up to 36% of surveyed US medical student applicants to EM didn’t decide until their fourth-year of medical school.¹¹ On the other hand, some students in a similar position may still attempt to pursue a career in EM only to develop specialty regret and potentially leave an unfilled position in a residency program.

Suggested Solutions

Decreased rotation availability and fewer EM residency-authored SLOEs will require rethinking the current hierarchy of letters and giving more value not only to non-traditional SLOEs, but also to letters authored by non-EM faculty. To increase efficacy and make the task easier for non-EM letter writers, CORD has assembled a committee to develop a template for writers, highlighting the attributes EM PDs specifically look for in a letter and in an applicant. This template, termed the “O-SLOE” for “off-service” or “other rotation,” is now available on CORD’s SLOE webpage (https://www.cordem.org/resources/residency-management/sloe/). This can be distributed to medical school deans and clerkship director organizations. Continued use of this template beyond this extraordinary academic year could increase the utility and rigor of non-EM evaluations in future application cycles. This same committee has updated the EM-faculty SLOE to allow writers to detail how COVID-19 has affected their student rotations (“on the reCORD,” CORD listserv communication, June 23, 2020).

The Coalition and EM community recommend that away rotations be discouraged for the 2020–2021 academic year with the exception of “learners who have a specialty interest and do not have access to a clinical experience with a residency program in that specialty in their school’s system,” and “learners for whom an away rotation is required for graduation or accreditation requirements.”^5,6 Given there are over 80 identified schools without an established EM residency program and there is a possibility of students being limited from performing an EM rotation at their own institution due to high volume of COVID-19 patients (Susana Tsao, DO, CORD listserv communication, May 4, 2020), we support these exceptions to the “no aways” policy. We also support the recommendation for institutions that are still hosting rotators to preferentially accept students who are unable to obtain a SLOE from their home institution.^12,13 A live document of schools without a home EM residency program can be found here (https://bit.ly/37UKYEp).

Proactive efforts should be made to increase EM rotation availability for EM-bound students. Potential strategies include shortening the length of the rotation (eg, from four weeks to three weeks) or reducing the number of shifts required per rotator, allowing more rotators per block. The timeframe for completion of EM rotations could be expanded beyond the traditional summer months, particularly considering the recently updated Electronic Residency Application Service (ERAS) application timeline with residency programs not being able to view residency applications until October 21, 2020.¹⁴

Even with utilization of these strategies, some EM-bound applicants may still not have a SLOE in time for file review. In such cases, we support the recommendation that an EM advisor write a letter of recommendation specifically incorporating the key elements of the SLOE.¹³ It should explicitly state that the applicant was unable to obtain a SLOE in time for file review secondary to COVID-19 and vouch for the applicant’s desire to pursue a career in EM, their career decision process, and potential for success. Additional insight into an applicant’s prior EM experiences beyond what can be gleaned from their curriculum vitae could also be helpful.

Board Scores

Challenges

After SLOEs, the USMLE Step 1 and Step 2 Clinical Knowledge (CK) exams are the next biggest factors of importance in selecting applicants for interviews.² In the 2018 National Resident Matching Program PD survey, 48% of PDs required USMLE Step 2 CK and 31% used a target score for Step 2 CK when considering applicants for interviews.

Most Prometric Testing centers, which administer the USMLE Step 1 and 2 CK, were closed until June 1, 2020, and are still not running at full capacity based on governmental ordinances and advice from the Centers for Disease Control and Prevention and the World Health Organization.¹⁵ Gradual and incomplete opening, paired with the backlog of individuals competing for standardized testing, has lead to delays in testing and will likely result in later release of scores for many applicants. Likewise, USMLE Step 2 Clinical Skills (CS) testing has been suspended for 12–18 months.¹⁶ IMG applicants may be disproportionately affected by the inability to complete Step 2 CS, previously a requirement for certification by the Educational Commission for Foreign Medical Graduates (ECFMG).¹⁷ PDs may not have the full complement of USMLE scores available to them that they have traditionally relied upon for applicant screening and rank list submission. Additionally, it is unknown how these delays will affect applicants’ ability to obtain USMLE scores in time for medical school graduation.

Suggested Solutions

There are potential solutions to compensate for the testing bottleneck. Six US medical schools have opened regional testing centers to allow for additional testing options outside of Prometric and the USMLE is working with eligible medical schools across the US to host one-day testing events to administer Step 1 and Step 2 CK in the near future.¹⁸. The ECFMG has created five new pathways to meet the requirements for ECFMG certification for those IMG applicants who have yet to complete Step 2 CS for the 2021 match cycle.¹⁷ We encourage programs to consider these pathways as a substitute for the Step 2 CS examination. Programs that previously required Step 2 CK or CS for interview or ranking should consider temporarily revising their approach and policies. Institutional expectations or state-level requirements for licensure should be clearly articulated and communicated to allow applicants to make educated application decisions.

However, we encourage residency programs to use this opportunity to take a new approach to the USMLE. Quantitative metrics such as USMLE Step 1 and 2 CK are commonly used to stratify and filter students. While scores do have some correlation with the likelihood of passing the American Board of Emergency Medicine qualifying exam,¹⁹ these scores do not correlate with clinical proficiency or success in EM residency.²⁰ Reducing the influence of USMLE scores when screening applicants has long been discussed among EM educators, with a goal to transition toward more holistic application review. The transition of USMLE Step 1 to a pass/fail score within the next few years,²¹ combined with the testing disruptions of the COVID-19 pandemic, present an opportunity for more rapid change.

Holistic application review involves programs performing honest self-assessment and appraisal of residency graduates to determine which character traits and attitudes are valued and associated with success in their program and then seeking out applicants with those qualities.²² Combining holistic application review with an understanding of a program’s own strengths and challenges in resident development can help programs identify and recruit applicants who are more likely to match successfully with them and succeed in training.

Medical Student Performance Evaluation (MSPE)

Challenges

The MSPE is traditionally released at the beginning of October, marking the unofficial start of interview offers to applicants. Programs often wait for the MSPE before sending interview invitations, as it may describe professionalism or academic concerns that do not appear elsewhere in the application. With fewer SLOEs to review, and varying clinical experiences between students, the MSPE may take on additional importance this year. As a result of cancelled clinical rotations, some applicants may not have completed the core clerkships that traditionally contribute significantly to the MSPE.

Suggested Solutions

ERAS has amended the residency application timeline to allow for MSPE release on October 21, the same day that residency programs will be able to begin reviewing applications.¹⁴ Delaying these components of the application cycle will effectively push back the start of interview season and might relax the time pressure on students, schools, and programs.

Advising students to complete most, if not all, core clerkships before the release of the MSPE will offer applicants a better chance at receiving interview offers than if some core experiences are incomplete and updated at a later time. If not possible, a mechanism to allow regular updates to the MSPE may be useful.

COVID-19 has not affected geographic regions equally. Programs will look to the MSPE to delineate the pandemic’s effect on the learning environment. We suggest adding a standardized “pandemic response” section to MSPEs. Schools should describe how they adapted, including what dates students were excluded from clinical experiences, what clerkship experiences may have been virtual, and any policies that prohibited students from seeking away rotations. Additionally, any action by students who went above and beyond to help during this time of crisis could be very useful information to EM PDs. If all schools use the MSPE to outline a school’s pandemic response, readers will be able to place a student’s record into context and identify outliers within a single school or between schools.

Numbers of Applicants/Applications

Challenges

Cancellation of EM away rotations and delays in senior electives while students complete core clerkships may decrease the total number of EM applicants, due to decreased exposure to EM and uncertainty about the specialty. Simultaneously, the number of applications per applicant may increase due to perceived deficiencies (eg, lack of SLOEs and/or USMLE scores, atypical MSPE) and the inability to hone residency and geographic preferences via audition rotations.

Suggested Solutions

While we cannot control students’ reactions to the uncertainty of this application cycle, the EM community can make efforts to mitigate other anxiety-provoking elements of the application process, foremost a lack of transparency. Clear communication will be critical, including informing students when they can expect to hear about interview offers, a program’s preferred method of contact for questions or updates, and clear expectations for wait-list status.

Programs might consider allowing students to submit an optional statement of purpose or intent, in which students have the opportunity to communicate a particular interest in a given program or region. Allowing students to communicate what they perceive to be their “fit” with a program can help application reviewers identify students who are likely to be high-yield candidates and successful matches. This was instituted by the otolaryngology community as a requirement in the 2015–16 application cycle as a program-specific paragraph added to the end of an applicant’s personal statement. While this measure was found to be effective in decreasing the overall number of applications per applicant, it is also thought to have made otolaryngology appear less welcoming to medical students as there was an overall decline in the number of medical students applying to otolaryngology after this initiative. As a result, the Otolaryngology Program Directors Organization has now made the program-specific paragraph optional.^23,24 Programs that wish to institute a program-specific paragraph as part of their application process are asked to publish these requests on ERAS when registering in ERAS Account Maintenance. This information will then be displayed to applicants as they research programs (Elise Lovell, MD, on behalf of Amy Mathis, Senior Director of ERAS, CORD listserv communication, May 11, 2020).

One low-cost, low-effort method for students and PDs that may allow PDs to identify high-yield candidates amidst increasing application numbers is the institution of preference signaling, or a “star” system, which gives students a limited number of “stars” to allocate to their most desired programs. This method is used in other professions and has been proposed for use in orthopedics and otolaryngology.^25-28 A computer-simulated model using otolaryngology match data found that applicants voluntarily adding preference data to their application enhanced the practical number of interview invitations for all applicants and could potentially allow more holistic file review of high-yield candidates to their program.²⁸ However, implementation would require the collaboration with the Association of American Medical Colleges (AAMC) and development of technology that may be challenging to institute in time for this year’s residency application cycle.

INTERVIEWS

Challenges

The interview and surrounding interactions are routinely cited by PDs as a major factor in ranking decisions.² The Coalition recommends that all residency programs commit to virtual interviews for the upcoming application cycle.⁵ Opportunities for social events and “second looks” may also be curtailed. These interactions play key roles in assessing goodness of fit, both from the applicant’s and the program’s perspective, and are cited as particularly important for ranking decisions by under-represented minorities in medicine.²⁹

Suggested Solutions

Video Interviewing

CORD supports replacing traditional in-person interviews with video interviews to try to maintain an equitable interview process for applicants and programs through this entire residency application cycle.⁶ Several programs have successfully demonstrated high satisfaction rates with video interviews, highlighting the advantages of reducing time and cost burdens for both the applicant and programs.^30-34 Video interviews have the potential to level the playing field for applicants of low socioeconomic status who may not have the financial ability to travel for interviews.^30,31. Reduced absences from clinical rotations would further enhance education in the fourth year of medical school.³⁴

Despite some of the intrinsic appeal of video interviews,^30,31 there are limitations compared to traditional on-site visits. Applicants face challenges accurately representing who they are as people, or in providing comfort in ranking a residency program,^30,32 as many subtle indicators occur outside the actual interview discussion. Other perceived disadvantages are the inability to learn about a city or program and difficulty interacting with current residents and faculty.³¹ One study showed that neither applicants nor interviewers were comfortable making video interviewing the only means of interviewing,³² although another showed that some applicants prefer an initial video interview with the ability to later visit a program.³¹ This may be an option for some programs if travel restrictions and social distancing are in effect in the fall but later lifted prior to rank list submission.

For recommendations on the mechanics of conducting virtual interviews, consider reviewing the tips for PDs and interviewers published by the AAMC and the Compendium of Resources published by the Coalition.^35-37

Highlighting the Program Beyond the Interview

We must prepare for no in-person visits to residency programs. While this may offer challenges in showcasing a residency program, this is not insurmountable.

Residencies may find it beneficial to leverage existing experiences taking place at their institutions, rather than creating entirely new content for applicant consumption. For example, video of residents interacting with faculty during a small group session may provide invaluable information to applicants about didactic quality, how faculty and residents interact, and resident camaraderie.

If not already available, programs should consider creation of expanded content (written, photographic, video) that highlights their program’s goals, strengths, and educational philosophy as well as what they are looking for in an applicant. New content will likely need to be created to replace the tour of the ED, resident spaces, the hospital and the surrounding geographic region. While this runs the risk of advantaging programs with the time and resources to produce professional-appearing content, these costs likely pale in comparison to that of running full interview days, and the potential advantages to program and applicants in terms of increased information to make their residency decisions may outweigh these risks.

Wide distribution of content will be key. An institution residency webpage, an external website, and social media accounts, if allowed by the institution, will likely be the best options for highlighting this material. Video content could also be uploaded onto YouTube or Vimeo. Given the high utilization of the Emergency Medicine Residents’ Association (EMRA) Match website by medical students, programs should ensure that their webpages and social media sites are updated on their EMRA Match profile.

Helping Applicants get a “Feel” for a Program and Assess for Fit

Programs could consider hosting mini virtual-EM rotations with the ability for students to “attend” a short series of video didactic conferences at outside sites. This could be particularly beneficial for programs that traditionally depend on their EM rotation experience as a recruitment tool for outside rotators. Benefits to the student include the abilities to get a glimpse of a program’s teaching styles, facilitate interaction with faculty and residents at other EM programs beyond a single interview day, and make up for lost educational opportunities due to the reduction of the typical number of EM rotations. Virtual rotations could be particularly beneficial for IMG applicants who will likely have difficulty obtaining even one EM rotation due to travel restrictions. We encourage institutions hosting virtual EM rotations to strongly consider accepting IMG students into their rotation to help balance this inequity. To ensure a good ratio of faculty to learners, students should only be able to participate in a small number of these, equivalent to the number of away rotations typically performed. Students’ selections of which of these experiences to participate in could give programs insight into what type of program (or where) the student is ideally looking to match.

For students who are invited to interview with a program, an invitation could be extended to “attend” conferences by sharing a virtual forum link. Asking selected faculty or residents to remain online with the students after the conference could be another way for applicants and faculty/residents to get to know each other.

Programs could “host” online pre-interview socials or lunch-time “hangouts” with residents, with the ability to break up into smaller rooms or even one-on-one conversations. Programs could also consider hosting a “hangout” in the spring or summer to generate interest in their program, and record/post it for interested applicants to review. A topic for consideration during these “hangouts” is the kind of resident that really shines in their program, and what kind of applicant might have struggles or find the environment less palatable. This might help attract best-fit applicants.

Programs could also consider using a tool that allows interviewees to guide their interview in a way that is meaningful for them and allows a more accurate impression of themselves compared with a traditional interview, as used by one otolaryngology residency program.³⁸ The tablet-based interactive Candidate Assessment Tool allows residency applicants to select questions via a homepage of prerecorded video clips from key leaders in the institution, covering a variety of topics, interests and Accreditation Council for Graduate Medical Education core competencies.

EXPLORATION OF NEW METHODS/METRICS FOR APPLICANT ASSESSMENT

Challenge

With the potential disruption of metrics and methods that have traditionally been highly valued in the residency selection process,² programs will have challenges identifying and stratifying applicants who may be successful in their residency program. The combination of this pandemic and USMLE Step 1 moving to pass/fail creates an opportunity for graduate medical education (GME) to develop and explore new methods to better holistically review applications. Another goal of many EM residency programs is to increase the number of under-represented minorities in one’s program, as evidenced by the surge in under-represented minority EM clerkship scholarships.³⁹

Suggested Solutions

While the EM community was not interested in continuing the AAMC Standardized Video Interview pilot as a new metric for EM residency selection,⁴⁰ there may be alternative tools that we can explore to help identify successful applicants to our EM residency programs. Pre-hire assessments are used by eight of the top 10 US private employers and by 57% of large US employers.⁴¹ Some of these assessments have demonstrated utility in undergraduate medical education and GME as well.^42-48

For example, emotional intelligence testing has been shown to have positive correlation with medical school success.⁴³ Personality testing to assess for fit is widely used in other industries and some data indicate that it may be effective for residency selection as well.⁴² Situational judgment testing through methods such as the Computer-based Assessment for Sampling Personal Characteristics (CASPer), has demonstrated moderate predictive validity to national licensure outcomes in Canada⁴⁴ and is required by all medical school applicants in Canada and two US medical schools.⁴⁵ Its utilization in general surgery has been associated with overall performance in residency,⁴⁶ allowed for more general surgery interview offers to under-represented minorities in medicine,⁴⁸ and did not detract applicants from applying to the general surgery programs that implemented its use.^45,48

While it is neither feasible, nor advisable, to incorporate these assessments for all EM applicants during this application cycle, now is a better time than ever to begin exploring and validating these, or other methods, for potential future use in the EM residency application process. Altus Assessments, the developer of CASPer, and the National Board of Medical Examiners are collaborating to explore the use of CASPer during this residency application season (Elise Lovell, MD, on behalf of Altus Assessments and the National Board of Medical Examiners, CORD listserv communication, June 21, 2020). Programs interested in participating in this research project can fill out this form (http://bitly.ws/9s73).

CONCLUSION

While COVID-19 presents significant challenges to medical education and the residency application process, the GME community can view this pandemic as an opportunity to explore disruptive change. Just as quarantine orders eliminated barriers to virtual meetings and appointments overnight, we have the chance to make important changes to the residency application process that will benefit programs and applicants for years to come (Tables 1 and 2). We should embrace this opportunity while simultaneously working to preserve the essential components of the process. Only time will tell how the pandemic will influence the match process, and what adaptations will be most helpful. The CORD Application Process Improvement Committee is committed to making the most of the situation and plans to follow up with a report detailing the impacts and best practices for addressing similar challenges in the future.

Footnotes

Section Editor: Danya Khoujah, MBBS

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Alexis Pelletier-Bui, MD; Cooper Medical School of Rowan University, Department of Emergency Medicine, 1 Cooper Plaza, 152 Kelemen, Camden, NJ 08103. Email: pelletier-bui-alexis@cooperhealth.edu. 9 / 2020; 21:1105 – 1113

Submission history: Revision received May 20, 2020; Submitted June 30, 2020; Accepted August 7, 2020

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Important guidance for medical students on clinical rotations during the coronavirus (COVID-19) outbreak. Available at: https://www.aamc.org/news-insights/press-releases/important-guidance-medical-students-clinical-rotations-during-coronavirus-covid-19-outbreak. Accessed April 24, 2020.

2. Results of the 2018 program director survey. Available at: https://www.nrmp.org/wp-content/uploads/2018/07/NRMP-2018-Program-Director-Survey-for-WWW.pdf. Accessed April 24, 2020.

3. Reed T. ERs have been quiet during COVID-19. What happens when the patients come back?. Available at: https://www.fiercehealthcare.com/hospitals-health-systems/ers-have-been-quiet-during-covid-19-what-happens-when-patients-come-back. Accessed April 24, 2020.

4. Mulcare MR, Suh EH, Tews M, et al. Third-year medical student rotations in emergency medicine: a survey of current practices. Acad Emerg Med. 2011;18:S41-7.

5. Final report and recommendations for medical education institutions of LCME-accredited, U.S. osteopathic, and non-U.S. medical school applicants. Available at: https://www.aamc.org/system/files/2020-05/covid19_Final_Recommendations_05112020.pdf. Accessed May 14, 2020.

6. Consensus statement on the 2020–2021 residency application process for US medical students planning careers in emergency medicine in the Main Residency Match. Available at: https://www.cordem.org/globalassets/files/2020-res-app-consensus-statement-v2.pdf. Accessed June 24, 2020.

7. Negaard M, Assimacopoulos E, Harland K, Van Heukelom J. Emergency medicine residency selection criteria: an update and comparison. AEM Educ Train. 2018;2(2):146-53.

8. Love JN, Smith J, Weizberg M, et al. Council of Emergency Medicine Residency Directors’ Standardized Letter of Recommendation: The program director’s perspective. Acad Emerg Med. 2014;21(6):680-7.

9. Jarou Z. EMRA and CORD Student Advising Guide. 2019.

10. EMRA Match. Available at: https://webapps.emra.org/utils/spa/match#/search/map. Accessed April 24, 2020.

11. Ray JC, Hopson LR, Santen SA, et al. Choosing emergency medicine: influences on medical students’ choice of emergency medicine. PLoS One. 2018;13(5):e0196639.

12. Shandro J, Kessler R, Schrepel C, et al. Advising medical students during COVID-19: the case for a single emergency medicine rotation for all. AEM Education and Training. 2020.

13. Katirji L, Smith L, Pelletier-Bui A, et al. Addressing challenges in obtaining emergency medicine away rotations and standardized letters of evaluation due to COVID-19. West J Emerg Med. 2020;21(3):538-41.

14. ERAS 2021 residency timeline. Available at: https://students-residents.aamc.org/applying-residency/article/eras-timeline-md-residency/. Accessed May 14, 2020.

15. Special notice from Prometric regarding COVID-19 (coronavirus). Available at: https://www.prometric.com/corona-virus-update. Accessed June 24, 2020.

16. USMLE suspending Step 2 Clinical Skills Examination. Available at: covid.usmle.org/announcements/usmle-suspending-step-2-clinical-skills-examination. Accessed June 24, 2020.

17. Requirements for ECFMG certification for 2021 match. Available at: ecfmg.org/certification-requirements-2021-match/. Accessed June 24, 2020.

18. Preparations for event testing. Available at: covid.usmle.org/announcements/preparations-event-testing. Accessed June 24, 2020.

19. Harmouche E, Goyal N, Pinawin A, et al. USMLE scores predict success in ABEM initial certification: A multicenter study. Western J Emerg Med. 2017;18(3):544-49.

20. Wagner JG, Schneberk T, Zobrist M, et al. What predicts performance? A multicenter study examining the association between resident performance, rank list position, and United States Medical Licensing Examination Step 1 Scores. J Emerg Med. 2017;52(3):332-40.

21. InCUS-Invitational conference on USMLE scoring. Available at: https://www.usmle.org/incus/. Accessed April 21, 2020.

22. Holistic admissions. Available at: https://www.aamc.org/services/member-capacity-building/holistic-review. Accessed April 21, 2020.

23. Stepan KO, Kaul VF, Raquib AR, et al. An evaluation of the program-specific paragraph in the otolaryngology residency application. Laryngoscope. 2018;128(11):2508-13.

24. Kramer S. Is the program-specific paragraph responsible for declining application numbers? A commentary. Otolaryngol Head Neck Surg. 2018;158(2):215-6.

25. Salehi PP, Azizzadeh B, Lee YH. Preference signaling for competitive residency programs in the NRMP. J Grad Med Educ. 2019;11(6):733-4.

26. Bernstein J. Not the last word: want to match in an orthopaedic surgery residency? Send a rose to the program director. Clin Orthop Relat Res. 2017;475(12):2845-9.

27. Whipple ME, Law AB, Bly RA. A computer simulation model to analyze the application process for competitive residency programs. J Grad Med Educ. 2019;11(1):30-5.

28. Salehi PP, Benito D, Michaelides E. A novel approach to the National Resident Matching Program—The Star System. JAMA Otolaryngol Head Neck Surg. 2018;144(5):397-8.

29. Agawu A, Fahl C, Alexis D, et al. The influence of gender and underrepresented minority status on medical student ranking of residency programs. J Natl Med Assoc. 2019;111(6):665-73.

30. Shah SK, Arora S, Skipper B, et al. Randomized evaluation of a web based interview process for urology resident selection. J Urol. 2012;187:1380-4.

31. Daram SR, Wu R, Tang SJ. Interview from anywhere: feasibility and utility of web-based videoconference interviews in the gastroenterology fellowship selection process. Am J Gastroenterol. 2014;109:155-9.

32. Edje L, Miller C, Kiefer J, et al. Using Skype as an alternative for residency selection interviews. J Grad Med Educ. 2013;5:503-5.

33. Hariton E, Bortoletto P, Ayogu N. Using video-conference interviews in the residency application process. Acad Med. 2017;92(6):728-9.

34. Hariton E, Bortoletto P, Ayogu N. Residency interviews in the 21st century. J Grad Med Educ. 2016;8:322-4.

35. Virtual interviews: Tips for program directors. Available at: https://www.aamc.org/system/files/2020-05/Virtual_Interview_Tips_for_Program_Directors_05142020.pdf. Accessed May 15, 2020.

36. Virtual interviews: Tips for interviewers. Available at: https://www.aamc.org/system/files/2020-05/Virtual_Interview_Tips_for_Interviewers_05142020_0.pdf. Accessed May 15, 2020.

37. Compendium of resources for the implementation of recommendations in the final report and recommendations for medical education institutions of LCME-accredited, U.S. osteopathic, and non-U.S. medical school applicants. Available at: https://www.aamc.org/system/files/2020-05/covid19_Compendium_Resources_05112020.pdf. Accessed May 15, 2020.

38. Platt MP, Akhtar-Khavari V, Ortega R, et al. The interactive candidate assessment tool: a new way to interview residents. Otolaryngol Head Neck Surg. 2017;156(6):1150-1153.

39. Visiting elective scholarship programs for underrepresented minorities. Available at: https://www.saem.org/cdem/resources/medical-student-resources/underrepresented-minority-scholarships. Accessed April 24, 2020.

40. AAMC standardized video interview evaluation summary. Available at: https://students-residents.aamc.org/applying-residency/article/svi-evaluation-summary/. Accessed April 24, 2020.

41. Weber L. Today’s personality tests raise the bar for job seekers. Available at: https://www.wsj.com/articles/a-personality-test-could-stand-in-the-way-of-your-next-job-1429065001?ns=prod/accounts-wsj. Accessed April 23, 2020.

42. Merlo LJ, Matveevskii AS. Personality testing may improve resident selection in anesthesiology programs. Med Teach. 2009;31(12):e551-4.

43. Cook CJ, Cook CE, Hilton TN. Does emotional intelligence influence success during medical school admissions and program matriculation?: a systematic review. J Educ Eval Health Prof. 2016;13:40.

44. Dore KL, Reiter HI, Kreuger S, et al. CASPer, an online pre-interview screen for personal/professional characteristics: prediction of national licensure scores. Adv in Health Sci Educ. 2017;22:327-36.

45. Shipper ES, Maer LM, Merrell SB, et al. Pilot evaluation of the Computer-Based Assessment for Sampling Personal Characteristics test. J Surg Res. 2017;215:211-8.

46. Gardner AK, Dunkin BJ. Evaluation of validity evidence for personality, emotional intelligence, and situational judgment tests to identify successful residents. JAMA Surg. 2018;153(5):409-16.

47. Gardner AK, Cavanaugh KJ, Willis RE, et al. Can better selection tools help us achieve our diversity goals in postgraduate medical education? Comparing use of USMLE Step 1 scores and situational judgment tests at 7 surgical residencies. Acad Med. 2020;95(5):751-7.

48. Gardner AK, Cavanaugh KJ, Willis RE, et al. If you build it, will they come? Candidate completion of preinterview screening assessments. J of Surg Educ. 2019;76(6):1534-38.