Author	Affiliation
Moises Gallegos, MD, MPH	Stanford University School of Medicine, Department of Emergency Medicine, Palo Alto, California
Adaira Landry, MD, MEd	Harvard Medical School, Department of Emergency Medicine, Boston, Massachusetts
Al’ai Alvarez, MD	Stanford University School of Medicine, Department of Emergency Medicine, Palo Alto, California
Dayle Davenport, MD	Rush University Medical Center, Department of Emergency Medicine, Chicago, Illinois
Martina T. Caldwell, MD, MS	Henry Ford Health System, Department of Emergency Medicine, Detroit, Michigan
Melissa Parsons, MD	University of Florida College of Medicine – Jacksonville, Department of Emergency Medicine, Jacksonville, Florida
Michael Gottlieb, MD	Rush University Medical Center, Department of Emergency Medicine, Chicago, Illinois
Sreeja Natesan, MD	Duke University School of Medicine, Department of Emergency Medicine, Durham, North Carolina

Background
Critical appraisal
Pre-interview preparedness strategies
Invitation and interview strategies
Post-interview strategies
Limitations
Conclusion

BACKGROUND

Efforts to advance diversity, equity, and inclusion (DEI) in medicine are dependent on deliberate attention toward residency recruitment.^1,2 The benefits of diversity in medicine are well known, including cultural sensitivity and competence, expanded delivery of healthcare in low-resource settings, and improved intellectual discussion within training cohorts.³ It is important to develop pipeline pathways for racial/ethnic UIM groupsa to increase the number of residency candidates.⁴ A study of the 20 largest Accreditation Council for Graduate Medical Education (ACGME) resident specialties observed that no residency program represented Black or Latino populations at comparable rates to the United States population.⁵ For emergency medicine (EM), it was predicted to take 54 years to achieve a similar representative proportion of the Latino population.⁵ That study emphasizes the continued need to support diversity, equity, and inclusion through improved parity in medical clinician representation. While this article focuses on recruitment of UIM applicants, there are other populations that do not fall under the strict definition of UIM that are at risk of underrepresentation or exclusion, such as students from rural, LGBTQ+, or religious communities, and special consideration for these applicants should also be taken.^6-8

Fundamental to UIM recruitment is recognition and mitigation of bias. While bias exists at all stages of recruitment, it is most notable in high-impact metrics such as the United States Medical Licensing Examination (USMLE) Step examinations, Standardized Letters of Evaluation (SLOE), the Medical Student Performance evaluations (MSPE), and induction into the Alpha Omega Alpha (AOA) Honor Medical Society.^9-11 Holistic review emphasizes balanced consideration of these metrics with additional components such as personal statement, extracurricular activities, and lived experience.^12,13 The impact of bias in the UIM recruitment process is consequential: applicants may be disregarded during pre-interview screening or ranked lower post-interview,⁹ further hindering the mission to increase diversity in medicine.²

Unfortunately, there is no standardized process to increase holistic review and minimize bias in resident selection. Using current literature, we outline best practice recommendations for implementing holistic review and mitigating bias in residency recruitment to promote DEI.

CRITICAL APPRAISAL

This is the ninth article in a series of evidence-based best practice reviews from the Council of Residency Directors in Emergency Medicine (CORD) Best Practices Subcommittee.^14-20 With the guidance of a medical librarian, we used MEDLINE via PubMed to search for articles published from inception to February 4, 2021, using keywords and medical subheadings focused on diversity, equity, and inclusion (Appendix). We searched the bibliographies of relevant articles for any additional studies. The search yielded 2080 articles, of which 115 were deemed to be relevant for inclusion in this review. Articles were independently screened by two authors who searched for those that address holistic review and bias in recruitment and interviewing. We included articles if either author recommended the relevance of the study. When supporting data was not available, recommendations were made based on the authors’ combined experience and consensus opinion. According to the Oxford Center for Evidence-Based Medicine criteria, we provide the level and grade of evidence for each best practice statement (Table 1).²¹ This manuscript was reviewed by the CORD Best Practices Subcommittee and posted to the CORD website for peer review and feedback from the CORD medical education community.

PRE-INTERVIEW PREPAREDNESS STRATEGIES

Defining DEI Goals for Recruitment Season

Increasing DEI efforts and improving the recruitment of UIM residency applicants requires purposeful planning at programmatic, departmental, and institutional levels.^2,6,22-25 Advanced preparation ahead of the recruitment season can facilitate holistic review and contribute to decreasing bias in the selection process. Residency leadership should first define what diversity means for the program, including measurable outcomes and consequences of not achieving these results.^10,24,26 A statement of purpose can aid tracking and accountability of progress toward set goals.^26-28 Acknowledgment of DEI in mission statements demonstrates residency program investment in diversity as a core value.^2,6,24 There should be a clear call for increased representation of UIM residents, including a definition of the role the institution will take.^27,29 With early and clear communication of a commitment to, and goals for, diversity recruitment, residency programs can position themselves for success throughout the interview and selection process.

Assess Program Readiness

Commitment to increasing the number of UIM residents is defined by the internal discussions and actions that address the value of diversity, not simply match results.¹² Recruiting diverse residents should be included as part of a program’s strategic plan.^24,31 Support from the department chair and organizational leadership is key to the success of diversity initiatives.^{2,22,24,30,32,33} Programs should undergo an internal review process of current culture toward and readiness to enact targeted recruitment efforts for DEI.^6,7 To achieve meaningful success for DEI in recruitment and departmental attitudes, programs need to embrace and foster an environment of change.^31,34 The Association of American Medical Colleges (AAMC) presents a four-step process for assessing institutional culture and climate (Table 2).³⁵

Programs can also complete diversity engagement surveys to assess an institution’s level of engagement and inclusion, and perceptions within the group.^6-8 Programs should work to increase awareness, interest, and engagement in DEI efforts through department-wide educational sessions.^2,33 Programs should highlight how they will foster the career and academic interests of UIM trainees.^36,37 Support should be proactive, such as assigning resident mentors early, asking UIM trainees about individual needs, and providing early in-service exam preparation for all matriculating residents with marginal USMLE and other assessment scores.² Programs should avoid blindly targeting UIM trainees with services such as test prep, however, as this can reinforce bias and stereotypes.

Formation of Diversity Committees

Programs should create diversity committees with an understanding of program goals and objectives for the recruitment of UIM applicants. Valuing UIM status during interview screening and selection and greater UIM faculty representation is a program characteristic associated with higher resident diversity.^38-40 Committee members should be included in all planning phases of recruitment and should include UIM and non-UIM faculty, residents, and staff.^11,25,50,51

The formation of a diversity recruitment committee can be impactful.^23,25 In just one year, the Denver Health Emergency Medicine residency program doubled the number of UIM applicants interviewed, relying on a diversity committee to inform recruitment practices.⁵⁰ Similarly, the Highland Emergency Medicine residency program in Oakland, CA, experienced a doubling of diversity representation in their residency group after implementation of recruitment initiatives spearheaded by a diversity committee.⁵² A diversity committee can have immediate and measurable impacts on UIM recruitment. It should be reiterated, however, that success in recruiting UIM candidates is most predicated upon the creation of a welcoming, supportive, and inclusive culture at the program, not just match statistics.

Representation of UIM faculty is integral to recruiting UIM resident applicants.^20,38-40 Recruitment and retention of UIM faculty are discussed in a separate review as part of the CORD Best Practices series.²⁰ Mindful attention should be made to not assign UIM faculty with work that is unaligned with their personal interests, underrecognized by promotions committee, and uncompensated despite the time investment. It is important to recognize the potential for UIM individuals to experience a “minority tax,” or disproportionate burden of work.¹

Accessing UIM Applicants

It is difficult to recruit diverse candidates, however, if they do not exist within the applicant pool. Recruitment can take on a variety of forms depending on the target populations and the desired messaging.^44,53 Dedicated outreach to UIM students can lead to increased interest in a given program.^6,29,54-56 Reaching UIM applicants requires more than just simple communication as programs need to demonstrate a commitment to diversity and service.^29,57,60 Programs should display their commitment, efforts, and successes with DEI efforts on their websites,^2,6,25 and should provide contact information for a point person, faculty or staff, to address questions about DEI within the program.

Recruitment can be enhanced through early enrichment and pathway programming.^57-59 In addition to medical school interest groups, there may be a benefit to connecting with pre-medical organizations at the university level,²³ and creating enrichment programs as early as the elementary and high school levels.^2,29 The UIM applicants may not have personal or professional networks to initially steer them toward medical school and subsequently assist with residency applications.⁶⁰ Early outreach can occur by way of faculty presence at dedicated conferences sponsored by UIM student organizations, and faculty volunteering as mentors through sponsored programs.^2,6,25,29,51

An underused tool in UIM recruitment is a formal collaboration with minority medical student organizations,⁶¹ It should be clear, however, that a lack of attendance or participation with these groups should not affect the applicant’s consideration or rank-list position. The Student National Medical Association proposes a five-phase recruitment strategy using minority medical student organizations to increase the number of UIM students entering medical school. Increased matriculation of UIM medical students will directly contribute to the applicant pool for residency and the strategies suggested can be adapted to residency recruitment.⁶¹

 

Best Practice Recommendations

 

Define clear and prioritized goals for diversity-related residency recruitment. (Level 5, Grade D)

 

Assess program readiness to implement diversity-related recruitment and support UIM trainees that match. (Level 4, Grade C)

 

Mitigate bias through inclusion of bias training and predetermined scoring rubrics for screening, interviews, and ranking. (Level 3, Grade B)

 

Create DEI committees to inform and steer diversity-related recruitment. (Level 3, Grade B)

 

Ensure representation of UIM faculty in the screening, interview, and selection process but avoiding tasking UIM faculty with too much during the recruitment cycle. (Level 5, Grade D) (Level 4, Grade C)

 

Begin recruitment of UIM applicants early through directed and expanded efforts such as enrichment, outreach, and pathway programming. (Level 5, Grade D)

 

Collaborate with minority student groups in early mentorship and advisory programs for UIM applicants. (Level 5, Grade D)

 

INVITATION AND INTERVIEW STRATEGIES

At every step of the process, programs should approach recruitment with a lens to promote diversity, ensure inclusion, support equity, and uncover and address biased and racist practices.⁶² Programs should go beyond simply recognizing bias, aiming to actively mitigate it, aligning with the ACGME Common Program Requirements to improve diversity.⁶³ Individuals involved in recruitment, interviewing, and ranking should complete implicit bias training,^{10,30,33,40-43} and programs should conduct sensitivity discussions and self-reflection to promote learning about biases.⁴⁴ Interviewers should undergo training and preparation as a group to decrease variability and bias in applicant evaluations.⁴⁸

Approach to Holistic Review

A standardized holistic review process that aligns with each institution’s mission, vision, and values will shift the focus away from a traditionally metrics-driven selection process to a more inclusive process. Holistic review focuses on the importance of the applicant and their stories, rather than achieving certain demographic numbers.^{2,12,28,62,64,65} As there is no universal approach to holistic review, it is important to recognize that the process is subject to bias as program leadership determines its implementation.^13,24

Holistic review has more readily made its way into undergraduate medical education.⁶⁶ Residency recruitment continues to rely heavily on performance and assessment metrics.^67,68 Principles of holistic review in medical school admissions can be extrapolated to inform residency recruitment. The AAMC offers a holistic review primer for program directors to identify experiences, attributes, competencies, and metrics grounded in a program’s mission.^69,70 The AAMC Advancing Holistic Principles Advisory Committee promotes core principles for holistic review (Figure 1).⁶⁹

Holistic review addresses the need to balance personal attributes with performance and aptitude.^6,7,12,71 It shifts the practice of preferentially valuing academic achievement-based metrics to considering the entire application.^{2,6,7,10,64,71} In this approach, numerical benchmarks, such as test scores and class rank, do not prematurely eliminate or accelerate applicants prior to the evaluation of the entire application.²⁵ There have been different models of holistic review suggested in the medical education literature.^{1,2,43,48,55,62} Review committees should begin with a self-audit of current practices and make appropriate changes that best fit the program’s goals.⁶ Notably, the search for applicants who “align well” with a program, a concept known as “fit,” introduces bias that must be acknowledged and addressed.⁴³ Figure 2 summarizes qualities and characteristics proposed for holistic review in place of traditional metrics. Over reliance on metrics such as exam scores and AOA status will impact recruitment of candidates who are underrepresented in medicine or systematically disadvantaged.^{6,11,12,26,72}

The impact of holistic review on in-service and medical board examination pass rates is not yet well documented. Nehemiah et al demonstrated no significant change for surgical in-training exam scores after the implementation of holistic review and an accompanying increase in UIM diversity.⁶⁵ Aibana et al involved stakeholders and committee members in deciding a new Step 1 threshold unlikely to affect board passing rates.⁷⁸ Below we explore the value and harm of core components of the application and strategies to optimize a holistic review.

Applicant Selection for Residency Interview

No single, uniformly accepted evaluation system exists for offering residency interviews, thereby allowing for subjectivity, bias, and inconsistency when selecting candidates.^7,11,26,74 Scoring rubrics for all phases of recruitment, from interview selection to ranking, should be decided ahead of time.^47,75-78 Rubrics should reflect the level of importance that experiences, attributes, competencies, or metrics represent for a program, and can help de-emphasize metrics that can bias selection against the UIM applicant.^77,79 The AAMC provides a guided activity for Applicant Criteria Identification and Prioritization as part of its holistic review capacity building resources.⁸⁰ Table 3 highlights examples of scoring rubrics that incorporate concepts of holistic review.

Clinical Grades and Letters of Recommendation

For EM applicants, their clinical evaluation hinges on the sub-internship SLOE. This summative form provides a rating and ranking of the student as well as descriptive commentary of their performance. The SLOE in EM is a step toward decreasing bias through structured reporting of performance assessment; however, it is not entirely free from it.^82,83 Narrative evaluations for men are more supportive than for women and UIM students.^84,85 The UIM students with similar clerkship grades had more negative comments and fewer positive comments compared to their White counterparts.⁸⁶ The SLOE is often perceived as the most objective assessment of the student’s clinical competency and potential as a rising resident; however, the influence of bias in EM SLOE rankings and language has been insufficiently studied.^87,88

Traditional letters of recommendation are often reflective of a student’s network and support system, which may be more difficult to develop for women and UIM students. They are also subject to language bias. Women and UIM students are more likely to be characteried by grindstone words such as “organized” or “hardworking” as compared to superlatives reflecting high achievement potential used in letters for White male students.^{9,62,84,87,89-91}

Medical Student Performance Evaluation

The MSPE is a comprehensive review of a student’s interests, activities, and, most notably, clinical performance. Some schools provide rankings of the student in comparison to their peers. Only 2% of medical schools provide comparative data consistently in all five appendices (pre-clinical courses, clerkships, professional attributes, overall performance, and medical school information page).^88,92 Furthermore, the MSPE can be fraught with the use of biased language and descriptions based on the applicant’s gender⁸⁴ and race/ethnicity.^85,90,93 White students were more likely to be described as “outstanding,” “exceptional,” and “best.”⁹³ In contrast, the word “competent” was more often used to describe Black and Hispanic students but was only perceived to carry a positive connotation 37% and 33% of the time, respectively.⁹³

Alpha Omega Alpha

Acceptance into AOA is often used to signal academic excellence. However, awards and accolades have been shown to be given less often to UIM students, and not all institutions participate in AOA.⁸⁸ Membership in AOA was six times more likely for White students than for Black students.⁹⁴ Use of this award as a differentiating factor can be discriminatory and disadvantage UIM students.⁹⁵

Standardized Exams

Standardized examinations have been shown to predict academic success on in-training and board exams but not to predict success in residency or an ability to provide safe and quality care overall.^46,74,96,97 Despite this, USMLE Step 1 scores are commonly used as a screening tool.^9,11,62,97 The USMLE is subject to systemic biases associated with any standardized test, such as accessibility and affordability of test prep. Given that UIM applicants have lower USMLE scores on average,^98,99 an over-reliance on test scores as a screening tool can lead to UIM applicants being excluded from a more in-depth review that may have otherwise earned them an interview invitation.^{11,12,26,52,62,72,98,100} In 2020, it was announced that the USMLE Step 1 exam will be scored as pass or fail based on previous evidence of poor utility. The USMLE Step 2, as well as other standardized exams such as the Comprehensive Osteopathic Medical Licensing Examination, will still report numerical scores.¹⁰¹

Personal Statement

Personal statements allow applicants to share stories of inspiration, resilience, and future goals,¹⁰² enabling them to showcase their interests and skills. The value placed on the personal statement is variable, however,¹⁰³ and may introduce bias such as gender-based differences in writing.¹⁰⁴ Personal statements are not effective in predicting medical student performance,¹⁰⁵ and utility for residency selection is unclear.

Interview Process and Considerations

The interview allows for scoring on behavioral-related metrics, such as grit, distance traveled in life experience, and emotional intelligence.¹⁰⁶ Steps should be taken to standardize the interview process as much as possible to minimize bias.^46,107 The interview should follow a set structure. A standard pool of questions should be determined ahead of time and interviewers can be assigned specific questions.⁷⁸ Interviewers should receive the same instructional training and have access to the same amount of information from applications.^30,44,54

As performance metrics can bias perceptions,¹⁰⁸ programs should consider blinded interviews in which exam scores are not provided to interviewers.¹⁰⁹ Interviewers should represent a diverse pool of faculty, residents, and staff and should receive protected time to support the commitments needed for thoughtful interviewing.³⁰ The COVID-19 pandemic required that the 2020 recruitment cycle be done virtually. Programs should decide whether they will offer virtual or in-person interviews, and all interviews should be done in the same format to avoid bias.¹¹⁰

Travel considerations

The UIM trainees experience greater financial challenges from the high cost of medical education.^111,112 Digital interviewing contributed to less financial burden from traveling. The emphasis on away rotations in EM, however, creates a potential hurdle for UIM applicants. Clerkship diversity scholarships have been shown to correlate with increased residency diversity in EM, especially for Black and Latino residents.¹¹³ Scholarships and financial assistance can attract UIM applicants who otherwise would be unable to rotate at, and may not have considered, a particular program. Funding for UIM recruitment efforts demonstrate institutional commitment to diversity recruitment.^50,51

Consideration for Historically Black Colleges and Universities

Students from historically Black colleges and universities (HBCU) often rely on away rotations for their sub-internship experience as their home institution may have limited exposure to EM or lack an emergency department. The ability to fund travel and lodging limits the options of rotation location.¹¹² If students are unable to travel due to financial restrictions, their opportunity to be exposed to new clinical environments and potential mentors is limited.^50,51,113 Partnerships between HBCUs and neighboring EM residency programs can help promote diversity.¹¹⁴

 

Best Practice Recommendations

 

Apply an equity lens to each step of the recruitment process to expose existing bias and allow for correction. (Level 5, Grade D)

 

Holistic review should be applied equitably across all applicants. (Level 4, Grade C)

 

Identify characteristics for holistic review that align with a program’s mission, vision, values. (Level 4, Grade C)

 

Avoid screening applicants solely on standardized examination scores or grades. (Level 3b, Grade C)

 

Standardize the structure of interviews in terms of logistics and questions asked. (Level 4. Grade C)

 

Ensure UIM faculty visibility and allow networking during the interview day or through structured asynchronous opportunities to engage with DEI topics. (Level 4, Grade C)

 

Partner with HBCUs and neighboring EM residency programs to help further promote diversity within the specialty. (Level 4, Grade C)

 

POST-INTERVIEW STRATEGIES

Ranking Considerations

The ranking process should be collaborative and conducted in a safe space with limited external influence from those not involved in the recruitment process.¹¹⁵ Members of the ranking group should be diverse in interests and backgrounds and often include the residency leadership team as well as additional core faculty, residents, and administrative staff. Individuals participating in rank meetings should be informed about characteristics identified of value to the residency program and used throughout the selection process.^11,25,50,51 One voice should not dominate, and there should be group discussion prior to deciding a rank position. A temporary ranking meeting might occur immediately following the day of an interview. For fairness to all candidates, the final ranking should begin at the conclusion of the interview season.^12,116

Second-look Opportunities and Post-interview Communication

There should be increased visibility and opportunity to network with faculty (both UIM and non-UIM) involved in recruitment and retention of UIM applicants.^25,50 If UIM recruitment faculty are not available during scheduled interview dates, asynchronous opportunities to discuss DEI within the program should be offered.⁵¹ A second-look visit can be organized to facilitate this.^23,50

Targeted recruitment of UIM applicants may benefit from ongoing dialogue throughout the interview process. Communication such as “thank you” emails should be done with heavy consideration of the potential to mislead or falsely assure an applicant. Caution should be taken when reaching out to UIM applicants to not breach National Resident Matching Program regulations. Programs should be clear about expectations for post-interview communication and should designate a pointperson for ongoing communication.

 

Best Practice Recommendations

 

Select diverse members for the rank committee. (Level 4, Grade C)

 

Conduct the rank meeting in a safe, private space with collaborative discussion. (Level 4, Grade C)

 

Inform committee members about the characteristics identified as valuable to the program before the ranking process. (Level 5, Grade D)

 

Ensure ranking is done based on scores from the predefined rubrics for screening and interviewing. (Level 5, Grade D)

 

Offer second look visits (on-site or virtually) to network with UIM faculty and discuss DEI within the program. (Level 4, Grade C)

 

Define clear expectations for follow up and designate a point person for communication. (Level 4, Grade C)

 

LIMITATIONS

The scope of this article was limited to holistic review and the impact of bias on recruitment in residency training. There are other topics (eg, pipeline/pathway efforts, faculty recruitment and retention) regarding DEI that will be covered in other reviews. While we performed a comprehensive search guided by a medical librarian in conjunction with expert consultation and bibliographic review, it is possible that we may have missed pertinent articles. In several instances, high-quality data was limited or lacking. In these instances, we relied upon expert opinion and group consensus for the best practice recommendations. Literature specific to EM and within graduate medical education is more limited; therefore, we included relevant articles from other medical specialties and health-related professions. We believe that EM, as a specialty, can learn from other colleagues across many disciplines.

CONCLUSION

Holistic review and the mitigation of bias are essential steps in the purposeful recruitment and selection of applicants who are underrepresented in medicine. Our article presents best practice recommendations for residency programs to prepare for and implement application review, applicant interviewing, and trainee selection in support of diversity, equity, and inclusion.

Footnotes

Defined by the Association of American Medical Colleges (AAMC), under-represented minority (URM) was a term reflecting the racial groups of Black, Mexican-American, mainland Puerto Rican, and Native American. (American Indian and natives of Alaska and Hawaii). This was expanded to “underrepresented in medicine”(URiM or UIM) in 2003 to encompass the racial and ethnic populations within medicine who are underrepresented when compared to their respective numbers in the context of the greater population. For this article, although we recognize the use of URiM or UIM interchangeably, for consistency, we use the term UIM.

Section Editor: Jeffrey Druck, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

Address for Correspondence: Moises Gallegos, MD, MPH, Stanford University School of Medicine, Department of Emergency Medicine, 900 Welch Road Suite 350, Palo Alto, CA 94304. Email: moisesg@stanford.edu. 5 / 2022; 23:345 – 352

Submission history: Revision received August 13, 2021; Submitted March 30, 2022; Accepted March 15, 2022

CORD Best Practice Committee 2020–2021

Michael Gottlieb, MD – Co-Chair

Rush University Medical Center

Sreeja Natesan, MD – Co-Chair

Duke University

 

John Bailitz, MD

Northwestern University, Feinberg School of Medicine

 

Brian Barbas, MD

Loyola University Medical Center

 

Jennie Buchanan, MD

Denver Health Medical Center

 

Richard Byyny, MD

Denver Health Medical Center

 

Guy Carmelli, MD

University of Massachusetts Medical School

 

Molly Estes, MD

Loma Linda University

 

Katja Goldflam, MD

Yale University

 

Andrew Grock, MD

University of California – Los Angeles

 

Jaime Jordan, MD

University of California – Los Angeles

 

Andrew King, MD

The Ohio State University

 

Krystin Miller, MD

The Ohio State University

 

Melissa Parsons, MD

University of Florida – Jacksonville

 

Alexander Sheng, MD

Boston Medical Center

 

Brian Wood, MD

St. Joseph’s Medical Center

 

REFERENCES

1. Arno K, Davenport D, Shah M, et al. Addressing the urgent need for racial diversification in emergency medicine. Ann Emerg Med. 2021;77(1):69-75.

2. Heron SL, Lovell EO, Wang E, et al. Promoting diversity in emergency medicine: summary recommendations from the 2008 Council of Emergency Medicine Residency Directors (CORD) Academic Assembly Diversity Workgroup. Acad Emerg Med. 2009;16(5):450-3.

3. Whitla DK, Orfield G, Silen W, et al. Educational benefits of diversity in medical school: a survey of students. Acad Med. 2003;78(5):460-6.

4. Abelson JS, Symer MM, Yeo HL, et al. Surgical time out: Our counts are still short on racial diversity in academic surgery. Am J Surg. 2018;215(4):542-8.

5. Bennett CL, Yiadom MYAB, Baker O, et al. Examining parity among Black and Hispanic resident physicians. J Gen Intern Med. 2021;36(6):1722-1725.

6. Gonzaga AMR, Appiah-Pippim J, Onumah CM, et al. A framework for inclusive graduate medical education recruitment strategies: meeting the ACGME Standard for a Diverse and Inclusive Workforce. Acad Med. 2020;95(5):710-6.

7. Thomas BR, Dockter N. Affirmative Action and holistic review in medical school admissions: Where we have been and where we are going. Acad Med. 2019;94(4):473-6.

8. Person SD, Jordan CG, Allison JJ, et al. Measuring diversity and inclusion in academic medicine: the Diversity Engagement Survey. Acad Med. 2015;90(12):1675-83.

9. Angus SV, Williams CM, Stewart EA, et al. Internal medicine residency program directors’ screening practices and perceptions about recruitment challenges. Acad Med. 2020;95(4):582-9.

10. Boatright D, Branzetti J, Duong D, et al. Racial and ethnic diversity in academic emergency medicine: How far have we come? Next steps for the future. AEM Educ Train. 2018;2(Suppl Suppl 1):S31-9.

11. Clayborne EP, Martin DR, Goett RR, et al. Diversity pipelines: the rationale to recruit and support minority physicians. J Am Coll Emerg Physicians Open. 2021;2(1):e12343.

12. DeBenedectis CM, Heitkamp DE, England E, et al. A program director’s guide to cultivating diversity and inclusion in radiology residency recruitment. Acad Radiol. 2020;27(6):864-7.

13. Nakae S, Porfeli EJ, Davis D, et al. Enrollment management in undergraduate medical school admissions: a complementary framework to holistic review for increasing diversity in medicine. Acad Med. 2021;96(4):501-6.

14. Gottlieb M, King A, Byyny R, et al. Journal club in residency education: an evidence-based guide to best practices from the Council of Emergency Medicine Residency Directors. West J Emerg Med. 2018;19(4):746-55.

15. Estes M, Gopal P, Siegelman JN, et al. Individualized interactive instruction: a guide to best practices from the Council of Emergency Medicine Residency Directors. West J Emerg Med. 2019;20(2):363-8.

16. Parsons M, Bailitz J, Chung AS, et al. Evidence-based interventions that promote resident wellness from the Council of Emergency Residency Directors. West J Emerg Med. 2020;21(2):412-22.

17. Natesan S, Bailitz J, King A, et al. Clinical teaching: an evidence-based guide to best practices from the Council of Emergency Medicine Residency Directors. West J Emerg Med. 2020;21(4):985-98.

18. Wood DB, Jordan J, Cooney R, et al. Conference didactic planning and structure: an evidence-based guide to best practices from the Council of Emergency Medicine Residency Directors. West J Emerg Med. 2020;21(4):999-1007.

19. Chathampally Y, Cooper B, Wood DB, et al. Evolving from morbidity and mortality to a case-based error reduction conference: evidence-based best practices from the Council of Emergency Medicine Residency Directors. West J Emerg Med. 2020;21(6):231-41.

20. Davenport D, Alvarez A, Natesan S, et al. Faculty recruitment, retention, and representation in leadership: an evidence-based guide to best practices for diversity, equity, and inclusion from the Council of Residency Directors in Emergency Medicine. West J Emerg Med. 2022;23(1):62-71.

21. Howick J, Chalmers I, Glasziou P, et al. Explanation of the 2011. Oxford Centre for Evidence-Based Medicine (OCEBM) levels of evidence (Background Document). Available at: https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence. Accessed August 13, 2021.

22. Raphael JL, Giardino AP, Harris T, et al. Perceptions revisited: pediatric chief resident views on minority housestaff recruitment and retention in pediatric residency programs. J Natl Med Assoc. 2014;106(1):58-68.

23. Pierre JM, Mahr F, Carter A, et al. Underrepresented in medicine recruitment: rationale, challenges, and strategies for increasing diversity in psychiatry residency programs. Acad Psychiatry. 2017;41(2):226-32.

24. Vick AD, Baugh A, Lambert J, et al. Levers of change: a review of contemporary interventions to enhance diversity in medical schools in the USA. Adv Med Educ Pract. 2018;9:53-61.

25. Lewis T, Tolbert J, Jones BL. Increasing resident racial and ethnic diversity through targeted recruitment efforts. J Pediatr. 2020;216:4-6.

26. Jarman BT, Kallies KJ, Joshi ART, et al. Underrepresented minorities are underrepresented among general surgery applicants selected to interview. J Surg Educ. 2019;76(6):e15-23.

27. Lim RF, Luo JS, Suo S, et al. Diversity initiatives in academic psychiatry: applying cultural competence. Acad Psychiatry. 2008;32(4):283-90.

28. Ko M, Ton H. The not underrepresented minorities: Asian Americans, diversity, and admissions. Acad Med. 2020;95(2):184-9.

29. Toney M. The long, winding road: one university’s quest for minority health care professionals and services. Acad Med. 2012;87(11):1556-61.

30. Mateo CM, Williams DR. More than words: a vision to address bias and reduce discrimination in the health professions learning environment. Acad Med. 2020;95(12S Addressing harmful bias and eliminating discrimination in health professions learning environments):S169-77.

31. Deas D, Pisano E, Mainous A, et al. Improving diversity through strategic planning: a 10-year (2002–2012) experience at the Medical University of South Carolina. Acad Med. 2012;87(11):1548-55.

32. Heron S, Haley L. Diversity in emergency medicine–a model program. Acad Emerg Med. 2001;8(2):192-5.

33. Mmeje O, Price EAN, Johnson TRB, et al. Galvanizing for the future: a bottom-up departmental approach to diversity, equity, and inclusion. Am J Obstet Gynecol. 2020;223(5):715e1-715.e7.

34. Rymer JA, Frazier-Mills CG, Jackson LR, et al. Evaluation of women and underrepresented racial and ethnic group representation in a general cardiology fellowship after a systematic recruitment initiative. JAMA Netw Open. 2021;4(1):e2030832.

35. Assessing Institutional Culture and Climate Webcast Supplemental Guide 2013. Assoc Am Med Coll. 2013;27.

36. Sanson-Fisher RW, Williams N, Outram S. Health inequities: the need for action by schools of medicine. Med Teach. 2008;30(4):389-94.

37. Adetoye M. Recruitment and retention: a guideline to help build trust with minority candidates. Acad Med. 2019;94(6):754.

38. Boatright D, Tunson J, Caruso E, et al. The impact of the 2008 Council of Emergency Residency Directors (CORD) Panel on Emergency Medicine Resident Diversity. J Emerg Med. 2016;51(5):576-83.

39. Jarman BT, Borgert AJ, Kallies KJ, et al. Underrepresented minorities in general surgery residency: analysis of interviewed applicants, residents, and core teaching faculty. J Am Coll Surg. 2020;231(1):54-8.

40. Okike K, Phillips DP, Johnson WA, et al. Orthopaedic faculty and resident racial/ethnic diversity is associated with the orthopaedic application rate among underrepresented minority medical students. J Am Acad Orthop Surg. 2020;28(6):241-7.

41. Johnson TJ, Ellison AM, Dalembert G, et al. Implicit bias in pediatric academic medicine. J Natl Med Assoc. 2017;109(3):156-63.

42. Capers Q. How clinicians and educators can mitigate implicit bias in patient care and candidate selection in medical education. ATS Sch. 2020;1(3):211-7.

43. Shappell E, Schnapp B. The F word: how “fit” threatens the validity of resident recruitment. J Grad Med Educ. 2019;11(6):635-6.

44. Spottswood SE, Spalluto LB, Washington ER, et al. Design, implementation, and evaluation of a diversity program for radiology. J Am Coll Radiol. 2019;16(7):983-91.

45. Cleveland Manchanda EC, Macias-Konstantopoulos WL. Tackling gender and racial bias in academic emergency medicine: the perceived role of implicit bias in faculty development. Cureus. 2020;12(11):e11325.

46. Spector AR, Railey KM. Reducing reliance on test scores reduces racial bias in neurology residency recruitment. J Natl Med Assoc. 2019;111(5):471-4.

47. Girod S, Fassiotto M, Grewal D, et al. Reducing implicit gender leadership bias in academic medicine with an educational intervention. Acad Med. 2016;91(8):1143-50.

48. Paterson QS, Hartmann R, Woods R, et al. A transparent and defensible process for applicant selection within a Canadian emergency medicine residency program. Can J Emerg Med. 2020;22(2):215-23.

49. Kiraly L, Dewey E, Brasel K. Hawks and doves: adjusting for bias in residency interview scoring. J Surg Educ. 2020;77(6):e132-7.

50. Tunson J, Boatright D, Oberfoell S, et al. Increasing resident diversity in an emergency medicine residency program: a pilot intervention with three principal strategies. Acad Med. 2016;91(7):958-61.

51. Winfield A, Schindlbeck M, Smith L. Additional strategies for underrepresented minority recruitment. Ann Emerg Med. 2020;75(2):313-4.

52. Garrick JF, Perez B, Anaebere TC, et al. The diversity snowball effect: the quest to increase diversity in emergency medicine: a case study of Highland’s emergency medicine residency program. Ann Emerg Med. 2019;73(6):639-47.

53. Vajapey S, Cannada LK, Samora JB. What proportion of women who received funding to attend a Ruth Jackson Orthopaedic Society meeting pursued a career in orthopaedics?. Clin Orthop Relat Res. 2019;477(7):1722-6.

54. Wusu MH, Tepperberg S, Weinberg JM, et al. Matching our mission: a strategic plan to create a diverse family medicine residency. Fam Med. 2019;51(1):31-6.

55. Van Voorhees AS, Enos CW. Diversity in dermatology residency programs. J Investig Dermatol Symp Proc. 2017;18(2):S46-s49.

56. Lin MP, Lall MD, Samuels-Kalow M, et al. Impact of a women-focused professional organization on academic retention and advancement: perceptions from a qualitative study. Acad Emerg Med. 2019;26(3):303-16.

57. Nivet MA, Berlin A. Workforce diversity and community-responsive health-care institutions. Public Health Rep. 2014;129(Suppl 2(Suppl 2)):15-8.

58. Muppala VR, Prakash N. Promoting physician diversity through medical student led outreach and pipeline programs. J Natl Med Assoc. 2021;113(2):165-168.

59. Kumar V, West DL. Bridging the equity gap. AJR Am J Roentgenol. 2019;213(4):785-91.

60. Lievens F. Diversity in medical school admission: insights from personnel recruitment and selection. Med Educ. 2015;49(1):11-4.

61. Rumala BB, Cason FD. Recruitment of underrepresented minority students to medical school: minority medical student organizations, an untapped resource. J Natl Med Assoc. 2007;99(9):1000-4.

62. Gerull KM, Enata N, Welbeck AN, et al. Striving for inclusive excellence in the recruitment of diverse surgical residents during COVID-19. Acad Med. 2021;96(2):210-2.

63. Common Program Requirements (Residency) ACGME. 2020. Available at: https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResidency2021.pdf. Accessed August 13, 2021.

64. Poole KG, Jordan BL, Bostwick JM. Mission drift: Are medical school admissions committees missing the mark on diversity?. Acad Med. 2020;95(3):357-60.

65. Nehemiah A, Roberts SE, Song Y, et al. Looking beyond the numbers: increasing diversity and inclusion through holistic review in general surgery recruitment. J Surg Educ. 2021;78(3):763-9.

66. Williams C, Kwan B, Pereira A, Moody E, et al. A call to improve conditions for conducting holistic review in graduate medical education recruitment. MedEdPublish. 2019;8(2):6.

67. Crane JT, Ferraro CM. Selection criteria for emergency medicine residency applicants. Acad Emerg Med. 2000;7(1):54-60.

68. Garber AM, Kwan B, Williams CM, et al. Use of filters for residency application review: results from the Internal Medicine In-Training Examination Program Director Survey. J Grad Med Educ. 2019;11(6):704-7.

69. Holistic review. Available at: https://www.aamc.org/services/member-capacity-building/holistic-review. Accessed August 13, 2021.

70. Applicant Criteria Identification and Prioritization. Available at: https://www.aamc.org/media/44576/download. Accessed August 13, 2021.

71. Patterson F, Roberts C, Hanson MD, et al. 2018 Ottawa consensus statement: Selection and recruitment to the healthcare professions. Med Teach. 2018;40(11):1091-101.

72. Damp JB, Cullen MW, Soukoulis V, et al. Program directors survey on diversity in cardiovascular training programs. J Am Coll Cardiol. 2020;76(10):1215-22.

73. Witzburg RA, Sondheimer HM. Holistic review–shaping the medical profession one applicant at a time. N Engl J Med. 2013;368(17):1565-7.

74. Shantharam G, Tran TY, McGee H, et al. Examining trends in underrepresented minorities in urology residency. Urology. 2019;127:36-41.

75. Fassiotto M, Hamel EO, Ku M, et al. Women in academic medicine: measuring stereotype threat among junior faculty. J Womens Health. 2016;25(3):292-8.

76. Dossett LA, Mulholland MW, Newman EA, et al. Building high-performing teams in academic surgery: the opportunities and challenges of inclusive recruitment strategies. Acad Med. 2019;94(8):1142-5.

77. Marbin J, Rosenbluth G, Brim R, et al. Improving diversity in pediatric residency selection: using an equity framework to implement holistic review. J Grad Med Educ. 2021;13(2):195-200.

78. Aibana O, Swails JL, Flores RJ, et al. Bridging the gap: holistic review to increase diversity in graduate medical education. Acad Med. 2019;94(8):1137-41.

79. Barceló NE, Shadravan S, Wells CR, et al. Reimagining merit and representation: promoting equity and reducing bias in GME through holistic review. Acad Psychiatry. 2021;45(1):34-42.

80. Holistic Principles in Resident Selection: An Introduction. 2020. Available at: https://www.aamc.org/system/files/2020-08/aa-member-capacity-building-holistic-review-transcript-activities-GME-081420.pdf. Accessed August 13, 2021.

81. Handbook for Holistic Review and Best Practices for Enhancing Diversity in Residency and Fellowship Programs. 2017. Available at: https://wiki.library.ucsf.edu/download/attachments/456075181/Holistic%20Review%20Best%20Practices%209.21.17-FINAL.pdf?version=1&modificationDate=1592508759000&api=v2. Accessed August 13, 2021.

82. Love JN, Doty CI, Smith JL, et al. The Eeergency medicine group Standardized Letter of Evaluation as a workplace-based assessment: the validity is in the detail. West J Emerg Med. 2020;21(3):600-9.

83. Wilson D, Laoteppitaks C, Chandra S. A comparison of Standardized Letters of Evaluation for emergency medicine residency applicants. West J Emerg Med. 2020;22(1):20-5.

84. Hewett L, Lewis M, Collins H, et al. Gender bias in diagnostic radiology resident selection, Does it exist?. Acad Radiol. 2016;23(1):101-7.

85. Low D, Pollack SW, Liao ZC, et al. Racial/ethnic disparities in clinical grading in medical school. Teach Learn Med. 2019;31(5):487-96.

86. Lee KB, Vaishnavi SN, Lau SKM, et al. Cultural competency in medical education: demographic differences associated with medical student communication styles and clinical clerkship feedback. J Natl Med Assoc. 2009;101(2):116-26.

87. Miller DT, McCarthy DM, Fant AL, et al. The Standardized Letter of Evaluation narrative: differences in language use by gender. West J Emerg Med. 2019;20(6):948-56.

88. Pope AJ, Carter K, Ahn J. A renewed call for a more equitable and holistic review of residency applications in the era of COVID-19. AEM Educ Train. 2021;5(1):135-8.

89. Heath JK, Weissman GE, Clancy CB, et al. Assessment of gender-based linguistic differences in physician trainee evaluations of medical faculty using automated text mining. JAMA Netw Open. 2019;2(5):e193520.

90. Diaz T, Navarro JR, Chen EH. An institutional approach to fostering inclusion and addressing racial bias: implications for diversity in academic medicine. Teach Learn Med. 2020;32(1):110-6.

91. Filippou P, Mahajan S, Deal A, et al. The presence of gender bias in letters of recommendations written for urology residency applicants. Urology. 2019;134:56-61.

92. Boysen-Osborn M, Yanuck J, Mattson J, et al. Who to interview? Low adherence by U.S. medical schools to medical student performance evaluation format makes resident selection difficult. West J Emerg Med. 2017;18(1):50-5.

93. Ross DA, Boatright D, Nunez-Smith M, et al. Differences in words used to describe racial and gender groups in Medical Student Performance Evaluations. PLOS ONE. 2017;12(8):e0181659.

94. Boatright D, Ross D, O’Connor P, et al. Racial disparities in medical student membership in the Alpha Omega Alpha Honor Society. JAMA Intern Med. 2017;177(5):659.

95. Boatright D, O’Connor PG, E Miller J. Racial privilege and medical student awards: addressing racial disparities in Alpha Omega Alpha Honor Society membership. J Gen Intern Med. 2020;35(11):3348-51.

96. Lypson ML, Ross PT, Hamstra SJ, et al. Evidence for increasing diversity in graduate medical education: the competence of underrepresented minority residents measured by an intern objective structured clinical examination. J Grad Med Educ. 2010;2(3):354-9.

97. Prober CG, Kolars JC, First LR, et al. A plea to reassess the role of United States Medical Licensing Examination Step 1 scores in residency selection. Acad Med. 2016;91(1):12-5.

98. Williams M, Kim EJ, Pappas K, et al. The impact of United States Medical Licensing Exam (USMLE) Step 1 cutoff scores on recruitment of underrepresented minorities in medicine: a retrospective cross-sectional study. Health Sci Rep. 2020;3(2):e2161.

99. Youmans QR, Essien UR, Capers Q. A test of diversity – what USMLE pass/fail scoring means for medicine. N Engl J Med. 2020;382(25):2393-5.

100. Lucey CR, Saguil A. The consequences of structural racism on MCAT scores and medical school admissions: The past Is prologue. Acad Med. 2020;95(3):351-6.

101. Makhoul AT, Pontell ME, Kumar NG, et al. Objective measures needed — program directors’ perspectives on a pass/fail USMLE Step 1. N Engl J Med. 2020;382(25):2389-92.

102. Melamed KH, Wang T. The personal statement: not just a question of if it is valued, but should it be valued?. ATS Sch. 2020;1(1):5-7.

103. Hinkle L, Carlos WG, Burkart KM, et al. What do program directors value in personal statements? A qualitative snalysis. ATS Sch. 2020;1(1):44-54.

104. Ostapenko L, Schonhardt-Bailey C, Sublette JW, et al. Textual analysis of general surgery residency personal statements: topics and gender differences. J Surg Educ. 2018;75(3):573-81.

105. Patterson F, Knight A, Dowell J, et al. How effective are selection methods in medical education? A systematic review. Med Educ. 2016;50(1):36-60.

106. Pritchett EN, Pandya AG, Ferguson NN, et al. Diversity in dermatology: roadmap for improvement. J Am Acad Dermatol. 2018;79(2):337-41.

107. Hughes RH, Kleinschmidt S, Sheng AY. Using structured interviews to reduce bias in emergency medicine residency recruitment: worth a second look. AEM Educ Train. 2021;5(Suppl 1):S130-S134.

108. Stephenson-Famy A, Houmard BS, Oberoi S, et al. Use of the interview in resident candidate selection: a review of the literature. J Grad Med Educ. 2015;7(4):539-48.

109. Fuchs JW, Youmans QR. Mitigating bias in the era of virtual residency and fellowship interviews. J Grad Med Educ. 2020;12(6):674-7.

110. Deitte LA, Mian AZ, Esfahani SA, et al. Going virtual: redesigning the interview experience. J Am Coll Radiol. 2020;18(2):337-9.

111. Davis MG, Haas MRC, Gottlieb M, et al. Zooming In versus flying out: virtual residency interviews in the era of COVID-19. AEM Educ Train. 2020;4(4):443-6.

112. Toretsky C, Mutha S, Coffman J. Breaking barriers for underrepresented minorities in the health professions. 2018. Available at: https://healthforce.ucsf.edu/publications/breaking-barriers-underrepresented-minorities-health-professions. Accessed August 13, 2021.

113. Duong DK, Samuels EA, Boatright D, et al. Association between emergency medicine clerkship diversity scholarships and residency diversity. AEM Educ Train. ;5(3):e10547.

114. Goines J, Iledare E, Ander D, et al. A model partnership: mentoring underrepresented students in medicine (URiM) in emergency medicine. West J Emerg Med. 2021;22(2):213-7.

115. Young ME, Thomas A, Varpio L, et al. Facilitating admissions of diverse students: a six-point, evidence-informed framework for pipeline and program development. Perspect Med Educ. 2017;6(2):82-90.

116. Fryer JP, Corcoran N, George B, Wang E, Darosa D. Does resident ranking during recruitment accurately predict subsequent performance as a surgical resident?. J Surg Educ. 2012;69(6):724-30.