Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

The Standardized Letter of Evaluation (SLOE) is designed to assist emergency medicine (EM) residency programs to differentiate applicants and is considered very important in the decision to interview a candidate.¹ The SLOE includes the applicant’s qualifications for EM, a narrative assessment of cognitive and non-cognitive attributes, and the applicant’s rank in GA and ARL categories, as compared to other applicants.

We became interested in personality factors that may put an applicant at a disadvantage when we observed less enthusiasm for applicants described as “quiet” in their SLOE narratives. This reaction was consistent with studies showing that being described as quiet on internal medicine clerkship evaluations was interpreted as a negative attribute or “red flag,” even when the comment was not linked to performance.^2,3 Further, introverted medical students and residents scored lower than extraverts on subjective clinical evaluations but not on objective assessments.^4,5 Although some studies found extraversion to be related to aspects of success in and outside of medical careers,^6,7 others identified more nuanced measures of personality and non-cognitive attributes to be related to success (eg, conscientiousness,^6,8 emotional stability,⁸ and proactivity⁹), qualities possessed by both introverts and extraverts. We found no studies suggesting that quiet individuals were unsuccessful in, or unsuited for, EM careers.

While residency programs strive to reduce bias in assessment and recruitment, there has been little research on how quiet students are perceived or whether a “quiet bias” exists in EM training. We compared the GA and ARL categories in the SLOEs of quiet EM applicants to non-quiet applicants.

METHODS

Study Design

We conducted a planned subgroup analysis of a retrospective cohort study of all core EM clerkship SLOEs submitted to one, four-year academic EM residency program in the 2016–2017 recruitment cycle. We excluded SLOEs from a non-Liaison Committee on Medical Education accredited school, and from students who had graduated from medical school during the application cycle. The study was approved by the institutional review board and the Association of American Medical Colleges.

Study Setting and Population

We compared SLOEs from applicants who were described as “quiet,” “shy,” and/or “reserved” —collectively referred to as “quiet” — to SLOEs from all other applicants, collectively referred to as “non-quiet.” We chose the descriptors “quiet, shy and reserved” because they are typically used to describe introverts.

Study Protocol

The SLOEs were downloaded from the Electronic Residency Application Service into REDCap electronic data capture tools hosted at University of California, San Francisco by JM and de-identified. Demographic information was self-identified by applicants. Gender identification was mandatory while race and ethnicity were optional. Data from SLOEs was extracted by AN and JG and included geographic region of medical school attended, GA (top 10%, top one-third, middle one-third, lower one-third), ARL (top 10%, top one-third, middle one-third, lower one-third, unlikely to be ranked), and narrative comments.

Data Analysis

We used descriptive statistics to describe demographic makeup of the study population with percentages where appropriate. We applied Pearson’s chi-square test to compare categorical data using R version 3.6 (The R Foundation for Statistical Computing, Indianapolis, IN) and presented this analysis with P-values.

RESULTS

We reviewed 1,582 SLOES from 696 applicants; 120 SLOEs from 107 applicants included the words “quiet,” “shy”, and/or “reserved” to describe the applicant’s personality. The distribution of quiet and non-quiet applicants was not significantly different across race, gender, and geographic region of medical school attended (Table 1). Neither was there a significant difference between quiet and non-quiet students by the gender of the SLOE writer (Table 2).

The distributions of quiet and non-quiet applicants on GA (P <0.001) and ARL (P < 0.001) were significantly different (Table 2). For GA, quiet applicants were significantly less likely to be ranked in the top 10% and top one-third categories combined (31% vs 60%) and more likely to be in the middle one-third category (58% vs 32%), compared to non-quiet applicants. Similarly, for ARL, quiet applicants were significantly less likely to be ranked in the top 10% and top one-third categories combined (33% vs 58%) and more likely to be in the middle one-third category (50% vs 31%) compared to non-quiet applicants (Table 2). Finally, we found no difference (P = 0.66) in the discrepancy between GA and ARL categories (Table 2).

DISCUSSION

Emergency medicine-bound students described in their SLOEs as quiet, shy, and/or reserved were less likely to be ranked in the top GA and ARL categories compared to non-quiet applicants. We found no differences among relationships between quiet applicants and geographic region of medical school, race or ethnicity, gender, or SLOE-writer gender. At face value, this suggests that quiet students may be perceived as less suited for EM clinical settings than non-quiet students. However, other studies have shown that emergency physicians are a heterogenous group with wide-ranging personality attributes and that this diversity may play an important role in team dynamics.^9,10

While we did not assess causality, our findings suggest the need to investigate the possibility that teaching and assessment practices in EM training favor the personality and learning style of extraverts, as shown in other clinical settings.^11,12 For example, teaching methods that include interactive-learning, peer-led discussion, and rapid-response questioning reward extraverts for assertiveness and allow them to overshadow their introverted peers.^11,12 Consequently, evaluators may unfairly perceive introverts as less motivated, knowledgeable, or prepared, which is reflected in poor performance evaluations.¹¹ Similarly, assessment criteria that value characteristics of extraverts (eg, initiates and leads discussions) may undervalue the strengths of introverts (eg, synthesizes information, listens before engaging, reflective).^11,12 Medical students who self-identify as introverts report they are aware of the “quiet” bias in medical training and often feel misunderstood and unfairly judged.¹²

Changes to instructional and assessment practices may create a more supportive environment for introverted learners. Instructional changes could include alternating leadership roles, providing reflection time for responses, and offering student-mentorship to help introverts navigate the learning environment.^6,11,12 Assessment changes such as increasing evaluator-student observations, using assessment tools that focus on skill acquisition, and referencing personality only as it relates to performance, may result in more equitable assessment.^5,12

The ranking disparity identified in this study has high-stakes implications for quiet, EM-bound students who may be at a disadvantage when competing for residency, and warrants further investigation to determine its cause. Examining teaching and assessment practices in the clinical environment may help identify ways to support quiet students in their medical training.

LIMITATIONS

This study has several limitations. We reviewed applications submitted to only one EM residency program, from a single recruitment cycle in 2016, which may not reflect current best practices for writing SLOEs. We did not determine the causality of ranking disparities observed in this study, nor did we assess the contribution of other performance measures such as clerkship grades or board scores. Describing students as quiet, shy, or reserved may not reflect their personality, but rather how they were perceived by their evaluator in the clinical setting in which they were observed. Applicants did not receive a personality inventory nor did they self-report their personality type.

CONCLUSION

Emergency medicine-bound students described as “quiet” in their Standardized Letters of Evaluation were less likely to be ranked in the top global assessment and anticipated rank list compared to non-quiet students. More research is needed to determine its cause and address potential biases in teaching and assessment practices.

Footnotes

Section Editor: John Burkhardt, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: John K. Quinn, MD, University of California, San Francisco, Department of Emergency Medicine, 400 Parnassus Ave., San Francisco, CA 94143. Email: john.quinn@ucsf.edu. 3 / 2023; 24:259 – 263

Submission history: Revision received January 13, 2022; Submitted January 20, 2023; Accepted December 20, 2022

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Love JN, Ronan-Bentle SE, Lane DR, et al. The standardized letter of evaluation for postgraduate training: a concept whose time has come?. Acad Med. 2016;91(11):1480-2.

2. Ginsburg S, McIlroy J, Oulanova O, et al. Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Acad Med. 2010;85(5):780-6.

3. Ginsburg S, Kogan JR, Gingerich A, et al. Taken out of context: hazards in the interpretation of written assessment comments. Acad Med. 2020;95(7):1082-8.

4. Davis KR, Banken JA. Personality type and clinical evaluations in an obstetrics/gynecology medical student clerkship. Am J Obstet Gynecol. 2005;193(5):1807-10.

5. Schell RM, Dilorenzo AN, Li HF, et al. Anesthesiology resident personality type correlates with faculty assessment of resident performance. J Clin Anesth. 2012;24(7):566-72.

6. Khan MA, Malviya M, English K, et al. Medical student personality traits and clinical grades in the internal medicine clerkship. Med Sci Educ. 2021;31(2):637-45.

7. Sutin AR, Costa PT, Miech R, et al. Personality and career success: concurrent and longitudinal relations. Eur J Pers. 2009;23(2):71-84.

8. Chisholm-Burns MA, Berg-Poppe P, Spivey CA, et al. Systematic review of noncognitive factors influence on health professions students’ academic performance. Adv Health Sci Educ Theory Pract. 2021;26(4):1373-445.

9. Pines JM, Alfaraj S, Batra S, et al. Factors important to top clinical performance in emergency medicine residency: results of an ideation survey and Delphi panel. AEM Educ Train. 2018;2(4):269-76.

10. Jordan J, Linden JA, Maculatis MC, et al. Identifying the emergency medicine personality: a multisite exploratory pilot study. AEM Educ Train. 2018;2(2):91-9.

11. Colley SL. Equitable education for introverts: providing learning opportunities that prepare nursing students to be leaders. Nurse Educ. 2018;43(5):224-226.

12. Davidson B, Gillies RA, Pelletier AL. Introversion and medical student education: challenges for both students and educators. Teach Learn Med. 2015;27(1):99-104.