Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

The academic community has traditionally viewed anonymous online communities (AOC) as poor resources for advising, recommending that students be wary of them or avoid them altogether.
¹
Common themes addressed against these forums include lack of commenter professionalism,
<sup>2

–

4</sup>
information inaccuracy,
⁵
and breach of ethics via malicious posting of falsified, incomplete, or privileged information.
<sup>2

,

3</sup>
Prior studies have also shown that anonymous AOC commenters may not necessarily reflect the entire applicant population.
⁶
Within emergency medicine (EM), students report information from AOCs such as Reddit and Student Doctor Net (SDN) as the “least trustworthy” compared to other advising resources.
⁷

Despite this, most specialties, including EM, have robust AOCs for medical students, boasting large user bases and robust discussion threads with heavy readership.
<sup>8

–

11</sup>
Within these anonymous forums, students discuss diverse topics about the application process, specialty-specific questions, and student experiences applying to, rotating at, or interviewing with specific programs.
<sup>12

,

13</sup>
For example, studies identify “program-specific information” as a common theme in otolaryngology- and radiology-applicant AOCs; however, their findings were limited in characterizing specific topics discussed.
<sup>12

,

13</sup>
There is also a consolidated, annually renewed, and user-generated Google Spreadsheet circulating within EM forums with a stated goal to “provide a central location for applicants to research different residency programs, view information about other applicants and where they are applying, and share information about away, interviews, and general advice.”
¹⁴

For EM, the discussion of authentic, program-specific experiences, such as the student’s interview day experience and interaction with residents, have historically been ranked as the top two factors in impacting their rank order, making this information highly valuable to both applicants and program stakeholders.
¹⁵
Our primary goal was to characterize what prospective EM applicants discuss in an AOC forum regarding their experiences with specific programs. Our secondary goal was to identify potentially useful information for program improvement and development.

METHODS

This was a qualitative, retrospective review of a publicly available AOC for EM rotations in 2020–2021. It was submitted for institutional review board review through Thomas Jefferson University and determined to not meet the definition of human subjects research. We analyzed extracted data from an online, time-stamped, and user-locked Google Sheet entitled “Official 2020–2021 Emergency Medicine Applicant Spreadsheet,” whose link can be found within popular AOCs such as Reddit, SDN, and Discord.
¹⁴
“The Spreadsheet” allows anonymous individuals to post requested information regarding specific EM programs. The spreadsheet contains multiple sub-pages, or “sheets,” to address different types of information an applicant might seek. This includes sheets listing program-specific facts such as “Program Benefits” and “Program Information”; sheets describing student experiences with a program like “Rotation Experience,” “Virtual IV (Interview) Impressions,” or “Name and Shame”; and sheets addressing miscellaneous application topics such as “Rejection/Wait List” or “Dropped Interviews” to help applicants coordinate logistics.

With permission from the page administrator, confirmed to be a current EM resident, we created a replica of the spreadsheet on September 12, 2021, for the purpose of this study. Upon review of all available sheets within the spreadsheet, the sub-pages entitled “Virtual IV” and “Rotation Impressions” were purposefully sampled via group consensus for analysis as they were felt to most likely include students’ direct impressions of programs. In contrast, we excluded sub-pages such as “Name and Shame” from qualitative analysis due to high likelihood of containing caustic and controversial narratives. As the purpose of this study was to investigate “what” is being said, not “who” is discussing them or to “whom” it is addressed, one investigator transferred all comments from the selected pages into a single dataset while removing potentially identifying user or program information.

We performed qualitative analysis primarily using the constant comparative method,
¹⁶
where excerpts of raw data are organized into groups according to attributes and those groups are further structured to formulate a new theory. The selected sub-pages yielded 183 individual narratives discussing students’ impression of the de-identified programs. A random number generator was used to select 20% of individual narratives as a convenience sample for investigators to inductively create a working codebook, de novo. All duplicates were identified and removed, until the excerpts were all unique. The dataset was independently analyzed by the investigators [ME, JG, RR, XCZ] to identify thematic content within each narrative for inductive coding. Individually identified themes were compared among investigators to generate common themes. These themes were organized into major “themes” and “sub-themes” to create the initial codebook. This was used by a single investigator (JG) to code the entire dataset. Additional sub-themes identified during this process were updated into the codebook under existing major themes. Upon completion, all themes and their associated excerpts were reviewed by the remaining investigators to ensure coding consensus. The entire dataset was reviewed until no additional themes were identified (see Appendix 1).

Upon reviewing the final codebook, we created two additional thematic categories: 1) potentially “modifiable” program attributes and 2) less likely or “non-modifiable” program attributes. Drawing upon our collective experiences, we defined “modifiable” subthemes as attributes most likely under the direct control of the education leadership and “non-modifiable” subthemes as attributes that are either truly non-modifiable or would require significant input from outside stakeholders to change. This distinction was made with the understanding that different programs have different abilities to modify certain attributes.

RESULTS

From 183 comments, 841 discrete statements were identified and coded. We identified six themes: working conditions; interpersonal relationships; learning experience; living conditions; postgraduate readiness; and online/virtual supplement, as shown in Figure and Table 1. The top two encoded themes—working conditions and interpersonal relationships—comprised 572 (68%) of the total coded statements (324 [38.5%] and 248 [29.5%], respectively).

Sub-themes identified within each theme (see Appendix 1 for a full listing of sub-themes with their corresponding number of coded statements) were then subdivided to represent modifiable and non-modifiable clerkship/program aspects (Tables 2 and 3). Modifiable sub-themes outnumbered non-modifiable sub-themes (60.7% vs 39.3%). The sub-themes housed within the theme of interpersonal relationships represented the largest single category of modifiable attributes with 248 (29.5%) statements. Comments on working conditions and learning experience were the second and third largest categories, with 109 (13%) and 118 (14%) comments, respectively. The majority of non-modifiable sub-themes were found within the theme of working conditions with 215 (25.6%) individual comments, which represented 65% of all non-modifiable comments. The second largest non-modifiable sub-theme was within the theme of living conditions, including comments on the local geography, cost of living, or nearby amenities.

DISCUSSION

Anonymous online communities have been historically viewed by clerkship and residency program leadership as unreliable forums for student discussion that foster confabulation of facts and operates on rumors and hearsay, a communication tool of the disgruntled, and not a resource to be taken seriously.
<sup>1

–

3

,

5

–

7</sup>
This is the first study to describe, in detail, the narrative content of students discussing their program impressions on an AOC. Our findings suggest that many of the discussed items are common considerations for a student seeking to find the ideal next stage of training. Mentors in EM have historically encouraged prospective EM applicants to inquire about interpersonal interactions and resident working conditions within a specific program. Our analysis reveals that students are also seeking more information and commenting on many of the same factors we have been advising them to seek out.
¹⁷

Moreover, analysis of the sub-themes reveals a unique trend toward potentially modifiable program attributes that, if addressed, could be mutually beneficial for programs and applicants. Topics such as perceived resident wellness, diversity, equity, and inclusion, opportunities for upward feedback, and effectiveness of on-shift teaching are all under the control of a program to potentially improve. Many of these topics are of rising importance to students.
<sup>18

,

19</sup>
The availability of this information raises a very interesting question for programs and recruitment: if programs were aware of these discussed topics and the student comments relative to each topic, would a program be likely to change internal element(s) to make itself more appealing to students?
²⁰

In light of the recent National Resident Matching Program (NRMP) results from 2022 and the continued downtrend of applications in 2023,
<sup>21

,

22</sup>
many EM programs must contend with a smaller applicant pool, which reduces the likelihood of filling programs, and overall program competitiveness for applicant recruitment. While we cannot predict future trends, our specialty has faced declining student applications for two years in a row with a rising number of residency programs and positions over the last several years. As traditional matching patterns begin to falter, residency leadership should consider addressing critical elements from AOCs, instead of ignoring them as tradition dictates.

An interesting final observation from our study is the relatively scarce number of comments from students on virtual or online components of a program. Our dataset reflected the first application cycle during the COVID-19 pandemic with radical paradigm shifts in away-rotation restrictions and students exploring virtual interview processes for the first time. Despite these unprecedented large-scale changes, only 4.2% of the total comments focused on the “virtual” aspect of program recruitment. This is in stark contrast to the significant amount of time spent by institutions and national organizations on virtual rotations, virtual tours, ongoing virtual interviews, virtual residency fairs, and virtual hangouts for students to socialize with residents. The data remains unclear based on this information from a single year to explain this lack of commentary. It may perhaps be due to lack of student participation in virtual rotations, given this was their first year being available as a rotation option, or perhaps virtual rotations were just simply not seen as appealing, thus demanding less discussion time on AOCs. Further analysis of subsequent years would be needed to fully analyze the effectiveness of virtual options for student applicants.

LIMITATIONS

Potential limitations rest largely on data fidelity.
<sup>2

,

3

,

5</sup>
Prior studies have also shown that AOC commenters may not necessarily reflect the entire applicant population.
⁶
There is limited-to-no demographic information provided on the analyzed AOC. Additionally, the 183 narratives analyzed from the spreadsheet are relatively low compared to the number of applicants ranking EM as their preferred specialty or the 273 EM programs in existence at the time of 2021 NRMP Match.
²¹
This may have put our analysis at risk of not reaching thematic saturation. Nevertheless, based on our collective experiences as EM residency applicants and as EM application advisers, we did not find any identified sub-themes particularly surprising or controversial. Although only a single AOC was analyzed in this study, we believe it to be fairly representative of commonly recurring student opinions and observations. For the purpose of this study, we specifically selected two sub-pages with the highest density of meaningful commentary for analysis; there is the potential that comments from other pages may reveal further themes or sub-themes.

CONCLUSION

Our qualitative analysis of a single anonymous online community revealed six major themes discussed among students with regard to EM residency programs: working conditions; interpersonal relationships; learning experience; living conditions; postgraduate readiness; and online/virtual supplement. Most of the sub-themes to these categories represented aspects of clerkships and residency programs that are potentially modifiable by the program leadership. These findings suggest that AOC narratives cover several topics that may serve as useful external feedback for EM residency programs or clerkships. Iterative review of program-specific AOC narratives could serve as additional data in determining whether a program’s internal improvement efforts are noticed by students. Additional studies may help characterize the level of interest by key program stakeholders to consider and make changes based on feedback from AOC sources.

Footnotes

Section Editors: Jeffrey Druck, MD, and Michael Gottlieb, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Molly Estes, MD, Loma Linda University, Department of Emergency Medicine, 11234 Anderson St, A-890, Loma Linda, CA 92354. Email: mollykestes@gmail.com
03 / 2024; 25:191 – 196

Submission history: Revision received March 15, 2023; Submitted October 18, 2023; Accepted November 21, 2023

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. This research was funded in part by an EMRA Be the Change Grant awarded in 2021.

REFERENCES

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.