Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Applying to residency is a challenging annual endeavor for medical students who are forced to make decisions about where they would like to train when they may have a limited amount of information about programs. They rely on residency websites, social media, personal communication, online forums, and in emergency medicine (EM) in particular, audition rotations and in-person visits.¹ These opportunities serve both as a pre-interview experience for the prospective student and as a place for information exchange. Students discuss and experience culture, residents, attending physicians, patient populations, support staff, and the physical location of the programs. Programs attempt to identify whether the student is a fit with the culture and gauge their ability to learn and grow, as well as their baseline medical knowledge. It has been shown that some of the most important factors in student selection of a program include how happy the residents seem, faculty-resident relationships, how well the residents work together, values of residents and faculty, and whether the residents spend time together outside of the residency and there are shared interests.² These are traits difficult to communicate without in-person experiences.

During the 2020–2021 application cycle, amidst the coronavirus 2019 (COVID-19) global pandemic, most of the opportunities that previously took place in person were not feasible. There was significant concern that many third-year medical students would not be able to gain clinical and program-specific experience before submitting their residency applications.³ Fewer and shorter clinical rotations were available for interested students because host institutions were attempting to limit COVID-19 exposure. COVID-19 also led to the creation of the “Consensus Statement on the 2020–2021 Residency Application Process for US Medical Students Planning Careers in Emergency Medicine in the Main Residency Match” by 10 medical education organizations, limiting each student to one EM rotation, typically at their home institution.⁴ Both residency programs and medical students worked to develop creative ideas to showcase their strengths and connect with each other through innovative methods not previously used. Students were attempting to gather information and evaluate programs while residency programs were working to identify alternative ways to share more than the facts and figures that websites typically offer.

One novel method emerged for students and residents to share information through virtual “meet and greet” events, sometimes referred to as “town hall events,” “get to know us sessions,” or “virtual tours.” These virtual events, hosted by the residency program, became a popular way to connect programs and students. The basic format was a series of sessions, or a single video session, hosted and promoted by a specific residency program, led by either current residents, faculty, or both. A brief presentation or introduction was typically provided, followed by a space for students to ask questions and glean information about specific program details. Some programs offered multiple sessions, each highlighting specific strengths.

Population Health Research Capsule

What do we already know about this issue?

Residencies use virtual “meet and greet” events to connect programs and students, but the effectiveness of these events has not been studied.

What was the research question?

Do virtual information sessions improve medical student knowledge of curriculum offerings and provide enough program exposure?

What was the major finding of the study?

Attendance at virtual information sessions improves the perceived knowledge of curriculum and enhances program exposure.

How does this improve population health?

Residency programs should consider incorporating these types of events to enhance medical student recruitment and convey important program information.

After performing a brief search through social media outlets, program websites, and virtual newsletters from EM professional organizations, we found that as of August 2020 at least 99 of 273 EM residency programs publicized or participated in a virtual event. This number significantly increased by the end of the 2020–2021 application cycle with 224 EM residency programs participating in a virtual event as of March 2021. These events ranged from Instagram or Facebook live events to Zoom gatherings and panels. Prior to this application cycle, such events were uncommon and did not appear to be a standard way of disseminating information to the residency applicant pool. Thus, there was no published research to date on the effectiveness of a live virtual recruitment event in providing useful information to EM applicants attempting to make residency selection decisions. With many EM residency programs transitioning to these virtual events for recruitment purposes, we believed it would be important to determine whether a virtual format is an effective method of improving applicant understanding of curriculum information and enhancing program exposure.

In this study our primary objective was to evaluate the usefulness of this kind of event at four different residencies within the US using pre- and post-event surveys. Specifically, does attendance at a virtual residency program information session improve the perceived knowledge of curriculum information and program exposure to medical students applying to an EM residency? We hypothesized that participation would allow for increased self-reported knowledge of various specific program details and increase the interest of the applicants in the program.

METHODS

This was a multicenter pre- and post-intervention survey of medical students attending a virtual residency information session at one of the four study sites. Site one is a three-year county hospital-based EM program with 45 residents in Arizona, which hosted one session. Site two is a community-based four-year EM program in Oklahoma with 30 residents, which hosted two sessions. Site three is a three-year county hospital-based EM program with 45 residents in Texas that hosted eight sessions. Finally, site four is a three-year community rural ED with 36 residents in New Jersey that hosted one session.

Each site was responsible for promoting its own event including advertising, date selection, time, virtual software platform used, and number of dates to offer their meetings. A standardized pre- and post-event survey was used by all four sites with voluntary participation by the students who attended each event. We provided each program with a survey template using Google Forms (Google LLC. Mountain View, CA), which each program could then individually brand without changing the content of the survey. Students were recruited before and after the session to fill out the survey information.

As the use of virtual information sessions is a relatively novel practice, there are no previously validated surveys on this topic. Students were instructed to indicate their agreement using a Likert scale with specific statements before and after the sessions. Questions were developed, reviewed, piloted, and refined by the research team and adapted from previously published literature indicating some of the top reasons medical students select a residency program.^2,5 Written consent was obtained prior to completion of the surveys by all subjects. We also collected demographic information was also collected including age, gender, medical school year of training, and geographic location. To ensure blinding and total anonymity, the last five digits of the participants’ phone numbers were used to link the pre- and post-event surveys. It was imperative to make it clear to the participants that the survey was both voluntary and anonymous to prevent concern about possible attribution or effect on their residency application. The project received a human subjects’ research exemption from the Valleywise Health Institutional Review Board and was completed in 2020.

Summary of all the respondents and questions were described along with comparison of pre-post matched responses linked using the last five digits of the respondents’ phone numbers. For each question, the highest level of agreement was assigned a score of 5 with the lowest level of agreement assigned as 1. We used the Wilcoxon signed-rank test to compare the score for each question by students who responded to both pre- and post-surveys. Responses were further dichotomized into “agree or strongly agree” vs all other responses; we used McNemar two-sided tests to evaluate changes in agreement pre- and post-surveys. For 10 questions, perfect favorable agreement would make a total agreement score of 50, which was used as a composite score to evaluate overall pre-post change in level of agreement. We used the Wilcoxon signed-rank test to compare differences in composite scores pre- and post-survey.

RESULTS

Overall, the pre-survey was completed by 195 attendees, and the post-event survey was completed by 123 attendees. Response proportions for pre-surveys ranged from 40.23% to 59.02% of total attendance across four locations, which were slightly higher than 21.88–49.18% for the post-event surveys (Table 1). Survey response and demographic information is displayed in Table 2, including a breakdown of which location session the participants attended. Distribution of those participants who chose to report gender was nearly evenly split between male and female. Many of the participants were between the ages of 26–30, and most were fourth-year medical students. Geographically, all areas of the country were represented with the largest number from the southwest United States and the lowest percentage of participants from the northwest or international.

Table 3 illustrates pre- and post-event survey summaries. Values ranged from 1 to 5 with 1 representing strongly disagree, and 5 representing strongly agree. The median and mean composite score to this 10-question survey improved from 32.19 to 45, and 31.45 and 44.2, respectively, in the pre- to post-event surveys. Of the total number of completed surveys 75 attendees from three institutions were matched as completing both the pre- and post-event surveys (35 from site one, 24 from site two, and 16 from site three). Unfortunately, at site four, no participants could be matched to link the pre- and post-event surveys.

We further analyzed these 75 participant responses to determine efficacy of the goals of the information sessions. Table 4 shows overall data results for these 75 matched responses, as well as location-specific results. All questions showed improvement from pre- to post-event surveys following the information sessions. Table 5 displays the matched pre- and post-event question results and demonstrates a change from before and after the virtual information session. For statistical analysis, “neutral,” “disagree,” and “strongly disagree” (1, 2 and 3) were clustered as a disagreement. Similarly, “agree,” and “strongly agree,” were clustered as an agreement to improve overall data analysis. Of note, 15 responses were missing from question one due to site four inadvertently editing their version of the survey, removing the question. These were removed for data analysis. McNemar P-values were < 0.001 for all 10 questions.

DISCUSSION

The ways in which residency programs and potential applicants interact will continue to change as the COVID-19 pandemic fluctuates and resolves. In this study we were able to show improvement toward agreement from pre- to post-event surveys after virtual recruitment events. The data clearly demonstrates through the students’ responses to questions that they not only obtained information about the program (number of elective rotations, general curriculum, emergency department layout, presence of specialty tracks, fellowship potential), but also information regarding the culture and “feel” of a program, which can sometimes be challenging to convey. Additionally, the results demonstrate that students were not only provided with tools for contacting program members, but that they were comfortable doing so after attending one of these sessions. As this study was designed to identify the perceived benefit of a virtual information session, further research opportunities include directly assessing applicant knowledge of residency-specific details.

As new techniques and ideas are developed to navigate a new virtual normal, it is important to ensure that students can make appropriate decisions about where they would like to match for residency. Many of these virtual sessions were developed to address the 2020–2021 travel and visiting restrictions. However, this type of event will likely be beneficial in the future as well. These virtual gatherings can continue to be used as a tool by programs to help showcase their culture, community, and strengths. They can also provide an opportunity for students who are unable to physically visit programs gain more information about a residency program.

LIMITATIONS

We identified various limitations throughout this research experience. The most significant limitation was the low response rate for the surveys. As the design included a pre- and post-event survey with no requirements for completion, there was a large discrepancy between the number of students who registered, completed the pre-event survey, and attended the event, and the number of students who completed the post-event survey. There were fewer participants for the post-surveys relative to the pre-surveys across each of the four locations resulting in fewer matchable pairs available for the pre-post response comparisons. Unfortunately, data was missing or incomplete due to the survey nature, and not all pre- and post-event survey responses could be matched, resulting in some incomplete data. As a result, we are only able to report results from students willing to complete both surveys. Due to the small data sample size, many of the numerical results may be impacted, and there is a potential for variation in the results. However, our limited data on the responses to questions indicated a dramatic improvement in perceived knowledge, with many areas showing considerable advancement in perceived knowledge of the program. While the individual percentage of improvement could change with a larger sample size, it is unlikely that the overall benefit of virtual information sessions for interested medical students would be rejected.

During the design of the protocol, we determined that comfort of the participants was paramount, as we did not want there to be undue pressure for a student to complete the survey or to feel that there would be a professional consequence associated with participation. Students received one email before the session and one after recruiting them to participate in the survey with no other correspondence, which likely hindered our response rate. Future studies and follow-up questionnaires regarding this topic may improve response rates by sending additional follow-up correspondence. One option would be to use a staff member not associated with the residency to send additional emails and collect responses to maintain objectivity, resulting in another layer of protection for students completing the questionnaire.

Additionally, there was no standardized structure on what was to be included at each of the study site information sessions. Each location was free to promote, advertise, and host its own session or multiple sessions without specific oversight or specific requirements. This was intentionally done to allow for the creativity, personality, and innovation of each program to shine through without external modification. While we emphasized both in writing and verbally our goal of anonymity, some students may have felt pressured to respond to the surveys in a positive nature as they were likely applying for a residency position with the program. Furthermore, as students were self-selecting which information sessions to attend, there is a potential for a positive selection bias. Finally, as these programs were recruiting in the same application cycle, there may have been attendance overlap by medical students participating in the individual surveys. Future studies on this topic may benefit from including sites from different geographic locations and with a wider range of program sizes to further strengthen results.

Additionally, our study questions were targeted for short-term follow-up and did not prove increased familiarity long-term or necessarily affect the program’s recruitment of candidates. Additional studies including how attendance at a virtual information session may affect a student’s rank list or match results would be areas of future investigation to further examine the benefit of virtual sessions.

CONCLUSION

It is important for medical students and EM residency programs to be able to critically evaluate each other to allow for the best possible match. Hosting virtual meet-and-greet events can be an effective way for programs to attempt to facilitate this type of information exchange when in-person interaction is limited. Attendance at a virtual residency program information session appears to improve the perceived knowledge of curriculum information and program exposure to medical students applying to an EM residency. Residency programs should consider incorporating these types of events to enhance medical student recruitment and convey important program information.

Footnotes

Section Editor: Jeffrey Druck, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Geoffrey Comp, DO, Valleywise Health, Department of Emergency Medicine 2601 E Roosevelt St Phoenix, AZ 85008. Email: Geoffrey_comp@valleywisehealth.org. 7 / 2022; 23:525 – 531

Submission history: Revision received September 6, 2021; Submitted April 11, 2022; Accepted April 6, 2022

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Schweitzer J, Hannan A, Coren J. The role of social networking web sites in influencing residency decisions. J Am Osteopath Assoc. 2012;112(10):673-9.

2. DeSantis M, Marco CA. Emergency medicine residency selection: factors influencing candidate decisions. Acad Emerg Med. 2005;12(6):559-61.

3. Gabrielson AT, Kohn JR, Sparks HT, et al. Proposed changes to the 2021 residency application process in the wake of COVID-19. Acad Med. 2020;95(9):1346-1349.

4. Farcy D, Haig A, Suter R, et al. Consensus Statement on the 2020, 2021 Residency Application Process for US Medical Students Planning Careers in Emergency Medicine in the Main Residency. Available at: https://www.emra.org/be-involved/be-an-advocate/working-for-you/residency-application-process/. Accessed March 29, 2021.

5. Love JN, Howell JM, Hegarty CB, et al. Factors that influence medical student selection of an emergency medicine residency program: implications for training programs. Acad Emerg Med. 2012;19(4):455-60.