Author	Affiliation
Sean B. Schnarr, MD	Cook County Health and Hospitals System, Department of Emergency Medicine, Chicago, Illinois
Victoria Gonzalez, MD	Cook County Health and Hospitals System, Department of Emergency Medicine, Chicago, Illinois
Neeraj Chhabra, MD	Cook County Health and Hospitals System, Department of Emergency Medicine, Chicago, Illinois

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Medical students interested in emergency medicine (EM) have multiple resources available to assist them during their residency application process.^1-3 The National Resident Matching Program (NRMP), for example, publishes data from medical students entering the match process within each medical specialty.⁴ The NRMP’s Charting Outcomes in the Match publications include applicants’ mean United States Medical Licensing Examination (USMLE) step scores, and Alpha Omega Alpha (AOA) Honor Medical Society status. Advisors, mentors, and other official resources provide applicants with additional information on the application process including application approaches, interview strategies, and general statistics for residency programs.^5,6 Despite these resources, medical students are often unaware of how their residency application compares to their peers, leaving applicants to use other, less official, resources with undetermined accuracy.⁷

One commonly used online resource is the Student Doctor Network (SDN, www.studentdoctor.net), which offers an online forum for students, residents, and attending physicians to discuss past and current experiences with the match process, among other topics. The SDN hosts forums for its online community by subject matter spanning all stages of medical education. The forums are available for public viewing, but posting is restricted to those with an account on the website. In recent years, it has become common for residency applicants to provide self-reported data from their own residency application.^8,9 These data can then be accessed by other potential applicants to evaluate the competitiveness of their own application. Applicants who use this data to inform their own application process must do so with caution, as these posts are anonymous and there is no mechanism to ensure their accuracy.

A comparison of self-reported SDN and NRMP data in the comparatively small field of radiation-oncology showed bias of aggregate self-reported test scores toward higher-scoring applicants.⁹ There are no studies to date comparing self-reported SDN data with NRMP’s published data in the larger field of EM. As SDN represents a potential source of important information for EM applicants, our goal in this study was to compare SDN data with NRMP data to determine whether self-reported SDN data is an accurate representation of the typical EM applicant.

METHODS

This was a retrospective analysis of self-reported applicant data within EM forums on the SDN. Those who reported on the SDN either used the forum system on annual threads for EM applicants or, in the case of the 2017–2018 application cycle, a Google Docs spreadsheet (Google LLC, Mountain View, CA) was created that allowed users to anonymously add their own data without creating an account on the SDN. Links to this spreadsheet were posted to the SDN and the website Reddit (www.reddit.com) (Reddit Inc., San Francisco, CA).¹⁰ With the forum system, respondents replied to the original thread with a post to provide their application information in a structured format under their SDN username. These were subsequently aggregated by the researchers. The spreadsheet allowed anonymous users to provide the same structured data in an already aggregated format.

We performed data collection and analysis for the 2014, 2016, and 2018 application cycles because those were the years with corresponding NRMP publications.^11-15 For the purposes of this study, DO and MD applicants were pooled. Given the different application experiences of international medical graduates (IMG) applying for EM residency, such as the average number of applications submitted, we excluded IMGs from analysis.^16,17

The variables collected from the SDN included those available in NRMP publications such as USMLE Step 1 and 2 scores, AOA status, and number of contiguous ranks as well as those commonly included in SDN forums, including number of residency applications and number of accepted interview invitations. Given the self-reported nature of the SDN, there were missing data points that were not included in analysis. We obtained comparison data from NRMP Charting Outcomes in the Match for 2014, 2016, and 2018. The NRMP provides means and proportions but not distributions, so we did not perform direct statistical comparisons with SDN data. We analyzed data by descriptive statistics using Microsoft Excel (Microsoft Corporation, Redmond, WA). Descriptive data are reported as means with 95% confidence intervals (CI) to match NRMP reports, where applicable, while medians and interquartile ranges (IQR) were used for non-parametric data not reported by NRMP. This study was reviewed by our institution’s institutional review board and deemed exempt because it used de-identified and publicly available information.

 

Population Health Research Capsule

What do we already know about this issue?

Medical students often use various online resources to help guide their residency application process.

What was the research question?

How accurate is self-reported data regarding emergency medicine applicants on the Student Doctor Network (SDN)?

What was the major finding of the study?

Self-reported data on SDN showed slight bias toward higher USMLE scores compared to objective NRMP data.

How does this improve population health?

Online resources are often relied on heavily by medical students, but should be used with caution when applying to emergency medicine residency.

 

RESULTS

In total, there were 360 applicants with self-reported information on the SDN in the years 2014, 2016, and 2018, representing 7.3% of all EM applicants during the time period. The majority (79%) of SDN applicants self-reported in the 2018 application cycle, which used a Google Docs spreadsheet instead of a typical SDN forum. This sample represented 14.5% of all EM applicants for that year. The mean USMLE Step 1 and Step 2 scores reported by applicants was 235 and 249, respectively. Table 1 shows cumulative, self-reported SDN applicant data for the included years. Table 2 shows a comparison of SDN data by year with corresponding data reported by the NRMP. As the NRMP data represents true population totals, 95% CIs were not calculated. In general, those who posted on the SDN had similar USMLE step scores and a similar number of contiguous ranks. For 2018, however, aggregated USMLE step scores from the SDN showed a higher average than reported by the NRMP. The mean number of applications submitted per applicant was 53 with a median of 45, indicating a positive skew.

DISCUSSION

From the information provided by EM applicants on the SDN and those compiled by the NRMP, the mean USMLE Step 1 and Step 2 scores reported for applicants by the NRMP was similar to those self-reported by applicants. While the average USMLE Step 1 and Step 2 scores were higher on the SDN self-reported data than the NRMP for all years compared, they were typically within the 95% CI of the mean. Exceptions to this were noted in 2018, indicating that for this year, the SDN had a bias toward higher scoring applicants. One possible explanation for this discrepancy is that applicants with lower scores may be less willing to publicly disclose their test scores, even anonymously.

Applicants should interpret anonymously self-reported examination scores with caution. The average number of contiguous ranks between the SDN and NRMP, however, were similar in all years studied. According to the NRMP, applicants with 12 contiguous ranks had approximately a 95% probability of matching, which is a valuable data point for future applicants.⁴ Given that most categories, in aggregate, appear similar to NRMP data while some show important differences, it is unclear how applicants should best use the SDN as a potential data source to inform their own application process.

With the use of Google Docs in 2018, there was a nine-fold increase in the number of users posting data using the SDN compared to 2016. This has been observed in previous studies comparing these two sources and is likely due to the ease of use, anonymity, and ability to access the spreadsheet from either the SDN or Reddit.^8,9 As more users contribute in future years, it is possible that the differences noted between SDN and NRMP data will decrease, as was seen with the percentage of applicants that claimed AOA status. Alternatively, given the ease with which users can anonymously post, some posts on the Google Doc may not be accurate and the spreadsheet may be unavailable periodically due to inappropriate and/or offensive posts and necessary maintenance.

There was considerable variability in the reported number of residency applications submitted by SDN users, with an interquartile range of 35 to 62.25 applications. It is unknown whether the range among applicants was due to counsel from advisors, perceived strength or weakness of individual applications, or a combination of the two. This is an important consideration for applicants as medical students are applying to more residency programs, often at significant personal cost.^18,19 At the current 2021 Electronic Residency Application Service (ERAS) fee structure, the average number of applications from the current study (53) would cost $1187 per applicant.²⁰ For medical students, most with limited to no income, this cost is unreasonable but may be deemed necessary to “keep up” with their peers. While advisors may counsel against an inordinate number of applications per applicant, students may be influenced by noting how many programs their peers report on the SDN that they are applying to. Applicants, advisors, and ERAS should explore ways to address the increasing number of applications and limit the costs of the application process to avoid placing applicants from less privileged financial backgrounds at a competitive disadvantage.

LIMITATIONS

There were several limitations to this study. For the 2018 application cycle, users did not self-report data on successful matching; so this was excluded from analysis. Although the anonymous forum dramatically increased the number of users who posted information, it is likely that many users simply stopped using the website after a successful match. As both NRMP aggregated data and SDN data are anonymous, direct comparisons of these data in individual applicants was not possible. Similarly, data collection techniques significantly differ between the two sources. Further, given the small sample size from the SDN, conclusions regarding its accuracy should be tempered.

CONCLUSION

Self-reported EM applicant data on the Student Doctor Network is similar to data provided by the NRMP with a bias in recent years toward higher self-reported standardized test scores. With the emergence of Google Docs as a centralized and more anonymous avenue for self-reporting data, a dramatic increase in applicants providing information was noted for the most recent application cycle. Whether this trend will provide more accurate data for potential EM residency applicants remains to be seen.

Footnotes

Section Editor: Douglas Ander, MD and Andrew Phillips, MD, MEd

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Sean Schnarr, MD, Bellevue Hospital Center, Department of Emergency Medicine, 1462 First Avenue, Third Floor Room 345A, New York, NY 10016. Email: Sean.Schnarr@nyulangone.org. 1 / 2022; 23:90 – 94

Submission history: Revision received June 9, 2021; Submitted December 15, 2021; Accepted October 27, 2021

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. EM match advice: Where did all the jobs go, and did the applicants follow?. 2021. Available at: https://www.aliem.com/category/academic/em-match-advice/. Accessed April 15, 2021.

2. Lutfy-Clayton L, Morrissey T, Pelletier-Bui A, et al. The Emergency Medicine Applying Guide. 2018. Available at: https://www.cordem.org/globalassets/files/committees/student-advising/2018-applying-guide-cord-cdem-emra-rsa.docx.pdf. Accessed April 15, 2021.

3. Manthey DE, Hartman ND, Newmyer A, et al. Trends in NRMP data from 2007–2014 for U.S. seniors matching into emergency medicine. West J Emerg Med. 2017;18(1):105-9.

4. The Match, National Resident Matching Program. Match Data & Report Archives. Available at: https://www.nrmp.org/report-archives/. Accessed April 15, 2021.

5. Overton DT. Advice for emergency medicine applicants. 2017. Available at: https://med.wmich.edu/sites/default/files/Advice%20for%20Emergency%20Medicine%20Applicants%202017.pdf. Accessed April 15, 2021.

6. Leung CG, Malone M, Way DP, et al. Preparing students for residency interviews in the age of COVID: lessons learned from a standardized video interview preparation program. AEM Educ Train. 2021;5(3):e10583.

7. Kukulski P, Ahn J, Babcock C, et al. Medical student self-assessment as emergency medicine residency applicants. AEM Educ Train. 2021;5(3):e10578.

8. Ramachandran V, Nguyen HY, Dao H. Does it match? Analyzing self-reported online dermatology match data to charting outcomes in the match. Dermatol Online J. 2020;26(4).

9. Sura K, Wilson LD, Grills IS. Comparison of self-reported data on Student Doctor Network to objective data of the National Resident Matching Program. J Am Coll Radiol. 2017;14(12):1594-7.

10. Attention all EM Applicants: Official 2017– 2018 Applicant & Interview Google Sheet. 2017. Available at: https://www.reddit.com/r/medicalschool/comments/6srbi5/attention_all_em_applicants_official_20172018/. Accessed April 15, 2021.

11. Charting Outcomes in the Match: Senior Students of U.S. Osteopathic Medical Schools 2018. 2018. Available at: https://www.nrmp.org/wp-content/uploads/2018/06/Charting-Outcomes-in-the-Match-2018-Osteo.pdf. Accessed April 15, 2021.

12. Charting Outcomes in the Match: U.S Allopathic Seniors 2018. 2018. Available at: https://www.nrmp.org/wp-content/uploads/2018/06/Charting-Outcomes-in-the-Match-2018-Seniors.pdf. Accessed April 15, 2021.

13. Charting Outcomes in the Match for U.S. Osteopathic Medical Students and Graduates 2016. 2016. Available at: https://www.nrmp.org/wp-content/uploads/2016/09/Charting-Outcomes-US-Osteopathic-2016.pdf. Accessed April 15, 2021.

14. Charting Outcomes in the Match for U.S. Allopathic Seniors 2016. 2016. Available at: https://www.nrmp.org/wp-content/uploads/2016/09/Charting-Outcomes-US-Allopathic-Seniors-2016.pdf. Accessed April 15, 2021.

15. Charting Outcomes in the Match 2014. 2014. Available at: https://www.nrmp.org/wp-content/uploads/2014/09/Charting-Outcomes-2014-Final.pdf. Accessed April 15, 2021.

16. Zhang XC, Jarou Z, Kellogg A, et al. International Medicine Graduate (IMG) Emergency Medicine Applying Guide. 2016. Available at: https://www.cordem.org/globalassets/files/student-resources/applying-guide—international-applicant.pdf. Accessed April 15, 2021.

17. Charting Outcomes in the Match: International Medical Graduates, 2018. 2018. Available at: https://www.nrmp.org/wp-content/uploads/2018/06/Charting-Outcomes-in-the-Match-2018-IMGs.pdf. Accessed April 15, 2021.

18. Blackshaw AM, Watson SC, Bush JS. The cost and burden of the residency match in emergency medicine. West J Emerg Med. 2017;18(1):169-73.

19. Dermark JTV, Wald DA, Corker JR, et al. Financial Implications of the emergency medicine interview process. AEM Educ Train. 2017;1(1):60-9.

20. Fees for ERAS Residency Applications. Available at: https://students-residents.aamc.org/applying-residencies-eras/fees-eras-residency-applications. Accessed April 15, 2021.