Author	Affiliation
Jordan R. Pollock, BS	Mayo Clinic Alix School of Medicine, Scottsdale, Arizona
Jeffery A. Weyand, BS	Washington State University Elson S. Floyd College of Medicine, Spokane, Washington
Amy B. Reyes, MLIS	Arizona State University, Tempe, Arizona
Shiva Senemar, BS	Arizona State University, Tempe, Arizona
Aunika L. Swenson, MD	Stanford University, Department of Emergency Medicine, Palo Alto, California
Rachel A. Lindor, MD, JD	Mayo Clinic, Department of Emergency Medicine, Phoenix, Arizona
James L. Homme, MD	Mayo Clinic, Department of Emergency Medicine, Rochester, Minnesota

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Emergency medicine (EM) is a popular specialty among medical students, evidenced by the growing number of EM residency applicants over the last 10 years, with 2903 applicants in 2011 and 3640 in 2020.¹ Many applicants depend on the internet as a primary source of information when researching different residency programs.² This has been particularly true for the 2020–2021 residency cycle due to COVID-19-related restrictions on travel and in-person activities. Even prior to this change, information available on the internet was often a determining factor in prospective applicants’ decisions to apply for rotations or residency interviews. A survey of EM applicants found that 78% claimed information provided in the residency program website influenced their decision to apply to a particular program.³ In addition, 41% of applicants decided not to apply to at least one program solely based on the information available on the residency program website.³ Accordingly, website development, content, and accessibility are increasingly important for residency programs.

To assist medical students in navigating the staggering number of residencies across the United States, databases such as the Fellowship and Residency Electronic Interactive Database (FREIDA) have been designed to allow convenient access to program websites and information on residencies and fellowships.⁴ However, little is known about the quality of information available through these websites in the field of EM. Prior studies have evaluated residency websites for other specialties,^5-9 resulting in various recommendations for areas of improvement among their respective program websites to both help applicants and increase recruitment.^6,9-11 The main purpose of our study was to provide an in-depth analysis of EM residency website content for prospective EM applicants. To our knowledge, this study is the first to evaluate these variables in the specialty of EM.

METHODS

This study was exempt from institutional review bgoard approval because it involves publicly available information. Our methods were adapted from a previous study analyzing otolaryngology residency websites.⁶ We obtained a list of 256 EM programs from FREIDA in April 2020. When a link to a program was not available on FREIDA, we performed a Google search to find the program website. Residency programs without a functional residency website or a website that could not be found were excluded. When two websites were available for the same program, we used information from both the institutional and the non-institutional program website. We did not include Facebook, Instagram, or other social media page information.

We searched the available websites of these programs for 38 items listed in Supplemental Table. Items fell into seven general categories: general program information; application process; research; facility information; current resident information; lifestyle; and social media use. These 38 items were included in our study based on our literature review of previous analyses of residency and fellowship websites in other specialties.^5-13 As descriptive studies, they analyzed a heterogenous list of variables on the websites of interest. The 38 items included in our study are largely based on this literature search.

Understandably, some factors are more important to applicants than others, such as patient volume, curriculum, faculty, research, and simulation training centers.¹⁴ Although the importance of these factors varies from applicant to applicant, the majority of the variables applicants deem as necessary or desirable information were included in our list of 38 items, in addition to many more items of potential interest we included based on our literature review.¹⁵ We also added a few additional items to make our study more comprehensive based on items we believe could be lacking from other studies in our literature review, such as social media, resident pictures, and resident hobbies. We also tested the websites for functionality by determining whether the link provided on FREIDA led directly to the residency homepage or required multiple clicks to get to the homepage. The data were collected by three authors (JW, AR, SS) from April 15–May 15, 2020.

As the data contained in residency websites can be subjective, we created a standardized process to evaluate the websites, similar to the previous studies in other specialties.^5-9,12 First, we only searched for the presence or absence of items, with no attempt made to grade the quality or accuracy of the content. Second, we excluded any information that was not directly listed on the residency website, such as links to external materials, which may contain general, non-specific information for the program of interest. Lastly, we piloted our search criteria for five programs to resolve ambiguity through independent review by four authors. After this instruction, data collectors independently gathered the data from the 251 remaining websites (HW, AR, SS). When these three authors encountered websites or criteria that were unclear, these criteria were marked and reviewed by a fourth author (JP) and classified accordingly.

We performed a descriptive analysis of the data, including means and standard deviations. To calculate the percentage of items present in each subcategory, we added up the total number of items present among all 249 websites in that subcategory. We then divided this number by the denominator, which was calculated by multiplying the number of variables present in a subcategory by the number of websites examined (249).Microsft Excel 2020 version 16 (Microsoft Corp., Redmond, WA) was used for statistical analysis.

RESULTS

Of 256 EM residency programs included in our study, seven programs did not have websites on FREIDA and were not accessible by Google search. Of the 249 websites evaluated, 110 websites (44%) provided a direct link from FREIDA to the residency homepage, while 107 (43%) programs required multiple clicks to get to the residency homepage. Thirty-two programs (13%) did not have a link on FREIDA.

On average, websites contained 20 of 38 items (53%) with a standard deviation of 6.35. Only 16 programs (6%) contained at least three-quarters of the items of interest. One program contained 37 of 38 items, and no program contained all items. The items that were least commonly available on websites included information on housing and neighborhoods (17%) and social media links (19%) (Table). Within the social media category, the most common forms of social media were Twitter (15%), Facebook (12%), and Instagram (8%) (Figure 1). The items most commonly provided by websites included program description (98%), blocks and rotations (91%), faculty listing (88%), and description for each year of residency (87%). The percentage of each general category is included in Figure 2.

DISCUSSION

As prospective applicants evaluate EM programs, careful planning and research is essential. The internet is easily accessible, and multiple studies have shown the importance of websites in recruitment.^9,12,16,2 Our study suggests databases such as FREIDA are useful tools to navigate residency options, with 87% of EM programs providing links to their sites. However, over half of these links required multiple steps, suggesting even this resource could be improved. Our results also demonstrate that many websites are lacking information that is potentially valuable to residency applicants, with an average site missing nearly half of the information we evaluated. We believe enhancing website content could improve the application process for all parties.

The “People” section on websites provided widely varying amounts of information. Despite previous analyses demonstrating that this is the most popular content on EM residency websites,¹⁷ this information was present on only 63% of EM sites. Similarly, resident biographies and a description of resident hobbies and/or interests were included on fewer than half of EM residency websites. Because an applicant is unlikely to meet all current residents during an interview, resident information on program websites could be the only exposure of such applicants to the unique personalities and backgrounds of residents in the program. Emergency medicine residency programs may benefit from improving these areas of their websites, while being cautious to protect the personal information of their residents.

The presence of social media on residency websites was also limited. Despite the rise of social media for recreational and professional purposes, only 19% of programs contained links to a form of social media for their EM program. In a study done involving nearly 1000 medical students, 68% of students reported using social media to learn about residency programs and 10% reported that the information found in the social media pages influenced their decisions on where to apply.¹⁸ Similarly, a survey of 142 prospective EM residents led to the recommendation that programs should highlight social activities to improve resident recruitment,² and social media is an efficient way to display social activities. Professional social media integration with EM residency websites appears currently underutilized.

Additionally, lifestyle factors including salary, vacation, meal allowance, and housing costs were often not found on websites. For example, salary information was listed on 63% of websites, vacation time was listed on 57% of websites, and housing and neighborhood information was listed on 17% of websites. Not only are these items important factors to consider when choosing a residency program, but they can be difficult to ask about during an interview, as applicants may worry that asking questions about compensation and benefits give the wrong impression to the program leadership and residents. These topics are discussed during interviews, but these details can be forgotten. Accessible information on lifestyle, which is currently lacking among many EM residency websites, could eliminate this source of potential inquiry or recall concern for applicants.

As the number of residency programs in EM continues to grow,¹⁹ it has become increasingly difficult for applicants to choose where to apply for visiting clerkships or residency interviews. In 2020. US medical school graduates applying for EM residency applied to an average of 58 programs¹ despite data from the Association of American Medical Colleges demonstrating diminishing returns for applicants applying to more than 32–39 programs.²⁰ While the increased number of applications may be due to the increasingly competitive nature of EM,^21,22 future studies should aim to determine whether lack of online information affects the number of applications. The residency application and interview process is costly for applicants and programs. Providing applicants with more information to guide decisions regarding which programs to apply to and interview at stands to benefit both parties, especially if it results in a better matching of applicants likely to fit a particular program.

The 2020–21 residency application cycle posed a new challenge for applicants and programs. The lack of availability of visiting rotations and in-person interviews contributed to increased uncertainty among applicants, and most were unable to evaluate programs in person. In-person interviews served not only for programs to interview candidates, but for candidates to evaluate programs. Therefore, more than ever, a robust source of information available to applicants on a residency website serves to benefit both applicant and program alike.

LIMITATIONS

Limitations of our study include the subjective nature of analyzing residency program websites. However, we feel our method of data collection was standardized sufficiently to control for ambiguity. Another limitation was the lack of established standardized criteria for evaluating websites. We based our list of 38 items on our literature search of previous residency website analyses in other specialties, and also relied on papers relating to what EM applicants deem important and the expertise of the authors of our study.^5-13 However, the purpose of our study was not to define the most important items for EM residency applicants, but rather to assess the presence or absence of items on EM residency websites. Inherently, there are items present in our list that may not be very important to some applicants, and items missing which may be important to some applicants.

Future study is needed to provide an updated list of the most important criteria EM applicants could be interested in. The number of programs with “People” sections could be underestimated as many of these programs might have these sections on their social media websites rather than their official residency program websites. Lastly, only including items listed directly on the EM residency website rather than on external links could underestimate the presence of items on websites in our study. However, this was an important factor to determine the accessibility of information and user-friendly status of the websites. Our study does not address accuracy or quality of information contained on websites.

CONCLUSION

Residency program website quality is important to EM applicants, and our study identifies several areas where programs could focus efforts for website renovation, including improving the integration of social media and providing information on residents and residency lifestyle. The results from our study can be used to improve EM residency websites to the mutual benefit of both applicants and residency programs.

 

Population Health Research Capsule

What do we already know about this issue?

Residency program websites have been analyzed in other specialties. However, a comprehensive analysis of emergency medicine (EM) residency websites is lacking.

What was the research question?

Which components of EM residency websites are most common, and which are least common?

What was the major finding of the study?

We identified several areas for website renovation, such as social media integration and residency lifestyle.

How does this improve population health?

The results from our study can be used to improve EM residency websites to the mutual benefit of both applicants and residency programs.

 

Footnotes

Section Editor: Danya Khoujah, MBBS

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Jordan R. Pollock, BS, Mayo Clinic Alix School of Medicine, 13400 E. Shea Boulevard, Scottsdale, Arizona 85259. Email: pollock.jordan@mayo.edu. 7 / 2021; 22:937 – 942

Submission history: Revision received October 5, 2020; Submitted April 7, 2021; Accepted April 8, 2021

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Ramsay N. EM Match 2020 By the Numbers. 2020. Available at: http://www.emra.org/students/newsletter-articles/em-match-2020-by-the-numbers/. Accessed April 7, 2021.

2. Mahler SA, Wagner M-J, Church A, et al. Importance of residency program web sites to emergency medicine applicants. J Emerg Med. 2009;36(1):83-8.

3. Gaeta TJ, Birkhahn RH, Lamont D, et al. Aspects of residency programs’ web sites important to student applicants. Acad Emerg Med. 2005;12(1):89-92.

4. FREIDA Residency Program Database | Medical Fellowship Database | AMA. Available at: https://freida.ama-assn.org/. Accessed April 7, 2021.

5. Silvestre J, Tomlinson-Hansen S, Fosnot J, et al. Plastic surgery residency websites: a critical analysis of accessibility and content. Ann Plast Surg. 2014;72(3):265-9.

6. Svider PF, Gupta A, Johnson AP, et al. Evaluation of otolaryngology residency program websites. JAMA Otolaryngol Neck Surg. 2014;140(10):956-60.

7. Savan M, Youssef I, Gupta A, et al. An assessment of the comprehensiveness of radiation oncology residency websites. Appl Rad Oncol. 2020;9(1):24-7.

8. Hamid S, Gibney B, Niu B, et al. An Evaluation of the content of Canadian and American nuclear medicine fellowship websites. Can Assoc Radiol J. 2020.

9. Khan MS, Hayat J, Marsia S, et al. How well do we represent ourselves: an analysis of cardiology fellowships website content. Future Cardiol. 2020;16(4):281-7.

10. Oladeji LO, Yu JC, Oladeji AK, et al. How useful are orthopedic surgery residency web pages?. J Surg Educ. 2015;72(6):1185-9.

11. Patel BG, Gallo K, Cherullo EE, et al. Content analysis of ACGME-accredited urology residency program webpages. Urology. 2020;138:11-5.

12. Cleary MX, Yi PH, Ahlawat S. Evaluation of musculoskeletal radiology fellowship websites. Curr Probl Diagn Radiol. 2021;50(3):379-83.

13. Ashack KA, Burton KA, Soh JM, et al. Evaluating dermatology residency program websites. Dermatol Online J. 2016;22(3):13030.

14. Zigrossi D, Ralls G, Martel M, et al. Ranking programs: medical student strategies. J Emerg Med. 2019;57(4):e141-e145.

15. Deloney LA, Perrot LJ, Lensing SY, et al. Radiology resident recruitment: a study of the impact of web-based information and interview day activities. Acad Radiol. 2014;21(7):931-7.

16. Embi PJ, Desai S, Cooney TG. Use and utility of web-based residency program information: a survey of residency applicants. J Med Internet Res. 2003;5(3):e22.

17. Dyer S, Dickson B, Chhabra N. Utilizing analytics to identify trends in residency program website visits. Cureus. 2020;12(2):e6910.

18. Schweitzer J, Hannan A, Coren J. The role of social networking web sites in influencing residency decisions. J Am Osteopath Assoc. 2012;112(10):673-9.

19. Nelson LS, Keim SM, Baren JM, et al. American Board of Emergency Medicine Report on Residency and Fellowship Training Information (2017–2018). Ann Emerg Med. 2018;71(5):636-48.

20. Apply smart: data to consider when applying to residency. Available at: https://students-residents.aamc.org/applying-residency/filteredresult/apply-smart-data-consider-when-applying-residency/. Accessed April 7, 2021.

21. Vassar L. Here’s how many residency programs med students really apply to. 2015. Available at: https://www.ama-assn.org/residents-students/match/heres-how-many-residency-programs-med-students-really-apply. Accessed April 7, 2021.

22. Guttman D. A medical student’s perspective: Selecting the best emergency medicine residency program. J Emerg Med. 2005;28(1):105-6.