Author	Affiliation
Stephen Villa, MD	University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California; David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California
Hannah Janeway, MD	University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California
Kian Preston-Suni, MD, MPH	University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California; David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California; Greater Los Angeles VA Healthcare System, Department of Emergency Medicine, Los Angeles, California
Ashley Vuong, MD	University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California
Ignacio Calles, MD	University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California
James Murphy, MD	University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California
Taylor James, MD	University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California
Jaime Jordan, MD, MAEd	University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California; David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California
Andrew Grock, MD	David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California; Greater Los Angeles VA Healthcare System, Department of Emergency Medicine, Los Angeles, California
Natasha Wheaton, MD	University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California; David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California

Background
Curriculum design
Objectives
Impact/ effectiveness
Conclusion

BACKGROUND

The coronavirus 2019 (COVID-19) pandemic drastically altered educational and residency application landscapes for emergency medicine (EM)-bound medical students by restricting in-person visiting clerkships.^1-5 Visiting clerkships traditionally have been critical for EM-bound students.⁶ Medical students who are EM bound typically complete at least two EM clerkships to gain additional experience and explore varied training environments. Visiting clerkships also provide Standardized Letters of Evaluation (SLOE) – crucial to the residency application – and allow for reciprocal exposure between the student and program.^6-8 In response to COVID-19 restrictions, many medical schools and residencies pivoted to online education and cancelled visiting rotations.^9,10 With the loss of in-person visiting clerkships, a novel virtual curriculum was urgently needed to fill this void.

As an alternative to traditional in-person visiting clerkships, we created, implemented, and evaluated a virtual “visiting” clerkship with a focus on advanced and less commonly taught topics (ie, social EM and professional development). Based on the most recent recommendations, many institutions have removed restrictions on in-person rotations but continue to limit visiting rotations to one per student.¹¹ Looking forward, the virtual environment creates a unique opportunity for programs to continue to meet their applicants more in depth, in addition to circumventing geographic and socioeconomic barriers often faced by students participating in traditional visiting rotations.

CURRICULUM DESIGN

Rather than replicating a traditional clerkship virtually, we designed our curriculum to focus on advanced medical topics: social determinants of health; structural competency; and professional identity formation^{12, 13} by employing Kern’s method for curriculum development.¹⁴ We identified educational needs as institutional COVID-19 restrictions were released. We performed a needs assessment including data from our postgraduate year one class as near peers. Our topic list was further refined by consensus among the author group, which included a clerkship director, associate program directors, medical education fellow, and senior EM residents. We developed goals and objectives informed by the topic list and the additional goals of exposing students to our residency program and social EM, as well as advancing professional identity formation. Our traditional in-person sub-internship experience typically covers medical knowledge topics commonly seen in the emergency department as well as skills to help learners thrive while rotating in person. In addition to being vastly different from a traditional experience given that it would be delivered virtually, we felt that this rotation could possibly serve as an ideal environment to cover social EM and professional identity formation, topics that would benefit from minimal interruptions or competing pressures.

OBJECTIVES

Curriculum goals included teaching advanced EM clinical knowledge, introducing social EM and professional identity formation, and providing exposure to our residency program. See Table 1 for course goals and objectives.

When choosing educational methods, we used the conceptual framework developed by Brown et al to maximize online learning and engagement. This framework encourages expectation management, learner engagement, and “nudging.”¹⁵ Our orientation outlined expectations, including asynchronous assignments and recommended norms for small group. We prioritized interactive teaching modalities and active learning to maximize engagement such as small-group learning among as well as our “Virtual Escape Room” (Appendix A) and simulation. Our small group, case-based discussions used a flipped classroom model, an effective and recommended modality for virtual instruction.^16-22 All small-group facilitators were reminded of the best practices for online, small group teaching,²³ which included use of introductions, learner-directed questioning to encourage equal participation, and “nudging” – reminders for learners to actively participate (See Appendix B).

Each virtual clerkship session was held on weekdays over two weeks for a total of 10 instruction days. The students were expected to complete various asynchronous learning assignments (estimated two hours daily) and attend four hours of Zoom (Zoom Video Communications, Inc, San Jose, CA) sessions daily. (See Appendix C for example schedule and specific content.) Cases from “Foundations of EM,” a national free, open-access online resource, were used to teach medical knowledge.²⁴ Social medicine instruction was done using modules from the International and Domestic Health Equity and Leadership (IDHEAL) Section from the University of California, Los Angeles.²⁵ Chosen modules included Language, Incarceration, Gender Identity, Race, and Homelessness, and assigned readings from those modules were delivered to learners via email. The Virtual Escape Room consisted of a tricyclic antidepressant overdose case, created by authors AV and TJ (Appendix A) with inspiration from another published escape room.²⁶ The virtual simulation was carried out over Zoom, and cases from our traditional clerkship were used, which cover pediatric anaphylaxis, motorcycle trauma, hypothermia, and abdominal aortic aneurysm. The “Communities of Practice Panel” consisted of a panel of faculty/attendings who practice in different EM environments (ie, tertiary referral center, county, community, Veterans Administration, and critical access community). Many of these modalities have previously been highlighted as effective teaching modalities.^27,28 Within the professional identity formation theme, learners read Carol Dweck’s “Mindset” to prepare for a book club-type discussion – a modality previously well-received by other learners.²⁹ Finally, students were introduced to basic pedagogical techniques and practiced non-medical teaching sessions for their peers and faculty; feedback was provided.

Asynchronous assignments consisted of Emergency Medicine Reviews and Perspective C3 podcasts, free open-access medicine (FOAM) curated by the Academic Life in EM (ALiEM) AIR series, Foundations of EM “Frameworks,” and articles introducing topics of social EM.^24,25,30,31 Asynchronous content was designed to correspond to daily synchronous content; specifics can be found on Appendix C. We made our virtual clerkship available to all fourth-year medical students applying into EM via the Visiting Student Application Service (VSAS) website and offered it twice during the 2020–2021 academic year.³³ Given our predicted teaching resources, we estimated an ideal class size of less than 25 students per session. Ultimately, 26 students enrolled (nine in the first session, 17 in the second session). Attendance at all sessions was mandatory.

We recruited a group of residents and faculty to teach for a total of 24 lecturers and 26 small-group facilitators across both sessions. Facilitating the clerkship during the two-week session required one of three clerkship directors to be present on Zoom four hours per day, in addition to administrative tasks related to that day’s activities.

We assessed medical knowledge with a 30-item, peer-reviewed, multiple-choice test consisting of questions donated by RoshReview (Rosh Review LLC, Huntington Woods, MI), a commercial question bank company. RoshReview validates questions against real-world exam performance such as the in-training exam for EM residents.³² Questions were chosen by the course directors by a systems-based approach (ie, neurology, cardiovascular, etc) with the goal of choosing questions that were reflective of clerkship’s curriculum. Students did not have the ability to see the answers to the questions after taking the pre-test. Because the test was conducted at home, students theoretically could access open-access content in real time. Students completed the same medical knowledge test on the first and last days of the clerkship.³⁴ We calculated mean scores and compared pre-and post-tests using a paired t-test, analyzed with the software statistical package Stata 16.1 (StataCorp, College Station, TX). Test scores had no bearing on final grades.

Each two-week clerkship contained five social EM sessions, which were assessed by an anonymous survey exploring previous experience with the topic and comfort with applying content learned to the clinical environment (Appendix D). This tool had been previously used in residency education by the IDHEAL group. It was developed by content experts after literature review to maximize content validity. We analyzed questions relevant to this clerkship. Participants were sent an online anonymous survey after each of the five sessions. In total, we sent a total of 130 surveys (five surveys per participant) and analyzed the data descriptively.

We assessed students’ overall attitude toward the course with a 16-item evaluative survey consisting of 11 multiple-choice, one slider scale, and four free-response items. The survey was created by SV, who has a Master’s in Education, had advanced training in survey design and experience in qualitative research, and was the course director, all providing content validity evidence. We read survey items aloud among the author group and piloted the survey with a small reference population to optimize response process validity. The survey was distributed on the last day of class (Appendix E). We calculated and reported descriptive statistics for survey questions with discrete answer choices. For free-response data within the survey, two authors (SV and AV) performed a thematic analysis. SV trained AV, a senior resident, to perform a thematic qualitative analysis. The analysts independently reviewed the data and later met to establish a final coding scheme, which they then independently applied to all data. After applying the final coding scheme, they identified discrepancies and finalized themes. The simple percent agreement between the two analysts was 80.3%. Discrepancies were resolved via in-depth discussion and negotiated consensus.

This study was deemed exempt by the University of California, Los Angeles Internal Review Board (IRB #20-002014) approved on November 19, 2020.

IMPACT/ EFFECTIVENESS

Twenty-six students participated in the virtual clerkship representing 22 medical schools and all regions of the US. All students completed the medical knowledge pre- and post-test. Mean test scores improved from 21.5 (standard deviation [SD] +/−2.6) to 22.9 (SD +/− 1.24) (P = 0.006), effect size 0.68, 95% confidence interval, 0.12–1.24.

Of the 130 IDHEAL post-module surveys administered, 98 (75%) were completed. Of the modules chosen, incarceration was least likely to have been previously covered with only 6% (1/18) of respondents having prior instruction. Eighty-nine percent (87/98) of respondents “strongly agreed” that these topics were important for patient care in the ED, and 66% (65/98) felt more confident after completing the modules. See Table 2 for full results.

Almost all (25/26, 96%) students completed the end-of-rotation evaluative survey. Of all respondents, 95% “strongly agreed” or “agreed” that the rotation should be repeated in the future, and all “strongly agreed” or “agreed” that the rotation would impact the way they ranked our program. Major themes from the qualitative analysis are described in Table 3.

Prior literature has demonstrated knowledge acquisition and retention from virtual curricula, and we saw similar results, albeit our study demonstrated only a modest improvement.^35,36 One explanation for the lack of larger change is that our assessment items may not have been perfectly aligned with our curriculum as the questions were pulled from a standard question bank. For future versions, we would strongly consider constructing and validating our own internal assessment of medical knowledge to be better aligned with our objectives. Additionally, students rated themselves as more confident in discussing and managing social medicine topics. Ideally, we would be able to conduct a repeat assessment at a predetermined timepoint to assess whether the social EM content had modified their practice as residents. While we emphasized our program’s strong social EM vision, other programs may replicate this curriculum to focus on their own strengths. In the past, visiting clerkships have acted as a recruiting tool for residencies,⁸ and virtual clerkships may also allow residencies to highlight strengths and successfully recruit.

Students from 22 institutions participated in our clerkship at minimal cost to them (only the cost to apply via VSAS). In contrast, EM applicants averaged 1.9 visiting rotations costing almost $1000 per rotation in 2019.^37-39 While many institutions have implemented scholarships for under-represented in medicine students, virtual clerkships remove financial barriers for all students and may be an invaluable option for students with familial or other obligations.⁴⁰ Virtual clerkships represent an additional strategy to help mitigate the socioeconomic barriers of visiting rotations.

While students perceived our virtual experience to be valuable, several challenges were encountered. The clerkship required significant administrative efforts and a large number of facilitators to create the intimate small-group experiences critical to its success. There was no protected time or funding for the instructors. Overall, at least 40 hours per clerkship among NW and SV were required, which did not account for planning and time from all instructors who volunteered their time. These requirements may be adjusted by limiting the number of students enrolled. Furthermore, as we look to the future, simultaneously administrating a virtual clerkship and in-person clerkship will likely require significant additional administrative support.

While this rotation ultimately served 25 students, we considered this rotation to be a success as we were fortunate to match three interns of our current class from the Virtual Clerkship. If we were to repeat this clerkship again, we would expand our evaluation efforts as it was limited, mainly only allowing for assessment in small groups. One other addition would be to encourage asynchronous communication among students and faculty. Examples of communication would be continued improvement of the environment and incorporation of daily questions expanding on the day’s content to further enhance spaced repetition. Lastly, interest in our virtual clerkship was likely increased due to COVID-19 restrictions on in-person opportunities. Future versions will require more advertisement and may not bolster as much interest.

We envision we will offer both versions of each clerkship separately moving forward. However, we likely would not offer a formal SLOE to students who pursue the Virtual Clerkship given we cannot comment on their clinical skills in the virtual format. However, we would gladly write a letter of recommendation as, in some ways, program leadership may get to know these students in a more personal way, especially with certain aspects of the SLOE such as “commitment to EM.” Finally, we may incorporate some of the social EM content and other teaching modalities into the traditional clerkship.

CONCLUSION

This virtual clerkship was created in response to an acute educational need created by the COVID-19 pandemic. However, our experience suggests that virtual learning experiences may be valuable in the future as an adjunct to traditional in-person rotations. Virtual rotations provide flexibility allowing for the incorporation of topics not traditionally taught (eg, social EM), allow residencies and students increased access to one another, and may eliminate socioeconomic barriers advancing educational equity.

Footnotes

Section Editor: Douglas Ander, MD and Andrew Phillips, MD, MEd

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Stephen Villa, MD, David Geffen School of Medicine, Department of Emergency Medicine, 10833 Le Conte Avenue, Los Angeles, CA 90095. Email: sevilla@mednet.ucla.edu. 1 / 2022; 23:33 – 39

Submission history: Revision received July 14, 2021; Submitted September 30, 2021; Accepted November 4, 2021

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Kaul V, Gallo De Moraes A, Khateeb D, et al. Medical education during the COVID-19 pandemic. Chest. 2020.

2. Important guidance for medical students on clinical rotations during the coronavirus (COVID-19) outbreak. 2020. Available at: https://www-aamc-org.libproxy2.usc.edu/news-insights/press-releases/important-guidance-medical-students-clinical-rotations-during-coronavirus-covid-19-outbreak. Accessed December 23, 2020.

3. Final report and recommendations for medical education institutions of LCME-accredited, U.S. osteopathic, and non-U.S. medical school applicants. 2020. Available at: https://www-aamc-org.libproxy2.usc.edu/system/files/2020-05/covid19_Final_Recommendations_05112020.pdf. Accessed December 23, 2020.

4. Katirji L, Smith L, Pelletier-Bui A, et al. Addressing challenges in obtaining emergency medicine away rotations and Standardized Letters of Evaluation due to COVID-19 pandemic. West J Emerg Med. 2020;21(3):538-41.

5. Aiyer AA, Granger CJ, McCormick KL, et al. The impact of COVID-19 on the orthopaedic surgery residency application process. J Am Acad Orthop Surg. 2020;28(15):e633-e641.

6. Negaard M, Assimacopoulos E, Harland K, et al. Emergency medicine residency selection criteria: an update and comparison. AEM Educ Train. 2018;2(2):146-53.

7. Mueller PS, McConahey LL, Orvidas LJ, et al. Visiting medical student elective and clerkship programs: a survey of US and Puerto Rico allopathic medical schools. BMC Med Educ. 2010;10:41.

8. Haas MRC, He S, Sternberg K, et al. Reimagining residency selection: Part 1-A Practical Guide to Recruitment in the Post-COVID-19 Era. J Grad Med Educ. 2020;12(5):539-44.

9. Rolak S, Keefe AM, Davidson EL, et al. Impacts and challenges of United States medical students during the COVID-19 pandemic. World J Clin Cases. 2020;8(15):3136-41.

10. Wilcha R. Effectiveness of virtual medical teaching during the COVID-19 crisis: systematic review. JMIR Med Educ. 2020;6(2):e20963.

11. Moreno-Walton L, Jung J, Aintablian H, et al. Consensus statement on the 2021–2022 emergency medicine application cycle: Part 1: Emergency Medicine Rotations and Letters. 2021. Available at: https://www.emra.org/be-involved/be-an-advocate/working-for-you/match-cycle-2022/. Accessed July 12, 2021.

12. Manthey DE, Ander DS, Gordon DC, et al. Emergency medicine clerkship curriculum: an update and revision. Acad Emerg Med. 2010;17(6):638-43.

13. Thomas PA, Kern DE, Hughes MT, et al. Curriculum Development for Medical Education: A Six-Step Approach. 2016.

14. Kern DE. Curriculum Development for Medical Education: A Six Step Approach. 1998.

15. Brown A, Lawrence J, Basson M, et al. A conceptual framework to enhance student online learning and engagement in higher education. Higher Education Research & Development. 2019.

16. Kraut AS, Omron R, Caretta-Weyer H, et al. The flipped classroom: a critical appraisal. West J Emerg Med. 2019;20(3):527-36.

17. Liebert CA, Lin DT, Mazer LM, et al. Effectiveness of the surgery core clerkship flipped classroom: a prospective cohort trial. Am J Surg. 2016;211(2):451-7.

18. O’Connor EE, Fried J, McNulty N, et al. Flipping radiology education right side up. Acad Radiol. 2016;23(7):810-22.

19. Rose E, Claudius I, Tabatabai R, et al. The flipped classroom in emergency medicine using online videos with interpolated questions. J Emerg Med. 2016;51(3):284-91.

20. Chen KS, Monrouxe L, Lu YH, et al. Academic outcomes of flipped classroom learning: a meta-analysis. Med Educ. 2018;52(9):910-24.

21. Yiu SHM, Spacek AM, Pageau PG, et al. Dissecting the contemporary clerkship: theory-based educational trial of videos versus lectures in medical student education. AEM Educ Train. 2019;4(1):10-7.

22. Young TP, Bailey CJ, Guptill M, et al. The flipped classroom: a modality for mixed asynchronous and synchronous learning in a residency program. West J Emerg Med. 2014;15(7):938-44.

23. Boettcher J, Conrad R. The Online Teaching Survival Guide: Simple and Practical Pedagogical Tips. 2016.

24. 2021. Available at: Foundationsem.com. Accessed February 26, 2021.

25. International and Domestic Health Equity and Leadership. 2021. Available at: http://www.idheal-ucla.org/page-12/. Accessed February 26, 2021.

26. Cates A. Idea Series: Toxicology virtual escape room during COVID-19. 2020. Available at: https://www.aliem.com/idea-series-toxicology-virtual-escape-room-during-covid-19/. Accessed October 18, 2021.

27. Gottlieb M, Landry A, Egan DJ, et al. Rethinking residency conferences in the era of COVID-19. AEM Educ Train. 2020;4(3):313-7.

28. He S, Lai D, Mott S, et al. Remote e-work and distance learning for academic medicine: best practices and opportunities for the future. J Grad Med Educ. 2020;12(3):256-63.

29. Jordan J, Bavolek R, Dyne P, et al. A virtual book club for professional development in emergency medicine. West J Emerg Med. 2020;22(1):108-14.

30. EM:RAP C3 podcasts. Available at: https://www.emrap.org/c3. Accessed February 26, 2021.

31. ALiEM AIR series. Available at: https://www.aliem.com/aliem-approved-instructional-resources-air-series/. Accessed February 26, 2021.

32. Micheal S. Using Rosh Review to improve education outcomes (Part 1). 2017. Available at: https://www.roshreview.com/blog/unlocking-the-data-behind-rosh-review-to-improve-educational-outcomes-part-1/. Accessed October 18, 2021.

33. Visiting student learning opportunities. Available at: https://students-residents.aamc.org/attending-medical-school/article/visiting-student-learning-opportunities/. Accessed February 26, 2021.

34. ROSH review. 2021. Available at: https://www.roshreview.com. Accessed February 26, 2021.

35. Dost S, Hossain A, Shehab M, et al. Perceptions of medical students towards online teaching during the COVID-19 pandemic: a national cross-sectional survey of 2721 UK medical students. BMJ Open. 2020;10(11):e042378.

36. Kronenfeld J, Ryon E, Kronenfeld D. Medical student education during COVID-19: Electronic education does not decrease examination scores. Am Surg. 2020:1-7.

37. Away Rotations of U.S Medical School Graduates by Intended Specialty, 2019 AAMC Medical School Graduation Questionnaire (GQ). 2019. Available at: https://students-residents.aamc.org/media/9501/download. Accessed February 26, 2021.

38. Winterton M, Ahn J, Bernstein J. The prevalence and cost of medical student visiting rotations. BMC Med Educ. 2016;16(1):291.

39. Blackshaw AM, Watson SC, Bush JS. The cost and burden of the residency match in emergency medicine. West J Emerg Med. 2017;18(1):169-73.

40. Boatright D, Branzetti J, Duong D, et al. Racial and ethnic diversity in academic emergency medicine: How far have we come? Next steps for the future. AEM Educ Train. 2018;2(Suppl Suppl 1):S31-S39.