| Author | Affiliation |
|---|---|
| Stephen Villa, MD | University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California; David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California |
| Hannah Janeway, MD | University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California |
| Kian Preston-Suni, MD, MPH | University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California; David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California; Greater Los Angeles VA Healthcare System, Department of Emergency Medicine, Los Angeles, California |
| Ashley Vuong, MD | University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California |
| Ignacio Calles, MD | University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California |
| James Murphy, MD | University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California |
| Taylor James, MD | University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California |
| Jaime Jordan, MD, MAEd | University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California; David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California |
| Andrew Grock, MD | David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California; Greater Los Angeles VA Healthcare System, Department of Emergency Medicine, Los Angeles, California |
| Natasha Wheaton, MD | University of California – Los Angeles, Department of Emergency Medicine, Los Angeles, California; David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California |
Background
Curriculum design
Objectives
Impact/ effectiveness
Conclusion
ABSTRACT
Introduction
Safety concerns surrounding the coronavirus 2019 pandemic led to the prohibition of student rotations outside their home institutions. This resulted in emergency medicine (EM)-bound students having less specialty experience and exposure to outside programs and practice environments, and fewer opportunities to gain additional Standardized Letters of Evaluation, a cornerstone of the EM residency application. We filled this void by implementing a virtual clerkship.
Methods
We created a two-week virtual, fourth-year visiting clerkship focused on advanced medical knowledge topics, social determinants of health, professional development, and professional identity formation. Students completed asynchronous assignments and participated in small group-facilitated didactic sessions. We evaluated the virtual clerkship with pre- and post-medical knowledge tests and evaluative surveys.
Results
We hosted 26 senior medical students over two administrations of the same two-week virtual clerkship. Students had a statistically significant improvement on the medical knowledge post-tests compared to pre-tests (71.7% [21.5/30] to 76.3% [22.9/30]). Students reported being exposed to social determinants of health concepts they had not previously been exposed to. Students appreciated the interactive nature of the sessions; networking with other students, residents, and faculty; introduction to novel content regarding social determinants of health; and exposure to future career opportunities. Screen time, technological issues, and mismatch between volume of content and time allotted were identified as potential challenges and areas for improvement.
Conclusion
We demonstrate that a virtual EM visiting clerkship is feasible to implement, supports knowledge acquisition, and is perceived as valuable by participants. The benefits seen and challenges faced in the development and implementation of our clerkship can serve to inform future virtual clerkships, which we feel is a complement to traditional visiting clerkships even though in-person clerkships have been re-established.
BACKGROUND
The coronavirus 2019 (COVID-19) pandemic drastically altered educational and residency application landscapes for emergency medicine (EM)-bound medical students by restricting in-person visiting clerkships.1-5 Visiting clerkships traditionally have been critical for EM-bound students.6 Medical students who are EM bound typically complete at least two EM clerkships to gain additional experience and explore varied training environments. Visiting clerkships also provide Standardized Letters of Evaluation (SLOE) – crucial to the residency application – and allow for reciprocal exposure between the student and program.6-8 In response to COVID-19 restrictions, many medical schools and residencies pivoted to online education and cancelled visiting rotations.9,10 With the loss of in-person visiting clerkships, a novel virtual curriculum was urgently needed to fill this void.
As an alternative to traditional in-person visiting clerkships, we created, implemented, and evaluated a virtual “visiting” clerkship with a focus on advanced and less commonly taught topics (ie, social EM and professional development). Based on the most recent recommendations, many institutions have removed restrictions on in-person rotations but continue to limit visiting rotations to one per student.11 Looking forward, the virtual environment creates a unique opportunity for programs to continue to meet their applicants more in depth, in addition to circumventing geographic and socioeconomic barriers often faced by students participating in traditional visiting rotations.
CURRICULUM DESIGN
Rather than replicating a traditional clerkship virtually, we designed our curriculum to focus on advanced medical topics: social determinants of health; structural competency; and professional identity formation12, 13 by employing Kern’s method for curriculum development.14 We identified educational needs as institutional COVID-19 restrictions were released. We performed a needs assessment including data from our postgraduate year one class as near peers. Our topic list was further refined by consensus among the author group, which included a clerkship director, associate program directors, medical education fellow, and senior EM residents. We developed goals and objectives informed by the topic list and the additional goals of exposing students to our residency program and social EM, as well as advancing professional identity formation. Our traditional in-person sub-internship experience typically covers medical knowledge topics commonly seen in the emergency department as well as skills to help learners thrive while rotating in person. In addition to being vastly different from a traditional experience given that it would be delivered virtually, we felt that this rotation could possibly serve as an ideal environment to cover social EM and professional identity formation, topics that would benefit from minimal interruptions or competing pressures.
OBJECTIVES
Curriculum goals included teaching advanced EM clinical knowledge, introducing social EM and professional identity formation, and providing exposure to our residency program. See Table 1 for course goals and objectives.
Table 1Course goals and objectives of a virtual emergency medicine clerkship.
| Goals | Objectives |
|---|---|
| To build upon existing EM knowledge through less commonly taught core EM chief complaints
To expose students to the broad variety of ED practice environments and patient populations they will care for through panels and case-based discussions
To improve the knowledge base, and importance of justice in healthcare in caring for ED patients of diverse socioeconomic statuses, race, ethnicities, gender, and sexual orientations.
To introduce students to clinical and non-clinical niches in EM including toxicology, critical care, ultrasound, EMS, medical education, research, healthcare administration, and social determinants of health
To expose students to a variety of learning modalities including practicing their own teaching skills
To introduce the concept of professional skill-set development and how growth mindset may impact clinical encounters
|
By the conclusion of this rotation, the students should be able to:Medical Knowledge 1. Describe an approach to several commonly seen chief complaints in EM.Social EM 2. Compare how different practice environments, associated healthcare systems, and access to care affect care plans.
3. Discuss areas within medicine, including within EM, how biases may affect patient care and create strategies to overcome one’s own bias.
4. Describe how language, race, gender, homelessness, and addiction affect patient care. Professional Identity Formation 5. Describe clinical and non-clinical practice environments as well as niches within EM.
6. Understand the importance of a growth mindset over a fixed mindset and develop strategies to incorporate a growth mindset.
7. Apply principles of growth mindset to commonly experienced scenarios in the clinical setting.
8. Describe challenges of interviewing virtually.
9. Outline effective strategies for identifying medical content, learning, and organizing medical knowledge in the 21st century.
10. Describe challenges and opportunities of teaching in the 21st century.
11. Demonstrate ability to teach peers on pre-selected topic.
|
EM, emergency medicine; ED, emergency department; EMS, emergency medical services.
When choosing educational methods, we used the conceptual framework developed by Brown et al to maximize online learning and engagement. This framework encourages expectation management, learner engagement, and “nudging.”15 Our orientation outlined expectations, including asynchronous assignments and recommended norms for small group. We prioritized interactive teaching modalities and active learning to maximize engagement such as small-group learning among as well as our “Virtual Escape Room” (Appendix A) and simulation. Our small group, case-based discussions used a flipped classroom model, an effective and recommended modality for virtual instruction.16-22 All small-group facilitators were reminded of the best practices for online, small group teaching,23 which included use of introductions, learner-directed questioning to encourage equal participation, and “nudging” – reminders for learners to actively participate (See Appendix B).
Each virtual clerkship session was held on weekdays over two weeks for a total of 10 instruction days. The students were expected to complete various asynchronous learning assignments (estimated two hours daily) and attend four hours of Zoom (Zoom Video Communications, Inc, San Jose, CA) sessions daily. (See Appendix C for example schedule and specific content.) Cases from “Foundations of EM,” a national free, open-access online resource, were used to teach medical knowledge.24 Social medicine instruction was done using modules from the International and Domestic Health Equity and Leadership (IDHEAL) Section from the University of California, Los Angeles.25 Chosen modules included Language, Incarceration, Gender Identity, Race, and Homelessness, and assigned readings from those modules were delivered to learners via email. The Virtual Escape Room consisted of a tricyclic antidepressant overdose case, created by authors AV and TJ (Appendix A) with inspiration from another published escape room.26 The virtual simulation was carried out over Zoom, and cases from our traditional clerkship were used, which cover pediatric anaphylaxis, motorcycle trauma, hypothermia, and abdominal aortic aneurysm. The “Communities of Practice Panel” consisted of a panel of faculty/attendings who practice in different EM environments (ie, tertiary referral center, county, community, Veterans Administration, and critical access community). Many of these modalities have previously been highlighted as effective teaching modalities.27,28 Within the professional identity formation theme, learners read Carol Dweck’s “Mindset” to prepare for a book club-type discussion – a modality previously well-received by other learners.29 Finally, students were introduced to basic pedagogical techniques and practiced non-medical teaching sessions for their peers and faculty; feedback was provided.
Asynchronous assignments consisted of Emergency Medicine Reviews and Perspective C3 podcasts, free open-access medicine (FOAM) curated by the Academic Life in EM (ALiEM) AIR series, Foundations of EM “Frameworks,” and articles introducing topics of social EM.24,25,30,31 Asynchronous content was designed to correspond to daily synchronous content; specifics can be found on Appendix C. We made our virtual clerkship available to all fourth-year medical students applying into EM via the Visiting Student Application Service (VSAS) website and offered it twice during the 2020–2021 academic year.33 Given our predicted teaching resources, we estimated an ideal class size of less than 25 students per session. Ultimately, 26 students enrolled (nine in the first session, 17 in the second session). Attendance at all sessions was mandatory.
We recruited a group of residents and faculty to teach for a total of 24 lecturers and 26 small-group facilitators across both sessions. Facilitating the clerkship during the two-week session required one of three clerkship directors to be present on Zoom four hours per day, in addition to administrative tasks related to that day’s activities.
We assessed medical knowledge with a 30-item, peer-reviewed, multiple-choice test consisting of questions donated by RoshReview (Rosh Review LLC, Huntington Woods, MI), a commercial question bank company. RoshReview validates questions against real-world exam performance such as the in-training exam for EM residents.32 Questions were chosen by the course directors by a systems-based approach (ie, neurology, cardiovascular, etc) with the goal of choosing questions that were reflective of clerkship’s curriculum. Students did not have the ability to see the answers to the questions after taking the pre-test. Because the test was conducted at home, students theoretically could access open-access content in real time. Students completed the same medical knowledge test on the first and last days of the clerkship.34 We calculated mean scores and compared pre-and post-tests using a paired t-test, analyzed with the software statistical package Stata 16.1 (StataCorp, College Station, TX). Test scores had no bearing on final grades.
Each two-week clerkship contained five social EM sessions, which were assessed by an anonymous survey exploring previous experience with the topic and comfort with applying content learned to the clinical environment (Appendix D). This tool had been previously used in residency education by the IDHEAL group. It was developed by content experts after literature review to maximize content validity. We analyzed questions relevant to this clerkship. Participants were sent an online anonymous survey after each of the five sessions. In total, we sent a total of 130 surveys (five surveys per participant) and analyzed the data descriptively.
We assessed students’ overall attitude toward the course with a 16-item evaluative survey consisting of 11 multiple-choice, one slider scale, and four free-response items. The survey was created by SV, who has a Master’s in Education, had advanced training in survey design and experience in qualitative research, and was the course director, all providing content validity evidence. We read survey items aloud among the author group and piloted the survey with a small reference population to optimize response process validity. The survey was distributed on the last day of class (Appendix E). We calculated and reported descriptive statistics for survey questions with discrete answer choices. For free-response data within the survey, two authors (SV and AV) performed a thematic analysis. SV trained AV, a senior resident, to perform a thematic qualitative analysis. The analysts independently reviewed the data and later met to establish a final coding scheme, which they then independently applied to all data. After applying the final coding scheme, they identified discrepancies and finalized themes. The simple percent agreement between the two analysts was 80.3%. Discrepancies were resolved via in-depth discussion and negotiated consensus.
This study was deemed exempt by the University of California, Los Angeles Internal Review Board (IRB #20-002014) approved on November 19, 2020.
IMPACT/ EFFECTIVENESS
Twenty-six students participated in the virtual clerkship representing 22 medical schools and all regions of the US. All students completed the medical knowledge pre- and post-test. Mean test scores improved from 21.5 (standard deviation [SD] +/−2.6) to 22.9 (SD +/− 1.24) (P = 0.006), effect size 0.68, 95% confidence interval, 0.12–1.24.
Of the 130 IDHEAL post-module surveys administered, 98 (75%) were completed. Of the modules chosen, incarceration was least likely to have been previously covered with only 6% (1/18) of respondents having prior instruction. Eighty-nine percent (87/98) of respondents “strongly agreed” that these topics were important for patient care in the ED, and 66% (65/98) felt more confident after completing the modules. See Table 2 for full results.
Table 2International and Domestic Health Equity and Leadership sessions survey.
| Question/statement | Yes | No | |||
|---|---|---|---|---|---|
| Have you ever had formal instruction on social determinants of health? (n = 98) | 81 (83%) | 17 (17%) | |||
| Have you ever had formal instruction on social determinants of health during an emergency medicine rotation or departmental education conference? (n = 98) | 42 (43%) | 56 (57%) | |||
| Have you ever had formal instruction on topic discussed | |||||
| Topic (n = 98) | Yes | No | |||
| Language (n = 23) | 8 (35%) | 15 (65%) | |||
| Incarceration (n = 18) | 1 (6%) | 17 (94%) | |||
| Gender (n = 16) | 9 (56%) | 7 (44%) | |||
| Homelessness (n = 15) | 6 (40%) | 9 (60%) | |||
| Race (n = 16) | 11 (69%) | 5 (31%) | |||
| Unknown (n = 10) | 7 (70%) | 3 (30%) | |||
| Please rate your agreement with the following statement | Strongly agree | Agree | Neutral | Disagree | Strongly disagree |
| I learned about how this topic affects the health of my patients. | 79 (80%) | 18 (18%) | 0 (0%) | 0 (0%) | 1 (1%) |
| I feel more confident about how to address this topic when seeing patients in the ED. | 65 (66%) | 28 (29%) | 4 (4%) | 0 (0%) | 1 (1%) |
| This topic is important for the care of patients in the ED. | 87 (89%) | 10 (10%) | 0 (0%) | 0 (0%) | 1 (1%) |
ED, emergency department.
Almost all (25/26, 96%) students completed the end-of-rotation evaluative survey. Of all respondents, 95% “strongly agreed” or “agreed” that the rotation should be repeated in the future, and all “strongly agreed” or “agreed” that the rotation would impact the way they ranked our program. Major themes from the qualitative analysis are described in Table 3.
Table 3Themes identified from the end-of-rotation survey.
| Domain | Themes | Exemplar quotes |
|---|---|---|
| Favorites | ||
| Course design | Interactive education | Respondent 11: “Some of my favorite sessions were the teaching sessions, the escape room, and the simulation”Respondent 12: “board style cases from foundations”Respondent 23: “Enhancing my medical knowledge/clinical skills by participating in numerous simulations and mock oral board-style, small group exercises” |
| Topic variety | Respondent 9: “Incredible mix of content and social EM”Respondent 10: “Having this diversity of sessions made it easier to engage fully for the two hours that each topic was covered every day. It would’ve been tough to stay focused for 4 hours of medical knowledge didactics or 4 hours of social EM/career topics, so mixing it up was great”Respondent 20: “I like the mix of medical content with professional development and social issues.” | |
| Social determinants of health focus | Respondent 6: “The social EM aspect of this course was incredibly powerful and important”Respondent 11: “Social discussions are incredibly valuable and vital to the work providers (particularly in EM) do. The medical knowledge will always be hammered into us whether it’s in medical school or residency, but the social determinants of health are truly vital in understanding the populations we serve.”Respondent 12: “Learning about social aspects of EM from experts and listening to their experiences and perspectives” | |
| Professional identity formation | Networking with other students, residents, faculty | Respondent 10: “it was a great way to get to know more about my peers and build a bond.”Respondent 11: “I loved meeting other students from across the country.”Respondent 12: “I absolutely loved how many attendings, residents, and fellows I was able to meet. If I came there in person. I do not think I would have met even an eighth as many.” |
| Exposure to future career opportunities | Respondent 4: “it was great to get to know people who specialize in different areas and options for fellowship”Respondent 18: “information about the program, fellowships and culture” | |
| Least favorite/barriers | ||
| Technology related | Screen time | Respondent 1: “long zoom hours”Respondent 7: “Towards the end of the last week I was feeling the zoom fatigue” |
| Technical issues | Respondent 3: “Zoom challenges can be rough audio and freezing”Respondent 5: “only issue were the brief problems with Wi-Fi connectivity” | |
| Course design | Too much content for time allotted | Respondent 14: “I wish we had a bit more time on the foundations cases or had a follow up 10-15 minute review of the topics”Respondent 15: “I wish there was a little more small group time!” |
| Instructor orientation | Respondent 4: “Whenever you split people into small groups, ALL proctors should implement the round-robin approach for participation AND the proctor should tell the students when their turn is over. Most proctors did this, and I really appreciate it. When it didn’t happen, the sessions felt less fluid” | |
| Alignment of asynchronous and synchronous content | Respondent 9: “Some of the asynchronous resources were not too connected to the sessions that day.” |
EM, emergency medicine.
Prior literature has demonstrated knowledge acquisition and retention from virtual curricula, and we saw similar results, albeit our study demonstrated only a modest improvement.35,36 One explanation for the lack of larger change is that our assessment items may not have been perfectly aligned with our curriculum as the questions were pulled from a standard question bank. For future versions, we would strongly consider constructing and validating our own internal assessment of medical knowledge to be better aligned with our objectives. Additionally, students rated themselves as more confident in discussing and managing social medicine topics. Ideally, we would be able to conduct a repeat assessment at a predetermined timepoint to assess whether the social EM content had modified their practice as residents. While we emphasized our program’s strong social EM vision, other programs may replicate this curriculum to focus on their own strengths. In the past, visiting clerkships have acted as a recruiting tool for residencies,8 and virtual clerkships may also allow residencies to highlight strengths and successfully recruit.
Students from 22 institutions participated in our clerkship at minimal cost to them (only the cost to apply via VSAS). In contrast, EM applicants averaged 1.9 visiting rotations costing almost $1000 per rotation in 2019.37-39 While many institutions have implemented scholarships for under-represented in medicine students, virtual clerkships remove financial barriers for all students and may be an invaluable option for students with familial or other obligations.40 Virtual clerkships represent an additional strategy to help mitigate the socioeconomic barriers of visiting rotations.
While students perceived our virtual experience to be valuable, several challenges were encountered. The clerkship required significant administrative efforts and a large number of facilitators to create the intimate small-group experiences critical to its success. There was no protected time or funding for the instructors. Overall, at least 40 hours per clerkship among NW and SV were required, which did not account for planning and time from all instructors who volunteered their time. These requirements may be adjusted by limiting the number of students enrolled. Furthermore, as we look to the future, simultaneously administrating a virtual clerkship and in-person clerkship will likely require significant additional administrative support.
While this rotation ultimately served 25 students, we considered this rotation to be a success as we were fortunate to match three interns of our current class from the Virtual Clerkship. If we were to repeat this clerkship again, we would expand our evaluation efforts as it was limited, mainly only allowing for assessment in small groups. One other addition would be to encourage asynchronous communication among students and faculty. Examples of communication would be continued improvement of the environment and incorporation of daily questions expanding on the day’s content to further enhance spaced repetition. Lastly, interest in our virtual clerkship was likely increased due to COVID-19 restrictions on in-person opportunities. Future versions will require more advertisement and may not bolster as much interest.
We envision we will offer both versions of each clerkship separately moving forward. However, we likely would not offer a formal SLOE to students who pursue the Virtual Clerkship given we cannot comment on their clinical skills in the virtual format. However, we would gladly write a letter of recommendation as, in some ways, program leadership may get to know these students in a more personal way, especially with certain aspects of the SLOE such as “commitment to EM.” Finally, we may incorporate some of the social EM content and other teaching modalities into the traditional clerkship.
CONCLUSION
This virtual clerkship was created in response to an acute educational need created by the COVID-19 pandemic. However, our experience suggests that virtual learning experiences may be valuable in the future as an adjunct to traditional in-person rotations. Virtual rotations provide flexibility allowing for the incorporation of topics not traditionally taught (eg, social EM), allow residencies and students increased access to one another, and may eliminate socioeconomic barriers advancing educational equity.
Footnotes
Section Editor: Douglas Ander, MD and Andrew Phillips, MD, MEd
Full text available through open access at http://escholarship.org/uc/uciem_westjem
Address for Correspondence: Stephen Villa, MD, David Geffen School of Medicine, Department of Emergency Medicine, 10833 Le Conte Avenue, Los Angeles, CA 90095. Email: sevilla@mednet.ucla.edu. 1 / 2022; 23:33 – 39
Submission history: Revision received July 14, 2021; Submitted September 30, 2021; Accepted November 4, 2021
Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.
REFERENCES
1. Kaul V, Gallo De Moraes A, Khateeb D, et al. Medical education during the COVID-19 pandemic. Chest. 2020.
2. Important guidance for medical students on clinical rotations during the coronavirus (COVID-19) outbreak. 2020. Available at: https://www-aamc-org.libproxy2.usc.edu/news-insights/press-releases/important-guidance-medical-students-clinical-rotations-during-coronavirus-covid-19-outbreak. Accessed December 23, 2020.
3. Final report and recommendations for medical education institutions of LCME-accredited, U.S. osteopathic, and non-U.S. medical school applicants. 2020. Available at: https://www-aamc-org.libproxy2.usc.edu/system/files/2020-05/covid19_Final_Recommendations_05112020.pdf. Accessed December 23, 2020.
4. Katirji L, Smith L, Pelletier-Bui A, et al. Addressing challenges in obtaining emergency medicine away rotations and Standardized Letters of Evaluation due to COVID-19 pandemic. West J Emerg Med. 2020;21(3):538-41.
5. Aiyer AA, Granger CJ, McCormick KL, et al. The impact of COVID-19 on the orthopaedic surgery residency application process. J Am Acad Orthop Surg. 2020;28(15):e633-e641.
6. Negaard M, Assimacopoulos E, Harland K, et al. Emergency medicine residency selection criteria: an update and comparison. AEM Educ Train. 2018;2(2):146-53.
7. Mueller PS, McConahey LL, Orvidas LJ, et al. Visiting medical student elective and clerkship programs: a survey of US and Puerto Rico allopathic medical schools. BMC Med Educ. 2010;10:41.
8. Haas MRC, He S, Sternberg K, et al. Reimagining residency selection: Part 1-A Practical Guide to Recruitment in the Post-COVID-19 Era. J Grad Med Educ. 2020;12(5):539-44.
9. Rolak S, Keefe AM, Davidson EL, et al. Impacts and challenges of United States medical students during the COVID-19 pandemic. World J Clin Cases. 2020;8(15):3136-41.
10. Wilcha R. Effectiveness of virtual medical teaching during the COVID-19 crisis: systematic review. JMIR Med Educ. 2020;6(2):e20963.
11. Moreno-Walton L, Jung J, Aintablian H, et al. Consensus statement on the 2021–2022 emergency medicine application cycle: Part 1: Emergency Medicine Rotations and Letters. 2021. Available at: https://www.emra.org/be-involved/be-an-advocate/working-for-you/match-cycle-2022/. Accessed July 12, 2021.
12. Manthey DE, Ander DS, Gordon DC, et al. Emergency medicine clerkship curriculum: an update and revision. Acad Emerg Med. 2010;17(6):638-43.
13. Thomas PA, Kern DE, Hughes MT, et al. Curriculum Development for Medical Education: A Six-Step Approach. 2016.
14. Kern DE. Curriculum Development for Medical Education: A Six Step Approach. 1998.
15. Brown A, Lawrence J, Basson M, et al. A conceptual framework to enhance student online learning and engagement in higher education. Higher Education Research & Development. 2019.
16. Kraut AS, Omron R, Caretta-Weyer H, et al. The flipped classroom: a critical appraisal. West J Emerg Med. 2019;20(3):527-36.
17. Liebert CA, Lin DT, Mazer LM, et al. Effectiveness of the surgery core clerkship flipped classroom: a prospective cohort trial. Am J Surg. 2016;211(2):451-7.
18. O’Connor EE, Fried J, McNulty N, et al. Flipping radiology education right side up. Acad Radiol. 2016;23(7):810-22.
19. Rose E, Claudius I, Tabatabai R, et al. The flipped classroom in emergency medicine using online videos with interpolated questions. J Emerg Med. 2016;51(3):284-91.
20. Chen KS, Monrouxe L, Lu YH, et al. Academic outcomes of flipped classroom learning: a meta-analysis. Med Educ. 2018;52(9):910-24.
21. Yiu SHM, Spacek AM, Pageau PG, et al. Dissecting the contemporary clerkship: theory-based educational trial of videos versus lectures in medical student education. AEM Educ Train. 2019;4(1):10-7.
22. Young TP, Bailey CJ, Guptill M, et al. The flipped classroom: a modality for mixed asynchronous and synchronous learning in a residency program. West J Emerg Med. 2014;15(7):938-44.
23. Boettcher J, Conrad R. The Online Teaching Survival Guide: Simple and Practical Pedagogical Tips. 2016.
24. 2021. Available at: Foundationsem.com. Accessed February 26, 2021.
25. International and Domestic Health Equity and Leadership. 2021. Available at: http://www.idheal-ucla.org/page-12/. Accessed February 26, 2021.
26. Cates A. Idea Series: Toxicology virtual escape room during COVID-19. 2020. Available at: https://www.aliem.com/idea-series-toxicology-virtual-escape-room-during-covid-19/. Accessed October 18, 2021.
27. Gottlieb M, Landry A, Egan DJ, et al. Rethinking residency conferences in the era of COVID-19. AEM Educ Train. 2020;4(3):313-7.
28. He S, Lai D, Mott S, et al. Remote e-work and distance learning for academic medicine: best practices and opportunities for the future. J Grad Med Educ. 2020;12(3):256-63.
29. Jordan J, Bavolek R, Dyne P, et al. A virtual book club for professional development in emergency medicine. West J Emerg Med. 2020;22(1):108-14.
30. EM:RAP C3 podcasts. Available at: https://www.emrap.org/c3. Accessed February 26, 2021.
31. ALiEM AIR series. Available at: https://www.aliem.com/aliem-approved-instructional-resources-air-series/. Accessed February 26, 2021.
32. Micheal S. Using Rosh Review to improve education outcomes (Part 1). 2017. Available at: https://www.roshreview.com/blog/unlocking-the-data-behind-rosh-review-to-improve-educational-outcomes-part-1/. Accessed October 18, 2021.
33. Visiting student learning opportunities. Available at: https://students-residents.aamc.org/attending-medical-school/article/visiting-student-learning-opportunities/. Accessed February 26, 2021.
34. ROSH review. 2021. Available at: https://www.roshreview.com. Accessed February 26, 2021.
35. Dost S, Hossain A, Shehab M, et al. Perceptions of medical students towards online teaching during the COVID-19 pandemic: a national cross-sectional survey of 2721 UK medical students. BMJ Open. 2020;10(11):e042378.
36. Kronenfeld J, Ryon E, Kronenfeld D. Medical student education during COVID-19: Electronic education does not decrease examination scores. Am Surg. 2020:1-7.
37. Away Rotations of U.S Medical School Graduates by Intended Specialty, 2019 AAMC Medical School Graduation Questionnaire (GQ). 2019. Available at: https://students-residents.aamc.org/media/9501/download. Accessed February 26, 2021.
38. Winterton M, Ahn J, Bernstein J. The prevalence and cost of medical student visiting rotations. BMC Med Educ. 2016;16(1):291.
39. Blackshaw AM, Watson SC, Bush JS. The cost and burden of the residency match in emergency medicine. West J Emerg Med. 2017;18(1):169-73.
40. Boatright D, Branzetti J, Duong D, et al. Racial and ethnic diversity in academic emergency medicine: How far have we come? Next steps for the future. AEM Educ Train. 2018;2(Suppl Suppl 1):S31-S39.