Author	Affiliation
Max Berger, MD, MEHP	University of California Los Angeles, David Geffen School of Medicine, Department of Emergency Medicine, Los Angeles, California
Jack Buckanavage, MD	Mount Sinai School of Medicine, Department of Emergency Medicine, New York, New York
Jaime Jordan, MD, MAEd	University of California Los Angeles, David Geffen School of Medicine, Department of Emergency Medicine, Los Angeles, California
Steven Lai, MD	University of California Los Angeles, David Geffen School of Medicine, Department of Emergency Medicine, Los Angeles, California
Linda Regan, MD, MEd	Johns Hopkins University, Department of Emergency Medicine, Baltimore, Maryland

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Restrictions imposed on in-person education during the coronavirus 2019 (COVID-19) pandemic accelerated the need for virtual learning, including telesimulation.
<sup>1

,

2</sup>
Telesimulation has been defined as “a process by which telecommunication and simulation resources are utilized to provide education, training, and/or assessment to learners at an off-site location.”
³
Initial applications were in lower resource settings such as developing countries where learners did not have access to simulation centers or instructors.
<sup>4

,

5</sup>
Within telesimulation, different modalities have been described that vary in fidelity as well as location of the learners and instructors relative to each other and the simulation center.
<sup>6

–

8</sup>

Several published articles since early 2020 have described different institutions’ approaches to telesimulation since the pandemic.
<sup>1

–

2

,

9

–

13</sup>
Common themes include the need to modify learning objectives to virtual environments and to select the appropriate modality of telesimulation based on institutional needs and resources.
<sup>9

–

12</sup>
Different modalities of telesimulation have been described, including the following: 1) learners virtually observing and debriefing a live simulation
⁷
; 2) learners present with a manikin while instructors facilitate from a separate location
⁶
; 3) instructors present with a manikin while learners remotely participate
⁷
; and 4) completely remote option where learners and instructors both participate remotely from separate locations.
<sup>10

,

11</sup>

Limited data comparing telesimulation vs traditional simulation suggests that learner satisfaction with telesimulation or hybrid virtual and in-person simulation is similar, although this was not found in all studies.
<sup>7

,

13

,

14</sup>
A scoping review from 2021 highlighted the mixed data on student perception of telesimulation, with some of the included studies indicating remote facilitation of simulation being perceived as equally or more effective than live facilitation, while others found remote facilitation to be inferior.
¹⁴
Facilitator perception of telesimulation has not been well studied. Limited learning outcome data has suggested similar improvements between in-person simulation and telesimulation.
<sup>8

,

14</sup>

Our objective in this study was to describe the patterns of telesimulation usage in emergency medicine (EM) residency programs during COVID-19-induced, in-person learning restrictions as well as its anticipated utility moving forward. This information is crucial to understanding the value of telesimulation and its utility in medical education.

METHODS

Study Design, Setting, and Population

We conducted a cross-sectional survey study of faculty in charge of simulation for EM residency programs in the United States. We collected data from January–February 2022. After identifying EM residency programs and their websites from the EMRA Match database,
¹⁵
we searched each website for contact information for the director of simulation education. If there was no director designated, we emailed each residency’s program coordinator and/or director asking for contact information for the faculty in charge of residency simulation. Each program was allowed only one designated participant. This study was given exempt status by the University of California, Los Angeles Institutional Review Board (IRB#21-001336) and the Johns Hopkins University Homewood IRB (HIRB00013694).

Survey Development and Dissemination

Given the lack of any previously created survey applicable to this construct, the primary author (MB) developed a web-based survey tool with consideration of survey research best practices and Messick’s validity framework.
<sup>16

–

19</sup>
For content validity evidence, we first performed a literature review, and the author group of expert simulation educators and medical education researchers reviewed the survey for clarity and relevance to the construct. We defined telesimulation as including any simulation activity where “telecommunication and simulation resources are utilized to provide education, training, and/or assessment to learners at an off-site location.”
³
We piloted the survey with a group of simulation educators who were not included in the target sample to gather response process validity evidence. After piloting, we revised the survey for clarity. The final survey included multiple-choice items and one free-text response item (Appendix 1).

We invited participants by email and sent two targeted, follow-up invitations to non-responders at bi-weekly intervals. We administered the survey via Qualtrics (Qualtrics, LLC, Provo, UT). No individual identifying information was collected. To maximize response rate and minimize guessing, we did not require participants to complete all survey items. Participants were not compensated for participating in the study.

Data Analysis

We calculated and reported descriptive statistics for items with discrete answers. We conducted calculations using Qualtrics and Microsoft Excel for Mac (Microsoft Corp, Redmond, WA). We examined the answers to the free-text responses to identify common themes that would broaden the reader’s understanding of the data. Successive wave analysis was performed to assess the extent of possible nonresponse bias.
²⁰
We examined whether use of telesimulation during the pandemic, planned future use of telesimulation after in-person restrictions, and respondent program format (postgraduate years [ PGY] 1–3 vs 1–4) differed by wave. Bivariate chi-square tests for each variable of interest by wave were performed using Microsoft Excel for Mac, and P-values less than 0.05 were considered statistically significant. We used the consensus-based checklist for reporting of survey studies (CROSS) as reporting guidelines (Appendix 2).
²³

RESULTS

Of 139 simulation leaders we identified, 91 (65%) completed the survey with 87 (63%) completing all items. We report demographic data for survey respondents in Table 1, while respondents’ experience and perceptions of telesimulation are shown in Table 2. Prior to the COVID-19 pandemic, 9% (8/91) of survey respondents were using telesimulation in their curricula. There was a wide variety of prior experiences with telesimulation, with the most common being that they had heard of telesimulation but never been involved (44%). Ninety-two percent (84/91) of respondents reported that their institution prohibited in-person learning activities at some point during the COVID-19 pandemic. During in-person learning restrictions, 62% (56/91) used telesimulation in some form.

When survey respondents were asked about what format(s) of telesimulation were used, 11% (10/90) stated that they only used a completely virtual oral-boards style format, while the rest of those who used telesimulation reported using a hybrid or virtual format involving a patient monitor and/or manikin. The largest percentage of survey respondents felt that medical knowledge and communication/teamwork-focused cases were best suited for telesimulation (72% and 47% respectively), while most felt that procedure training was not well suited for telesimulation (62%). Thirty-eight percent (34/90) of respondents stated they planned to use telesimulation in some form in their curriculum moving forward, mostly for medical knowledge and communication/teamwork-focused cases (76% and 68%, respectively).

We received 14 free-text responses, and identified three major themes, described below with exemplar quotes. 1.

Telesimulation is a valuable alternative to in-person learning:“It has been the ‘better than nothing’ option but accepted by learners when other options are not feasible.”“It has exceeded expectations in how helpful it has been.”

2.

Telesimulation is an option for learners unable to participate in person:“We found that it’s a great option for residents with families or who have other extenuating circum-stances why they can’t participate in person, ie, breastfeeding moms, new parents, elder care, etc. Many of our residents who are between nights or between mid-shifts will log on and participate.”

3.

Telesimulation is challenging for procedural education:“Difficult to learn muscle memory for high acuity, low occurrence skills.”“Procedural training was the most difficult to simulate via telesim.”

 

For the wave analysis, the study included 91 respondents, including 42 in wave 1 (46%), 21 in wave 2 (23%), and 28 in wave 3 (31%). Results of the examined survey questions did not statistically differ by wave with all P-values > 0.05. (See Supplemental Table.)

DISCUSSION

Despite relatively low use of telesimulation within EM programs prior to the COVID-19 pandemic, we found that many EM residency programs (62%) quickly adapted to in-person learning restrictions by using telesimulation. While not all programs that trialed telesimulation plan to continue its use, 38% of respondent programs do plan to continue to use telesimulation, compared to 9% of programs using telesimulation prior to COVID-19. This represents a large increase in the overall usage of telesimulation within EM residencies. Our study also sheds light on how telesimulation can benefit EM programs. Being able to increase learner participation to include residents with family obligations or between night shifts could allow for increased return on investment for simulation resources and faculty time. Most respondents who plan to continue to use telesimulation reported that they will use it as a small percentage of their overall simulation curriculum, which highlights that telesimulation is not replacing in-person simulation but augmenting the traditional curriculum. This could be in a hybrid format that allows for increased participation, or as part of separate telesimulation days that could reduce the travel burdens on learners and instructors.

There was large variation in how programs conducted telesimulation during in-person restrictions. This is in line with prior literature and likely reflects individual program needs, preferences, and available resources.
<sup>1

,

2

,

8

–

11

,

13

,

22</sup>
Most described telesimulation as best suited for medical knowledge and communication/teamwork-focused cases, rather than for procedure teaching. This is interesting given that early descriptions of telesimulation in the literature mostly involved procedural teaching.
<sup>5

,

6</sup>
One possible explanation for this discrepancy is that those early studies involved duplicate simulators at remote locations, an expense that is likely not practical, or necessary, for a residency program given the ability to host procedure training as part of the in-person curriculum. While it is apparent that there are increased plans for the use of telesimulation compared to the pre-pandemic era, not all residency programs who used telesimulation during times of in-person restrictions are planning to continue to do so. The reasons for this are unknown but may relate to telesimulation resource availability or limited outcome data on its utility.

Based on our results, we believe that telesimulation can continue to be a valuable addition to the traditional in-person simulation curriculum, particularly in allowing for increased participation of learners and instructors, reducing resource costs such as simulation center and staff time, and allowing for a viable option to practice medical knowledge and communication-based competencies. Now that telesimulation has been established as an instructional strategy that will continue to be part of many EM residency curricula, it opens opportunities for future innovation and scholarship within simulation-based medical education. Additional investigation could compare different modalities of telesimulation on objective learning outcomes.
²³
It would also be interesting to explore the role of virtual and augmented reality within telesimulation.
<sup>24

,

25</sup>

LIMITATIONS

Despite multiple attempts, we were not able to obtain contact information for a simulation leader from all EM programs. However, the breakdown of PGY 1–3 vs PGY 1–4 programs of survey respondents (70% PGY 1–3 vs 30% PGY 1–4), approximating the actual distribution of the EM residency programs (81% PGY 1–3 vs 19% PGY1–4), suggests that the sample closely resembles that of the population.
⁸
Given our response rate of 65%, it is possible non-response bias affected our results, with participants with less interest or familiarity with telesimulation being less likely to respond. However, the results of our successive wave analysis failed to detect non-response bias for the selected survey questions.

There may be other influences affecting a program’s use of telesimulation that we were not able to capture, and in this survey study we examined only the opinions of faculty and not those of resident learners. Additionally, the literature-based definition of telesimulation we used may be overly broad and encompass more than what typical educators might consider telesimulation. Finally, we acknowledge that the survey was administered in 2022 with in-person learning restrictions just starting to be lifted, and how people are using telesimulation now may be changing. Future work could examine this evolving use of telesimulation within EM residency programs.

CONCLUSION

This study describes past and planned future use of telesimulation within EM residency programs. A large proportion of EM residencies trialed telesimulation during COVID-19-induced restrictions. Despite relatively low use of telesimulation prior to the pandemic, more EM programs plan to incorporate telesimulation moving forward as a limited portion of their overall simulation curriculum. Opportunities for further innovation and scholarship within this area of simulation education will be possible given this planned continued use.

Footnotes

Section Editors: Kendra Parekh, MD and Benjamin Holden Schnapp, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Max Berger, MD, MEHP, University of California Los Angeles, David Geffen School of Medicine, Department of Emergency Medicine, 1100 Glendon Ave., Suite 1200, Los Angeles, CA 90024. Email: MRBerger@mednet.ucla.edu
11 / 2024; 25:907 – 912

Submission history: Revision received June 14, 2024; Submitted September 20, 2024; Accepted September 26, 2024

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.