Background
Objectives
Curricular design
Impact/effectiveness
Supplementary information

BACKGROUND

High-quality medical care of patients in the emergency department (ED) is dependent on teams of qualified healthcare providers. Emergency physicians have critical roles in the functioning of these teams. They must possess the medical knowledge and clinical skills needed to diagnose and treat high acuity medical conditions, but they must also possess the teamwork and interpersonal skills needed to successfully coordinate efficient team-based care in high-stakes situations. The skills that contribute to leadership, teamwork, communication, resource utilization and problem solving are frequently referred to as crisis resource management (CRM) skills and are increasingly recognized as factors that impact patient safety in acute healthcare fields such as emergency medicine (EM). ^1-5 Conversely, the lack of CRM skills has been implicated in adverse patient outcomes and malpractice cases. ⁵

In the current era of outcomes-based medical education, the American Council for Graduate Medical Education (ACGME) requires EM residency programs to ensure trainees meet core competencies. The core competencies entitled “Interpersonal and Communication Skills” and “Systems-based Practice” involve various CRM skills such as teamwork, communication and resource utilization. ⁶ While residency programs are expected to teach and evaluate these skills, the ACGME does not specify how this training should be incorporated into residency education, and as a result many curriculums lack a dedicated plan for teaching these less tangible skills.

Historically, residents have been expected to learn CRM skills through role modeling, mentorship or trial and error using an apprentice-style approach to medical training. ⁷ However, better understanding of CRM suggests that these skills are teachable and measurable through more explicit approaches, similar to the medical knowledge and procedural skills taught across all residency programs. ⁸ To effectively equip trainees with the complete skillset needed to be successful, dedicated curricula designed to introduce, teach and reinforce CRM skills are warranted in residency education.

Several standardized team-training programs, such as “MedTeams” and “TeamStepps,” have been developed and disseminated nationally.^5,9 The broad CRM skills in these courses are applicable to a wide variety of healthcare settings and disciplines. However, specific healthcare environments such as the ED and specific learners such as EM residents may warrant specialized CRM skills training.^10,11

OBJECTIVES

Our first objective was to design a CRM course encompassing basic CRM principles critical to the practice of EM that could be feasibly incorporated into any EM residency curriculum. Our second objective was to evaluate the CRM course’s efficacy in helping novice EM residents develop CRM skills. We conducted a pilot study to measure improvement in CRM skills displayed by teams of EM interns during the course.

CURRICULAR DESIGN

We conducted a literature review to explore existing courses that encompass CRM principles and determine which CRM skills should be included in this course. Despite discovering extensive lists of CRM skills pertinent to EM, ^10-12 our search revealed just one previous CRM curriculum specifically designed for EM residents. ¹³ While the course was well received by participants, the authors did not attempt to measure the course’s effectiveness in improving CRM skills.

Our CRM course consists of two key components: 1) an introductory lecture presentation, and 2) a series of six specialized simulation scenarios. The 30-minute lecture introduces the key concepts and history of CRM in aviation and in healthcare and highlights the increasing recognition of CRM’s role in patient safety. The presentation also describes the specific roles of the leader and ED team members, and defines basic CRM terms such as “closed loop communication,” “shared mental model,” “workload management,” and “situational awareness.” Establishing this baseline knowledge aids in the discussions of CRM during debriefs of simulation scenarios and in the ED.

The majority of the CRM training involves a series of six high-fidelity simulation scenarios followed by structured debriefs. Various studies suggest that simulation is an ideal educational modality to teach and evaluate CRM skills.^7,10,14,15 Simulation not only introduces the importance of CRM skills, but also allows residents to deliberately practice the CRM skills in a safe environment where no patients are at risk.

The simulation cases in this course provide high-acuity EM situations that require effective CRM skills. The cases are designed to mimic the intense time pressures, rapidly evolving situations and high-acuity illnesses that are routinely experienced in the ED.^10,16 We chose all cases from our department’s existing simulation case bank and modified them to meet specific learning objectives. The six cases used during this pilot course are described in the Supplemental Table. While each case has specific objectives for the medical management of each diagnosis, all cases incorporate the uniform CRM objectives listed in Table 1.

Each simulation scenario runs for 10–15 minutes followed by a 30-minute debrief by a single facilitator. The scenarios use either a standardized patient actor or the SimMan3G patient simulator, and cast simulation faculty as the nurse, family member or consultant. Following each scenario, a debrief is conducted by a facilitator and addresses both the medical management learning objectives, and the uniform CRM learning objectives within each scenario.

A randomized control study conducted in anesthesia CRM training showed that observers of a simulation scenario can gain the same improvement in CRM skills as active participants in the scenario. ¹⁷ With this in mind, our CRM course includes active participation in some scenarios as well as observation of other scenarios. More specifically, all teams participate separately in Case 1 and Case 6. Team performances in Case 1 provide pretest data, and team performances in Case 6 provide posttest data to measure the impact of the overall course. For Cases 2, 3, 4, and 5, only one team participates in the scenario while the remaining teams are observers. Observing teams are instructed to take notes on both the medical management and the CRM of the case. The facilitator then engages both the participating team and the observing teams in discussion during the debrief. The use of observational learning in four of the six cases minimizes the need for multiple simulation rooms, larger simulation and debriefing staff and time that would be required to allow every team to participate in every case separately.

Finally, to recreate the ad hoc nature of teams in the ED setting, this course randomly distributes residents into small teams at three separate times over the two-week period. Random and frequent shuffling of the teams recreates a more realistic working environment for our trainees and allows them to work with different peers in the various simulation cases.

IMPACT/EFFECTIVENESS

The pilot study of this course involved a cohort of 14 EM interns who participated in the simulation scenarios during four separate days in July 2016. This CRM course was a component of a larger two-week curriculum designed to provide an introduction to residency and to the clinical practice of EM. The institutional review board of our institution reviewed this study and determined that it was exempt from requiring informed consent of participants or ongoing review. The interns were randomly divided into four teams and these teams were redistributed three times over the course to replicate the ad hoc nature of ED teams. By the end of the course, each intern participated in a total of three cases and observed three additional cases.

During every simulation scenario, the CRM skills demonstrated by the participating team were evaluated by two independent raters using the Ottawa CRM Global Rating Scale (GRS). This CRM evaluation tool has been shown to have acceptable construct validity and interrater reliability^18,19 and has previously been used to measure the development of CRM skills in EM residents. ¹⁵ The Ottawa GRS provides a seven-point scale to evaluate the overall CRM performance and CRM skills in five specific categories: leadership, problem solving, situational awareness, resource utilization, and communication. Two trained EM faculty, both with extensive expertise in simulation education, acted as independent raters throughout the course. Both raters had used the Ottawa GRS in one prior study but did not receive any extensive training in the use of the tool. The same two raters were present for all simulation scenarios during the course.

Results of this pilot study suggest that this CRM course was effective in improving CRM skills among teams of novice interns; however, the improvements were not statistically significant. The median pretest and posttest scores and the data analysis for overall CRM scores and CRM category scores are presented in Table 2. We used Number Cruncher Statistical Software for the analyses. ²⁰ Given that the unit of analysis is teams of interns and our sample size was N=4, we established a significance criterion of α= 0.05. To control for Type I errors, we conducted Bonferroni corrections (corrected p-value = number of comparisons/α = 6/0.05 = 0.0083).²¹

A Wilcoxon-signed rank test showed that the posttest scores for overall CRM performance were not statistically significantly higher than the pretest scores when using the Bonferroni corrected p-values. (All p-values were greater than the corrected p = 0.0083.) Similar data analysis for each category of CRM skills measured by the Ottawa GRS also demonstrated an improvement from pretest to posttest but also failed to meet statistical significance.

We calculated effect sizes to further aid in the interpretation of the data. The |r| effect size index was interpreted similar to a correlation coefficient |r|, with |r| = 0.10, 0.30 and > 0.50 interpreted as small, medium and large effects, respectively.²² The improvements in overall CRM and each CRM category all showed large effect sizes ranging from |r| = 0.581 to 0.601. The large effect sizes detected for each dependent measure implies that the posttest scores would reflect a significant improvement over the pretest scores with an increased sample size.

We conducted an interrater reliability analysis using a kappa statistic with linear weighting to determine consistency between the two faculty raters. A single analysis of all pretest and posttest scores by the raters was used in the calculation of the kappa statistic. The interrater reliability for the raters was found to be kappa = 0.74, 95% CI (0.66, 0.82). This kappa value represents good agreement between the two raters in this study. ²³

While pretest and posttest data were analyzed to determine the effectiveness of the CRM training, we also evaluated the CRM skills during Cases 2, 3, 4, and 5 to explore the potential value of the observational learning in this course. These scores were not included in the data analysis since only a single team actively participated in each of these cases while the remaining three teams observed. As shown in the figure below, each team showed an overall gradual improvement in CRM skills compared to the preceding teams, suggesting that observational learning of CRM was effective in this setting.

The results support previous evidence that simulation is an effective educational modality to teach CRM skills and that observational learning in simulation is an effective tool to optimize training and minimize the time and resources required. Furthermore, this course can be easily incorporated into existing EM residency educational curriculums since it is designed to be feasible with one simulation room, a single simulation debriefing facilitator, and minimal other staff or faculty to serve as confederates in the scenarios. It is an option for any EM residency without access to a large simulation center.

While these results suggest that CRM skills can be significantly improved through a short simulation-based educational intervention, there are a number of limitations. This study is limited by the small sample size and the setting of a single EM residency program. The CRM principles introduced and evaluated in this course reflect just a subset of those CRM skills needed for EM. It is our belief that adding basic CRM to the vocabulary and skillset of an EM intern sets the stage for continued appreciation, deliberate practice, and ongoing improvement of CRM skills during training and throughout a career in EM.

This pilot study also does not assess the retention of the CRM skills beyond the two-week course. Additional testing over multiple years of residency training would be valuable in detecting improvement and retention of CRM skills over time. Ultimately, further studies would be beneficial to determine if the CRM skills achieved through simulated scenarios affect behaviors or success of residents in the ED.

Supplementary Information

Footnotes

Section Editor: David P. Way, MEd

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Jessica R. Parsons, MD, MSMS, Drexel University College of Medicine, 245 N. 15th Street, 2nd Floor, MS 1011, Philadelphia, PA, 19102. Email: Jessica.parsons@DrexelMed.edu. 1 / 2018; 19:205 – 210

Submission history: Revision received June 15, 2017; Submitted October 18, 2017; Accepted October 29, 2017

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Manser T. Teamwork and patient safety in dynamic domains of healthcare: a review of the literature. Acta Anaesthesiol Scand. 2009;53(2):143-51.

2. Ziesmann MT, Widder S, Park J, et al. S.T.A.R.T.T.: development of a national, multidisciplinary trauma crisis resource management curriculum-results from the pilot course. J Trauma Acute Care Surg. 2013;75(5):753-8.

3. Boet S, Bould MD, Fung L, et al. Transfer of learning and patient outcome in simulated crisis resource management: a systematic review. Can J Anaesth. 2014;61(6):571-82.

4. Capella J, Smith S, Philp A, et al. Teamwork training improves the clinical care of trauma patients. J Surg Educ. 2010;67(6):439-43.

5. Risser DT, Rice MM, Salisbury ML, et al. The potential for improved teamwork to reduce medical errors in the emergency department. The MedTeams Research Consortium. Ann Emerg Med. 1999;34(3):373-83.

6. Chapman DM, Hayden S, Sanders AB, et al. Integrating the Accreditation Council for Graduate Medical Education Core Compe tencies into the Model of the Clinical Practice of Emergency Medicine. Ann Emerg Med. 2004;43(6):756-69.

7. Bearman M, O’Brien R, Anthony A, et al. Learning surgical communication, leadership and teamwork through simulation. J Surg Educ. 2012;69(2):201-7.

8. Morey JC, Simon R, Jay GD, et al. Error reduction and performance improvement in the emergency department through formal teamwork training: evaluation results of the MedTeams project. Health Serv Res. 2002;37(6):1553-81.

9. Clancy CM, Tornberg DN. Teamstepps: assuring optimal teamwork in clinical settings. Am J Med Qual. 2007;22(3):214-7.

10. Flin R, Maran N. Identifying and training non-technical skills for teams in acute medicine. Qual Saf Health Care. 2004;13:i80-84.

11. Flowerdew L, Brown R, Vincent C, et al. Identifying nontechnical skills associated with safety in the emergency department: a scoping review of the literature. Ann Emerg Med. 2012;59(5):386-94.

12. Small SD, Wuerz RC, Simon R, et al. Demonstration of high-fidelity simulation team training for emergency medicine. Acad Emerg Med. 1999;6(4):312-23.

13. Reznek M, Smith-Coggins R, Howard S, et al. Emergency medicine crisis resource management (EMCRM): pilot study of a simulation-based crisis management course for emergency medicine. Acad Emerg Med. 2003;10(4):386-9.

14. Shapiro MJ, Morey JC, Small SD, et al. Simulation-based teamwork training for emergency department staff: does it improve clinical team performance when added to an existing didactic teamwork curriculum?. Qual Saf Health Care. 2004;13(6):417-21.

15. Clarke S, Horeczko T, Carlisle M, et al. Emergency medicine resident crisis resource management ability: a simulation-based longitudinal study. Med Educ Online. 2014;19:25771.

16. Fernandez R, Vozenilek JA, Hegarty CB, et al. Developing expert medical teams: toward an evidence-based approach. Acad Emerg Med. 2008;15(11):1025-36.

17. Lai A, Haligua A, Dylan Bould M, et al. Learning crisis resource management: Practicing versus an observational role in simulation training – a randomized controlled trial. Anaesth Crit Care Pain Med. 2016;35(4):275-81.

18. Kim J, Neilipovitz D, Cardinal P, et al. A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study. Crit Care Med. 2006;34(8):2167-74.

19. Kim J, Neilipovitz D, Cardinal P, et al. A comparison of global rating scale and checklist scores in the validation of an evaluation tool to assess performance in the resuscitation of critically ill patients during simulated emergencies (abbreviated as “CRM simulator study IB”). Simul Healthc. 2009;4(1):6-16.

20. Hintze J. NCSS 9. 2013.

21. Bland JM, Altman DJ. Multiple significance tests: The Bonferroni method. BMJ. 1995;310(21):170.

22. Pallant J. SPSS Survival Manual. 2007.

23. Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70(4):213-20.