Author	Affiliation
Nur-Ain Nadir, MD, MEHP	Kaiser Permanente Central Valley, Department of Emergency Medicine, Modesto, California; University of Illinois-Peoria, Jump Simulation, Peoria, Illinois
Danielle Hart, MD, MACM	University of Minnesota, Hennepin County Medical Center, Department of Emergency Medicine, Minneapolis, Minnesota
Michael Cassara, DO, MSEd, CHSE	Donald and Barbara Zucker School of Medicine at Hofstra-Northwell, Department of Emergency Medicine, Manhasset, New York
Joan Noelker, MD, MACM	Washington University in St. Louis, Department of Medicine, Division of Emergency Medicine, St. Louis, Missouri
Tiffany Moadel, MD	Donald and Barbara Zucker School of Medicine at Hofstra-Northwell, Department of Emergency Medicine, Manhasset, New York
Miriam Kulkarni, MD	St. John’s Riverside Hospital, Department of Emergency Medicine, Yonkers, New York
Christopher S. Sampson, MD	University of Missouri-Columbia, Department of Emergency Medicine, Columbia, Missouri
Suzanne Bentley, MD, MPH	Icahn School of Medicine at Mount Sinai, Elmhurst Hospital Center, Department of Emergency Medicine and Medical Education, Simulation Center, Elmhurst, New York
Neel K. Naik, MD	New York Presbyterian, Weill Cornell Medicine, Department of Emergency Medicine, New York, New York
Jessica Hernandez, MD, MEH	University of Texas Southwestern Medical Center, Department of Emergency Medicine, Dallas, Texas
Sara M. Krzyzaniak, MD	Kaiser Permanente Central Valley, Department of Emergency Medicine, Modesto, California; University of Illinois-Peoria, Jump Simulation, Peoria, Illinois
Steven Lai, MD	Ronald Reagan UCLA Medical Center, Olive View-UCLA Medical Center, Department of Emergency Medicine, Los Angeles, California
Gregory Podolej, MD	Kaiser Permanente Central Valley, Department of Emergency Medicine, Modesto, California; University of Illinois-Peoria, Jump Simulation, Peoria, Illinois
Christopher Strother, MD	Icahn School of Medicine at Mount Sinai, Department of Emergency Medicine, Pediatrics, and Medical Education, New York, New York

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

With the Accreditation Council of Graduate Medical Education’s (ACGME) adoption of a competency-based (CB) educational framework, and the majority of emergency medicine (EM) residencies reporting at least one resident on probationary status, remediation has become a pressing topic in EM resident education.^1,2 Some residency training programs struggle with the paradox between the foundational premise of CB training (i.e., a time-independent path to competence for all learners) and the ACGME’s prescribed length of residency training (three or four years for EM residency training).^3,4 The fact that not all learners achieve competence at the same time or rates² further compounds the matter, necessitating remediation plans for learners falling outside the competency bell curve.^5,6

“Remediation” can be used to describe the status of a resident within a program (such as “probation”) or the “effort spent to improve a resident’s knowledge, skills, or attitudes.”⁷ In this project, remediation is defined as any additional training, instruction, or practice provided to residents found to be deficient in one of the six core competencies in EM.^5,7 Note that remediation is not necessarily equivalent to probation, which implies a formal notation on a resident’s academic file; remediation may occur informally without an annotation or the resident being formally under probation.⁵

The last two decades have seen an increase in the use of simulation pedagogies, such as simulation-based mastery learning (SBML) in EM resident education,^8-11 and there are reports of SBML successfully being used for procedural education.^9,12 SBML lends itself particularly to the CB approach in that it is time-independent, allowing learners to achieve mastery over time.^9,10,13 There are also anecdotal reports of success with other kinds of simulation models in EM. Simulation modalities such as high-fidelity patient simulators (mannequins), standardized patients, partial task trainers, computer screen-based simulation, virtual reality environments, and tabletop role-playing exercises such as oral board exam-style simulations have been used to create opportunities and safe environments for clinical training¹⁴ and anecdotally for remediation. While over 90% of EM residency programs use some form of simulation,^11,15 how exactly it is being used and the principles guiding its use vary widely. Current recommendations, in general, support the incorporation of simulation into curricula for instruction, identification of knowledge gaps, evaluation and remediation.^8,16-24

The successful use of simulation-based remediation (SBR) in other specialties and fields such as anesthesia, internal medicine, and nursing have been described, but the concept in general is under-reported.^{17,18,20,25-27} Evidence supporting the use of SBR within EM training is somewhat contradictory. In 2007 a Society of Academic Emergency Medicine (SAEM) task force on simulation research in EM cautioned against the use of SBR, contending that the term “remediation” could not be reliably applied, given differences in faculty perception of resident performance.²⁸ In 2012, the ACGME published 23 EM sub-competencies to be used in the assessment of EM residents. They suggest using simulation as one method to evaluate sub-competencies 1–11 and 16–23.²⁹ Some authorities have posited that since simulation could be used for specific sub-competency assessment, it could also be used for remediation within those same sub-competencies.^17,18 In 2016, the Council Of Residency Directors (CORD)-EM Remediation Task Force (RTF) recommended simulation for remediating multiple competencies, including patient-centered communication, teamwork and leadership.^16,30

 

Population Health Research Capsule

What do we already know about this issue?

With the Accreditation Council of Graduate Medical Education’s transition towards a competency-based framework, simulation-based remediation (SBR) has become a pressing topic; however, few guidelines exist to direct its use.

What was the research question?

Authors sought to develop consensus on the appropriate use of SBR in emergency medicine residency training programs.

What was the major finding of the study?

SBR can be used for remediating specific competencies provided there is process and outcome transparency.

How does this improve population health?

SBR can assist in remediating learners so as to produce clinically competent physicians, thereby promoting patient safety and quality of care.

 

Interestingly, many of the recommendations on the use of SBR arose from experts in EM residency leadership and remediation. Simulationists were possibly under-represented among the stakeholders making the aforementioned recommendations. This is relevant because some simulationists view the experience as formative and eschew the use of simulation for remediation purposes, arguing that remediation implies summative assessment, which intrinsically threatens the principle of “learner safety” integral to simulation-based education.^31-34 There are also concerns that simulation for high-stakes assessment requires consensus on case design standards and setting of minimum performance levels to ensure that the testing is valid.³⁵

With the exception of SBML and procedural remediation,⁹ there remains no clear consensus on when and how to appropriately use simulation for remediation in EM for other sub-competencies. The lack of specific recommendations and guidelines makes SBR planning difficult for both residency program and simulation leadership. To answer this need, the CORD-EM Simulation Community of Practice (COP), CORD-EM RTF and the SAEM Simulation Academy formed a joint collaboration, the Simulation Based Remediation Collaborative (SBRC), to clarify the role of SBR. This study was based on their work, and its objective was to build consensus on the appropriate use of SBR in EM.

METHODS

This study was deemed exempt by the local institutional review board. Using previously described methodology, we employed a modified Delphi approach to derive a set of consensus statements on using simulation for remediation in EM.^36-40 The study design is depicted in Figure 1. We conducted a literature search on simulation remediation practices using terms “simulation,” “remediation,” and “simulation based remediation,” and identified commonly occurring themes. Using these themes, we created an open-ended questionnaire. From May through June 2017, the CORD-EM listserv was queried for all programs with experience in SBR. The questionnaire was subsequently sent to the 18 programs that indicated experience in SBR. Responses to the questionnaire were assessed using thematic analysis⁴¹ by two EM simulation remediation experts (NN, GP) and an EM remediation expert (SK). We used the commonly occurring themes to create an initial survey, which was then piloted among a group of six EM simulation and remediation experts (NN, CS, MK, DH, JN, TM) and further refined based on their input.

We circulated the final primary survey to 52 experts with experience using simulation for remediation purposes, who had been identified a priori through their involvement in the CORD-EM Simulation COP, the RTF, and SAEM Simulation Academy and any publications or presentations on SBR. Experts were requested to rank each statement according to the following categories: “agree,” “modify,” or “disagree;” and survey program parameters were set to completion of all survey items. The study group analyzed the results of the primary survey, and Randolph’s free marginal kappa was calculated to gauge agreement for each statement. Randolph’s free kappa is a chance-adjusted measure of agreement for any number of cases, categories, or raters⁴² and has been used to measure agreement in studies with large numbers of raters (experts).^39,40

A free marginal kappa ≥ 0.6 was used to indicate good agreement.^42,43 We removed items without consensus (free marginal kappa < 0.4). Items with moderate agreement (free marginal kappa 0.4–0.59) were reworked into a second survey. We also analyzed narrative comments from the initial survey and comments pertaining to “modifying” statements through thematic analysis, and any newly emerging themes were incorporated into the second survey. The second survey was distributed to the same panel of experts who had responded to the initial survey. On the secondary survey we used a cut off of 80% to indicate strong agreement and we deemed 70% moderate agreement with respect to consensus.

RESULTS

41 of 52 invited individuals completed Round 1 of the survey for a response rate of 78%, and 31 out of the initial 41 participants completed Round 2 for a response rate of 76%. Sixty-six percent of participants identified themselves primarily as simulation faculty, 32% identified themselves primarily as program directors (PDs) or assistant/associate program directors (APDs), and 2% identified themselves primarily as core faculty. Four of those identifying themselves as PDs/APDs also had simulation training. Experts represented 30 programs from across North America (Table 1). The modified Delphi process yielded 38 statements with strong agreement, eight with moderate agreement and nine with no agreement within six themes: 1) role of simulation in remediation; 2) decision to use simulation in remediation; 3) SBR process; 4) debriefing SBR; 5) assessing and reporting SBR; and 6) defining and determining SBR success. The modified Delphi process yielded 11 statements with strong agreement, one with moderate agreement and five with no agreement within the theme “deficiencies best addressed by SBR.” The modified Delphi process yielded 10 statements with strong agreement, two with moderate agreement and 11 with no agreement for “sub-competencies best addressed by SBR” (Table 2). Consensus in the alignment of simulation modalities to competency being remediated was also achieved (Figure 2).

DISCUSSION

The results of our study show that there is strong agreement in many areas regarding SBR, including the belief that simulation can play a role in remediation. SBR should be part of a multifaceted remediation plan and not the sole remediation strategy. The residency leadership and the remediation faculty committee (or equivalent) should still be responsible for the overall remediation plan, with specific goals for the SBR components. These goals should be transparent to the learner and the faculty conducting the SBR. The methods used to assess the learner’s performance should be transparent and communicated to all stakeholders: the learner; the residency leadership; the remediation team; the clinical competency committee (CCC); and all other faculty involved in summative decisions regarding advancement. Although formative assessment is ideal, summative assessment may be employed, provided the process is clearly defined and transparent.

SBR may be used “diagnostically” and “therapeutically” to benefit the remediating learner. Diagnostic SBR provides a protected, standardized, and contextualized environment in which a learner’s performance gaps may be more precisely studied. In contrast to remediation where the struggling learner is situated within the clinical environment (under direct observation), the classroom, or an equivalent didactic setting, diagnostic SBR provides the conditions under which faculty and learners may safely and accurately explore the learners’ frames responsible for observed deviations from ideal performance. We posit that faculty are more likely to accurately identify the true reasons for performance gaps in the laboratory environment where SBR occurs (than in the clinical environment). While most faculty are likely able to directly observe their learners while contemporaneously working alongside them and identifying performance gaps, they are likely unable to learn why these gaps exist. In the challenging clinical milieu of today’s academic emergency department, where cognitively-loaded faculty and learners must balance the demands, expectations, and temporal pressures of patient care, there is no time, space, and privacy to support the reflection necessary for uncovering causes for performance gaps, which frequently tend to be multifactorial.^6,44 As many residents struggle in multiple domains,^19,21 diagnostic SBR may provide the best opportunity to identify one or more domains requiring attention. Diagnostic SBR, with a low resident-to-faculty ratio, may provide the best data to inform the development of an individualized remediation plan.

Modalities chosen for therapeutic SBR should be aligned with the learner’s needs. There is strong agreement that SBR is appropriate for areas such as application of medical knowledge, clinical reasoning, decision-making, communication, teamwork, leadership, crisis resource management (CRM), and cognitive overload/multitasking in high acuity situations. There is moderate agreement that SBR is not the best modality for developing foundational medical knowledge, as this may be best acquired through other means.

SBR seems most appropriate for the following sub-competencies: emergency stabilization, performance of a history and physical exam, diagnosis, pharmacotherapy, multitasking, and the procedural and communication milestones. Outside of CRM, simulation may not be the best modality for improving sub-competencies linked to the general competencies of systems-based practice (SBP) or problem-based learning and improvement (PBLI). While some aspects of PBLI could potentially be addressed in the debriefing portion of SBR (e.g., improving a learner’s insight through self-reflection following SBR, informing the development of an individualized development plan), this seems to be a small component of a larger PBLI remediation plan. While SBR for SBP may provide some opportunities for learners to practice mobilizing institutional or system resources to optimize patient care, the other aspects of this competency requiring remediation may necessitate the use of other strategies. Professionalism represents another domain more effectively addressed through means other than SBR. The Hawthorne effect could bias the assessment of a learner’s performance in a SBR conducted for professionalism concerns. While some learners may have difficulty with professionalism competencies in any circumstance, others may only display professionalism lapses when they are overly stressed, busy, frustrated, or not being directly observed.

The optimal number of SBR sessions required is difficult to define at the outset of a remediation plan and is dependent upon the focus of remediation and the learner’s progress. SBR focused on one domain requiring improvement has the potential to unmask another, which may necessitate a different simulation or non-simulation-based intervention. Learner improvement during each SBR session, therefore, informs the next steps to be taken. SBR for procedures incorporates the mastery learning approach,^9,10,45,46 where the learner deliberately practices a procedure under facilitation until it is completed safely and competently in the simulated environment.⁴⁷ For non-procedural SBR, learners should experience multiple simulation sessions of comparable cases with similar learning objectives in contrast to repeating the same exact simulation case (a practice that did not achieve agreement in our study) until those objectives are met.

Ideally, SBR would be conducted by faculty with formal simulation training or experience. However, only moderate agreement was obtained for the item “SBR may be conducted by the PD/APD (those ultimately involved in making progression decisions), as long as they have training in simulation.” One possible explanation for this moderate level of agreement is that residency leadership’s (PD/APD) direct involvement in SBR may be perceived as a threat to the principle of “learner safety.” While it may be optimal to have non-residency leadership faculty conduct SBR, we recognize the feasibility of this approach is dependent on the resources at the program; in some programs, the simulationist is part of the residency leadership. Programs should use the resources they have to optimize the learning and outcomes of SBR for their trainees.

With respect to assessment, our results, based on the final round of the Delphi panel, support the use of assessment tools with some validity evidence, similar to the work described by Blum et al.²⁵ While procedural assessment tools with validity evidence exist,^48,49 there are few simulation cases with validity evidence beyond content validity (i.e., internal structure, response process, relations to other variables, or consequential validity).^50,51 Various assessment tools for non-technical skills also have been found to have some validity evidence, but without a co-existing recommended simulation case for EM resident learners. If assessment tools with validity evidence are available for the domain requiring remediation, these tools should be used to promote best practice. However, to maintain the validity of judgments made by using such an assessment tool, the tool must be applied to a similar population of learners, implemented under similar conditions, using similarly trained raters.

Tools with demonstrated validity evidence in one circumstance do not automatically demonstrate the same characteristics when applied to other circumstances. Unless validity evidence is re-demonstrated in the new contexts, there may not be current validity evidence for the tools chosen. Therefore, in agreement with previously published works by Blum et al.,²⁵ we also recommend that summative (“high-stakes”) SBR should not be used as the solitary measure of a learner’s attainment of competency in a given domain. Summative SBR should be used as part of a comprehensive remediation plan providing multiple data points to be evaluated when assessing a resident’s progression. A single checklist or global rating scale should not be the only measure defining SBR success. The ultimate success of any remediation plan should be improvement in the learner’s performance in the clinical environment.

Given the challenges of residency length constraints and learner variability in achieving competence, SBR can provide extra time and opportunities for struggling learners to train contemporaneously to routine simulations in order to achieve mastery within the CB framework. Although similar to non-remediation simulations in principle, what differentiates SBR from the former is the absolute need for confidentiality for ensuring a psychologically safe learning environment, low resident to faculty ratios, and the need for absolute transparency between the learner and the program leadership regarding the process (goals, objectives, results) of SBR. Also included in the latter is clear delineation of assessment methods and how their results will be used, especially as pertaining to high-stakes decisions such as “promotion” or “probationary” status. Unlike routine simulations, where to preserve psychological safety and safe-space principles, learner performance is not discussed outside of the debriefing room,^32,33 in SBR learner performance is frequently discussed with residency leadership; the learners should be informed of this significant difference a priori.

LIMITATIONS

The main limitations of this study are that it only represents EM residency programs from the United States. Caution should be used in applying these results to nursing and medical students and to other specialties and geographical locations. Although we met the stated guidelines for the size of the Delphi panel,⁵² the panel may have missed experienced simulationists. Our expert selection was dependent on available publications and presentations on SBR, of which there is a significant dearth. Although some of our experts have published or presented on this topic, most were identified through their actual experience in SBR, which in itself may not make them “experts” per se. Given the obvious lack of data on the subject, this approach seems reasonable. Additionally, although our survey instruments were developed using an iterative process, the length of the initial instrument could have contributed to survey fatigue and potential bias. Finally, a lack of face-to-face discussions to resolve disagreements may have limited some of our findings.

CONCLUSION

This Delphi-based study, based on input from 30 ACGME-accredited EM programs across the United States, found agreement on many aspects of SBR. Simulation can be used diagnostically as well as therapeutically in remediation processes. Once a deficit is identified, simulation can be a helpful remediation tool for certain competencies and milestones, but simulation is not a one-size-fits-all approach that can be applied to every EM skill or competency. Simulation is best suited for remediation of procedural, patient care and communication milestones and less suited for remediation of systems-based practice and problem-based learning milestones. SBR can be one aspect, but should not be the sole component of a remediation plan. Similarly, SBR performance should only be one component of how remediation success is assessed by program leadership and the CCC, using tools with validity evidence when possible. These SBR assessments should be transparent between the simulation faculty, the learner, the program leadership and the CCC.

Footnotes

Section Editor: Jeffrey N. Love, MD, MSc, MHPE

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Nur-Ain Nadir, MD, MEHP, Kaiser Permanente Central Valley, Department of Emergency Medicine, 4601 Dale Road, Modesto, CA 95356. Email: nurainnadir81@gmail.com. 1 / 2019; 20:145 – 156

Submission history: Revision received July 13, 2018; Submitted October 19, 2018; Accepted October 26, 2018

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Silverberg M, Weizberg M, Murano T, et al. What is the prevalence and success of remediation of emergency medicine residents?. West J Emerg Med. 2015;16(6):839-44.

2. Carraccio C, Englander R, Van Melle E, et al. Advancing competency-based medical education: A charter for clinician-educators. Acad Med. 2016;91(5):645-649.

3. ACGME Program Requirements for Graduate Medical Education in Emergency Medicine. 2017:13-21. Available at: https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/110_emergency_medicine_2017-07-01.pdf. Accessed October 10, 2018.

4. Riebschleger MP, Haftel HM. Remediation in the context of the competencies: a survey of pediatrics residency program directors. J Grad Med Educ. 2013;5(1):60-3.

5. Smith J, Lypson M, Silverberg M, et al. Defining uniform processes for remediation, probation and termination in residency training. West J Emerg Med. 2017;18(1):110-3.

6. Guerrasio J. Remediation of the Struggling Medical Learner. 2013.

7. Weizberg M, Smith JL, Murano T, et al. What does remediation and probation status mean? a survey of emergency medicine residency program directors. Acad Emerg Med. 2015;22(1):113-6.

8. Ten Eyck RP. Simulation in emergency medicine training. Pediatr Emerg Care. 2011;27(4):333-44.

9. Griswold-Theodorson S, Ponnuru S, Dong C, et al. Beyond the simulation laboratory: A realist synthesis review of clinical outcomes of simulation-based mastery learning. Acad Med. 2015;90(11):1553-60.

10. Mcgaghie WC, Issenberg SB, Barsuk JH, et al. A critical review of simulation-based mastery learning with translational outcomes. Med Educ. 2014;48(4):375-85.

11. Okuda Y, Bond W, Bonfante G, et al. National growth in simulation training within emergency medicine residency programs, 2003–2008. Acad Emerg Med. 2008;15(11):1113-6.

12. Wayne DB, Barsuk JH, O’Leary KJ, et al. Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice. J Hosp Med. 2008;3(1):48-54.

13. Riley B. Using simulation-based medical education to meet the competency requirements for the single accreditation system. J Am Osteopath Assoc. 2015;115(8):504.

14. Chiniara G, Cole G, Brisbin K, et al. Simulation in healthcare: a taxonomy and a conceptual framework for instructional design and media selection. Med Teach. 2013;35(8):e1380-95.

15. McLaughlin S, Fitch MT, Goyal DG, et al. Simulation in graduate medical education 2008: a review for emergency medicine. Acad Emerg Med. 2008;15(11):1117-29.

16. Williamson K, Quattromani E, Aldeen A. The problem resident behavior guide: strategies for remediation. Intern Emerg Med. 2016;11(3):437-49.

17. Evans CJ, Harder N. A formative approach to student remediation. Nurse Educ. 2013;38(4):147-51.

18. Zeno B, Zigmont J, Boes T. Board 281 – Program Innovations Abstract How Do You Change a Resident in a Month? Using Simulation as Goal Directed Remediation of Resident Performance (Submission # 164) Board 282 – Program Innovations Abstract Simultaneous Simulation: Assessing Residents’ Ability to Manage and Triage Multiple Unstable Patients in a High Fidelity, High Stakes Environment. Simul Healthc. 2013;8(6):509.

19. Zbieranowski I, Takahashi SG, Verma S, et al. Remediation of residents in difficulty: a retrospective 10-year review of the experience of a postgraduate board of examiners. Acad Med. 2013;88(1):111-6.

20. Broadfoot K, Guerrasio J, Aagaard E. Beyond Procedures and Checklists : Using Simulation to Remediate Communication and Professionalism Skill Challenges. SGIM Forum. 2015;38(9):2-3.

21. Guerrasio J, Aagaard EM. Methods and Outcomes for the Remediation of Clinical Reasoning. J Gen Intern Med. 2014;29(12):1607-14.

22. Cleland J, Leggett H, Sandars J, et al. The remediation challenge: theoretical and methodological insights from a systematic review. Med Educ. 2013;47(3):242-51.

23. Hauer KE, Ciccone A, Henzel TR, et al. Remediation of the deficiences of physicians across the continuum from medical school to practice: A Thematic Review of the Literature. Acad Med. 2009;84(12):1822-32.

24. Steinert Y. The “problem” learner: Whose problem is it? AMEE Guide No. 76. Med Teach. 2013;35(4):1035-46.

25. Blum RH, Boulet JR, Cooper JB, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology. 2014;120(1):129-41.

26. Latini C. Bridging the clinical remediation gap – a concept mapping approach. Clin Simul Nurs. 2009;5(3):e144.

27. Camp S, Legge T. Simulation as a tool for clinical remediation: an integrative review. Clin Simul Nurs. 2018;16:48-61.

28. Bond W, Kuhn G, Binstadt E, et al. The use of simulation in the development of individual cognitive expertise in emergency medicine. Acad Emerg Med. 2008;15(11):1037-45.

29. The Emergency Medicine Milestone Project. 2012. Available at: http://www.acgme.org/Portals/0/PDFs/Milestones/EmergencyMedicineMilestones.pdf?ver=2015-11-06-120531-877. Accessed June 25, 2018.

30. Available at: http://cordrtf.wikidot.com. Accessed June 26, 2018.

31. Rudolph JW, Simon R, Raemer DB, et al. Debriefing as formative assessment: closing performance gaps in medical education. Acad Emerg Med. 2008;15(11):1010-6.

32. Wen LY, Gaba DM. In Response: Comment on “A survey of simulation utilization in anesthesiology residency programs in the United States.”. Anesth Analg. 2017;124(1):369.

33. Fanning RM, Gaba DM. The role of debriefing in simulation-based learning. Simul Healthc. 2007;2(2):115-25.

34. Hart D, Bond W, Siegelman JN, et al. Simulation for assessment of milestones in emergency medicine Residents. Acad Emerg Med. 2018;25(2):205-20.

35. Boulet JR, Murray D, Kras J, et al. Setting performance standards for mannequin-based acute-care scenarios: An examinee-centered approach. Simul Healthc. 2008;3(2):72-81.

36. Hsu C-C, Sandford BA. The Delphi technique. Dublin Natl Child Off. 2005;7(2):120-5.

37. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008-15.

38. Humphrey-Murto S, Varpio L, Gonsalves C, et al. Using consensus group methods such as Delphi and nominal group in medical education research. Med Teach. 2016;39(1):1-6.

39. Cumyn A, Harris IB. A comprehensive process of content validation of curriculum consensus guidelines for a medical specialty. Med Teach. 2012;34(8).

40. Frallicciardi A, Vora S, Bentley S, et al. Development of an emergency medicine simulation fellowship consensus curriculum: initiative of the Society for Academic Emergency Medicine Simulation Academy. Acad Emerg Med. 2016;23(9):1054-60.

41. Pope C, van Royen P, Baker R. Qualitative methods in research on healthcare quality. Qual Saf Health Care. 2002;11(2):148-152.

42. Randolph JJ. Free-marginal multirater kappa: an alternative to Fleiss’ fixed-marginal multirater kappa. Present Joensuu Univ Learn Inst Symp. 2005.

43. Warrens MJ. Inequalities between multi-rater kappas. Adv Data Anal Classif. 2010;4(4):271-86.

44. Katz ED, Dahms R, Sadosty AT, et al. Guiding principles for resident remediation: recommendations of the CORD remediation task force. Acad Emerg Med. 2010;17(Suppl 2):S95-103.

45. McGaghie WC. Mastery learning: It is time for medical education to join the 21st century. Acad Med. 2015;90(11):1438-41.

46. McGaghie WC, Issenberg SB, Cohen ER, et al. Medical education featuring mastery learning with deliberate practice can lead to better health for individuals and populations. Acad Med. 2011;86(11):8-9.

47. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;79(Supplement):S70-S81.

48. Hart D, Clinton J, Anders S, et al. Validation of an assessment tool for field endotracheal intubation. Mil Med. 2016;181(11):e1484-e1490.

49. McEvoy MD, Nietert PJ, Field LC, et al. NIH Public Access. Simul Heal. 2012;7(4):222-35.

50. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: Theory and application. Am J Med. 2006;119(2):166.e7-16.

51. Standards for Educational & Psychological Testing. 2014.

52. de Villiers MR, de Villiers PJT, Kent AP. The Delphi technique in health sciences education research. Med Teach. 2005;27(7):639-43.