Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Temporary transvenous cardiac pacing (TVP), a potentially life-saving intervention for critically-ill patients with unstable bradycardia,¹ is a vital component of the emergency physician’s skillset but is performed infrequently in clinical practice.² Although the Accreditation Council for Graduate Medical Education (ACGME) includes cardiac pacing in its list of key index procedures that are essential for the independent practice of emergency medicine (EM),³ residency programs cannot guarantee every trainee an opportunity to perform TVP during patient care. Unsurprisingly, limited exposure to TVP during training raises concerns about procedural competence,^4,5 and EM graduates feel underprepared to perform TVP.⁶ Checklist-based assessments conducted in a simulation setting reveal poor baseline TVP performance,⁷ with both EM residents and board-certified emergency physicians correctly performing, on average, fewer than half of the necessary steps of this critical procedure.^8,9

This reported lack of confidence and objective poor performance suggest the need for more effective models of TVP training during EM residency. Simulation provides an opportunity for learners to demonstrate procedural competence in a controlled setting.¹⁰ Mastery learning is a form of competency-based education in which all learners meet fixed achievement standards after individualized and variable amounts of educational time.¹¹ Simulation-based mastery learning (SBML) is a rigorous educational paradigm that employs an integrated bundle of seven features¹² (Table 1) and produces superior learning outcomes compared to non-mastery instruction.^13,14 SBML is a proven method for assessing and improving trainee performance for a variety of invasive procedures in the EM scope of practice, including lumbar puncture,¹⁵ central venous catheter insertion,¹⁶ and emergency department (ED) thoracotomy.¹⁷ SBML seems well-suited to improving TVP training and performance; however, to our knowledge no mastery learning curriculum exists for this procedure.

Our goal in this report was to describe the design and implementation of an SBML curriculum for TVP. Specific objectives included first establishing a minimum passing standard (MPS) for this procedure by consensus of an expert panel using the Mastery Angoff approach.¹⁸ Additional objectives included determining whether this curriculum led to achievement of the MPS by all learners and comparing baseline and post-training checklist scores for EM residents performing TVP in a simulated environment. We also present program evaluation data.

Population Health Research Capsule

What do we already know about this issue?

Transvenous pacing (TVP) is a potentially life-saving intervention, but opportunities for emergency medicine (EM) residents to develop competence with TVP are limited.

What was the research question?

Can a simulation-based mastery learning curriculum improve EM residents’ ability to perform TVP?

What was the major finding of the study?

Mean checklist score improved from 13.4/30 to 27.5 during initial post-testing. All learners reached mastery.

How does this improve population health?

After completing this curriculum, all learners demonstrated the ability to perform this rare but potentially life-saving procedure.

METHODS

Study Design and Setting

Like previously described SBML interventions,¹⁷ this prospective, cohort study used a pretest-post-test design following the mastery learning model (Figure 1).¹⁹ We conducted this study at a single, urban, academic medical center with a four-year EM residency during the 2019–2020 academic year. The institutional review board reviewed this study and determined it to be exempt.

All learners completed baseline diagnostic testing during a five-week period in October–November 2019. Those who did not demonstrate mastery during baseline testing by meeting or exceeding a predetermined MPS were required to complete an educational intervention, which took place between December 2019–February 2020. The intervention consisted of viewing a procedure video followed by deliberate practice with individual feedback. After the intervention, learners were again assessed using the checklist from February–April 2020; those who did not meet or exceed the MPS at initial post-testing completed additional deliberate practice and additional post-testing in May–June 2020 until all learners met or exceeded the MPS. Learners evaluated the curriculum after initial post-testing.

Selection of Participants

All postgraduate year (PGY) 1–4 EM residents at our institution were eligible for inclusion in this study. One resident was excluded due to participation in the study design and assessment.

Intervention

We created a series of learning objectives with increasing complexity for this curriculum. These included describing the indications and contraindications for TVP, listing the necessary equipment, demonstrating each step of the procedure, and then performing TVP to a mastery standard on a high-fidelity task trainer as assessed by a checklist. The first component of the educational intervention was an 11-minute video that we scripted and which was then filmed by professional videographers. The video begins with a review of indications, contraindications, relevant anatomy, and necessary equipment for TVP. Next, the video provides a narrated demonstration of each step of the procedure as detailed in the checklist. To encourage attention to the overall procedural skill and avoid rote memorization of the checklist itself, individual checklist items are not identified or displayed. The video was stored on a password-protected website and was shared only after all learners completed baseline testing. Learners were required to view the video prior to their deliberate practice session, and there was no limit on the number of times the video could be reviewed.

The second component of the educational intervention consisted of an individual, 30-minute session for each learner, designed to follow principles of deliberate practice (ie, an interactive discussion based on directly observed performance, using a “pause and discuss” model as events unfold).²⁰ These sessions were all led by a single instructor working with one learner at a time and followed a structured format to ensure consistency in session content. Each session began with a review of the checklist items most commonly not performed or performed incorrectly by all learners. Checklist items that were not performed or performed incorrectly by the individual learner were then reviewed, and correct performance was demonstrated using the simulator. Learners were able to practice these items as many times as necessary until correct performance was demonstrated. Finally, each learner practiced the entire procedure from beginning to end while receiving directly observed coaching and feedback.

Measurements

Learners were assessed using a 30-item, dichotomous (“correct” vs “incorrect/not done”) checklist created for this curriculum with input from emergency physicians, interventional cardiologists, and electrophysiologists. The checklist prompted learners to perform TVP in a patient with unstable bradycardia in whom an introducer sheath had already been placed. (During checklist development, the expert panel decided that placing the introducer sheath was a separate procedure.) A detailed description of the checklist’s design, content, and characteristics was previously reported, along with data demonstrating strong interrater reliability for checklist scores and the ability for the checklist to distinguish learners at different levels of training.⁸

A panel of 12 board-certified physicians (two cardiologists and 10 emergency physicians with experience performing and teaching TVP) established an MPS for the checklist using the Mastery Angoff approach.¹⁸ Panelists were recruited from our professional network and represent a diversity of geographic practice areas. None of the panelists participated in other aspects of the curriculum, including checklist design. One author facilitated the standard setting by leading a video conference call with the panelists that began by describing the purpose of the curriculum, the intended learners, and the concept of mastery learning. The checklist was then reviewed in detail.

Consistent with the Mastery Angoff approach, panelists were asked to estimate the percentage of well-prepared learners (defined as an individual who could safely and successfully perform the procedure in clinical practice with minimal or no supervision) who would correctly perform each checklist item after completing the curriculum. Panelists finalized their estimates after a group discussion and submitted their responses electronically. Each panelist’s individual MPS was determined by averaging their individual checklist item percentages. We then averaged these individual standards to determine the overall MPS for the checklist.

All learners were assessed by the same rater who stood adjacent to the task trainer and scored learners’ performance using a web-based version of the checklist (Qualtrics LLC, Provo, UT). The rater was blinded to the MPS until completion of the study. For compound items (for example, item 10: “Attaches the cap to the introducer sheath and turns the cap to lock into place on the sheath”), the item was scored as correct only if the learner performed all actions described in the item. To ensure all learners had the opportunity to demonstrate item 30 (“Retracts pacing wire only when balloon is deflated throughout procedure”), the TVP software was programmed so that successful capture could not be obtained on the first attempt. The same method was used for baseline assessment and post-testing. Learners did not have access to the checklist at any point during the curriculum.

We collected course evaluation data using an electronic survey (Qualtrics) following initial post-testing. Survey questions focused on whether learners felt the curriculum was a beneficial addition to their residency training, their enjoyment of the session, and whether learners felt confident performing TVP in the ED. Responses were scored on a Likert scale 1–5 (1 = strongly disagree; 5 = strongly agree).

Outcomes

The primary outcome measure was the mean number of correctly performed checklist items at each phase of the curriculum (baseline testing, initial post-testing, additional post-testing). Secondary outcome measures included program evaluation data obtained from a post-curriculum survey.

Analysis

Individual learner performance was determined by calculating the total number of correctly performed checklist items. Mean scores for all learners are reported with 95% confidence intervals (CI). We used the Wilcoxon signed-rank test to evaluate the difference in scores between baseline and initial post-testing. For program evaluation data, we calculated mean Likert scores with standard deviations. All analysis was performed using Stata 16.1 (StataCorp, College Station, TX).

RESULTS

Characteristics of Study Subjects

There were 59 residents eligible for inclusion in this study; two PGY-4 residents were unable to participate in the study due to scheduling conflicts. We collected data for 57 participants, including 15 PGY-1s, 15 PGY-2s, 15 PGY-3s and 12 PGY-4s. There were 38 (66.7%) male and 19 (33.3%) female residents enrolled.

Main Results

The expert panel set an MPS of correctly performing 28 (93.3%) checklist items. Performance data is displayed in Figure 2. No learners met the MPS during baseline testing. The mean score at baseline testing was 13.4 (95% CI 11.8–15.0). The mean score at initial post-testing was 27.5 (95% CI 26.9–28.1). The difference between the mean score at baseline testing and initial post-testing was statistically significant (P < 0.01).

Of the 57 residents enrolled, 36 (63.2%) achieved the MPS during initial post-testing. The mean score for the 21 residents (eight PGY-1s, six PGY-2s, five PGY-3s, and two PGY-4s) who did not meet or exceed the MPS during initial post-testing was 25.2 (95% CI 24.4–26.0). After completing one additional 30-minute, deliberate practice session, each of these 21 residents met or exceeded the MPS during additional post-testing. Checklist items that were most commonly not performed or performed incorrectly during baseline testing included the following: confirming the connection between the pacing wire and 2-millimeter adapters; confirming mechanical capture; securing the pacing generator after the procedure; and locking the introducer sheath cap O-ring. Complete data for checklist performance is presented in Table 2.

Course evaluations were completed by 51 (89%) of the participants. Learners indicated that the curriculum was a beneficial (mean 5.0, SD 0.2) and enjoyable (mean 4.9, SD 0.3) addition to their training, and they felt confident (mean 4.6, SD 0.6) performing TVP in the ED after completing the course.

DISCUSSION

This report describes the design and implementation of a SBML curriculum for training EM residents to perform TVP. After viewing a procedure video and completing individualized amounts of deliberate practice, all learners met or exceeded the MPS set for this procedure. These findings add to existing literature that shows SBML to be a highly effective form of competency-based medical education, particularly for infrequently performed invasive procedures.^15-17 The rigorous MPS set for this curriculum – 93.3% – is similar to the high standards set for other mastery learning interventions.^18,21

While mastery, defined as meeting the MPS, is ultimately a dichotomous variable, analyzing performance by checklist score allowed for a more granular understanding of a learner’s skill before and after undergoing the educational intervention. The poor baseline performance in this cohort – a mean score of 13.4 on a 30-item checklist – is not surprising for such an uncommon intervention and is consistent with the poor baseline performance observed in prior mastery learning studies of other procedures.^15-17 Previous TVP training in our program did not follow a competency-based model, and a majority of our residents had not performed TVP on a patient prior to the beginning of this curriculum; we suspect this limited experience is similar to that of trainees at other EM programs.

All learners in this report demonstrated the ability to meet or exceed the MPS after completing this curriculum, which included rigorous assessment, an educational intervention (watching the procedure video), and individualized amounts of deliberate practice with directly observed coaching and feedback, which are essential features of mastery learning. Although the mean checklist score improved significantly from baseline testing to initial post-testing, for more than one third of the learners, a single deliberate practice session did not result in meeting or exceeding the MPS. This is not unexpected. In mastery learning there is no “failure,” only the opportunity for additional practice and testing until all learners achieve the desired standard.

The mean checklist score for the learners who required additional practice was still notably improved from initial post-testing, with a narrower range of performance. Following one additional 30-minute, deliberate practice session, these learners all met or exceeded the MPS during additional post-testing. Despite the additional time commitment required to complete this curriculum, learners found it to be an enjoyable and beneficial addition to their training and felt confident in their ability to perform TVP after completing the course.

Procedural competence is critical to the practice of EM^22,23; however residency programs cannot guarantee trainees exposure to all procedures in the clinical environment prior to graduation.^24,25 This is particularly true for TVP, which, due to its rarity, is one of only three procedures for EM allowed by the ACGME to be exclusively performed through simulation prior to residency graduation.³ As growing numbers of learners in all specialties compete for access to a finite number of invasive procedures in teaching hospitals, clinical experience alone may no longer be sufficient to ensure procedural competence.²⁶ Notably, the 21 learners in this study who did not meet the MPS during initial post-testing included PGY-1 through PGY-4 residents, which suggests that skill level and need for additional practice cannot be predicted solely by level of training. Rigorous, competency-based training and assessment methods like mastery learning can – and should – be used to prepare EM residents to successfully and independently perform uncommon but potentially lifesaving interventions.

LIMITATIONS

There are several limitations to this report. A relatively small number of learners completed the curriculum due to the single-site design; a larger sample size could improve the generalizability of these findings. Prior to the creation of this SBML intervention, our residents completed TVP training during an annual “rare procedures” simulation session that did not follow a competency-based model and which residents attended as their clinical schedule allowed; we acknowledge that learners with alternate methods of prior training may demonstrate different levels of performance in this curriculum.

We were unable to track how many times the procedure video was viewed; however, all learners verbally attested to watching the video prior to their deliberate practice session. Finally, this curriculum required a significant investment of time and resources: learners participated in a minimum of three sessions (baseline testing, deliberate practice, and initial post-testing), each of which required direct observation and active involvement of a faculty member. Specialized equipment and support from simulation staff was also required. The resource-intensive nature of mastery learning may limit its feasibility in other settings.

Given the small sample size, we did not compare SBML to other methods of TVP training. However, mastery learning has been consistently shown to outperform non-mastery instruction.^13-15,27 In addition, we did not control for learners who had previous experience performing TVP during clinical practice. However, no learners in our cohort demonstrated mastery during baseline testing, which indicates that any residents with prior clinical TVP experience were not adequately prepared to meet or exceed the MPS set for this procedure. This is consistent with a previous analysis of other invasive procedures performed by internal medicine and EM residents (including central venous catheter insertion, lumbar puncture, and thoracentesis), which found that clinical procedure experience during residency was not sufficient to ensure competence.²⁸

Future research should investigate the optimal timing for repeat assessment and practice to ensure TVP skill retention. Assessment of other specialists who perform TVP, such as cardiologists and intensivists, should be explored. While challenging to assess, the effect of this curriculum on patient-level outcomes (ie, Kirkpatrick Level 3) should also be investigated. Finally, the deliberate practice and assessment components of this curriculum were conducted using a high-fidelity simulator; adapting the curriculum to incorporate low-fidelity models could also be explored.²⁹

CONCLUSION

This study demonstrated a significant improvement in emergency medicine residents’ ability to perform temporary transvenous cardiac pacing on a high-fidelity task trainer after completing a mastery learning curriculum and contributes to the growing body of literature that proves simulation-based mastery learning to be a highly effective method of competency-based medical education.

Footnotes

Section Editor: Chris Merritt, MD, MPH, MHPE

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Matthew R. Klein, MD, MPH, Northwestern University Feinberg School of Medicine, Department of Emergency Medicine, 211 E. Ontario Street, Suite 200, Chicago, IL 60611. Email: matthew.klein@northwestern.edu. 1 / 2023; 24:43 – 49

Submission history: Revision received June 16, 2022; Submitted October 11, 2022; Accepted October 12, 2022

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Harrigan R, Chan T, Moonblatt S, et al. Temporary transvenous pacemaker placement in the emergency department. J Emerg Med. 2007;32(1):105-11.

2. Sodek G, Domanovits H, Meron G, et al. Compromising bradycardia: management in the emergency department. Resuscitation. 2007;73(1):96-102.

3. Emergency Medicine Defined Key Index Procedure Minimums. 2017 Accessed November 1, 2020.

4. Murphy JJ. Problems with temporary cardiac pacing. Expecting trainees in medicine to perform transvenous pacing is no longer acceptable. BMJ. 2001;323(7312):527.

5. Sharma S, Sandler B, Cristopoulos C, et al. Temporary transvenous pacing: an endangered skill. Emerg Med J. 2012;29(11):926-7.

6. Druck J, Valley M, Lowenstein S. Procedural skills training during emergency medicine residency: Are we teaching the right things?. West J Emerg Med. 2009;10(3):152-6.

7. Ahn J, Kharasch M, Aronwald R, et al. Assessing the Accreditation Council for Graduate Medical Education requirement for temporary cardiac pacing procedural competency through simulation. Simul Healthc. 2013;8(2):78-83.

8. Klein M, Schmitz Z, Adler M, et al. Development of a rigorously designed checklist for assessment of emergency medicine resident performance of temporary transvenous cardiac pacing. AEM Educ Train. 2021;5(3):e10566.

9. Branzetti J, Adedipe A, Gittinger M, et al. Randomised controlled trial to assess the efficacy of just-in-time training on procedural performance: a proof-of-concept study to address procedural skill decay. BMJ Qual Saf. 2017;26(11):881-91.

10. Sawyer T, White M, Zaveri P, et al. Learn, see, practice, prove, do, maintain: an evidence-based pedagogical framework for procedural skill training in medicine. Acad Med. 2015;90(8):1025-33.

11. McGaghie W. Mastery learning: It is time for medical education to join the 21 century. Acad Med. 2015;90(11):1438-41.

12. McGaghie WC, Harris IB. Learning theory foundations of simulation-based mastery learning. Simul Healthc. 2018;13(3S Suppl 1):S15-S20.

13. Cook D, Brydges R, Zendejas B, et al. Mastery learning for health professionals using technology-enhanced simulation: a systematic review and meta-analysis. Acad Med. 2013;88(8):1178-86.

14. McGaghie W, Issenberg S, Cohen E, et al. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011;86(6):706-11.

15. Barusk J, Cohen E, Caprio T, et al. Simulation-based education with mastery learning improves residents’ lumbar puncture skills. Neurology. 2012;79(2):132-7.

16. Barsuk J, McGaghie W, Cohen E, et al. Use of simulation-based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit. J Hosp Med. 2009;4(7):397-403.

17. Miller D, Zaidi H, Sista P, et al. Creation and implementation of a mastery learning curriculum for emergency department thoracotomy. West J Emerg Med. 2020;21(5):1258-65.

18. Barsuk J, Cohen E, Wayne D, et al. A comparison of approaches for mastery learning standard setting. Acad Med. 2018;93(7):1079-84.

19. McGaghie W, Barsuk J, Wayne D. AM Last Page: Mastery learning with deliberate practice in medical education. Acad Med. 2015;90(11):1575.

20. Eppich W, Hunt E, Duval-Arnould J, et al. Structuring feedback and debriefing to achieve mastery learning goals. Acad Med. 2015;90(11):1501-8.

21. Prenner S, McGaghie W, Chuzi S, et al. Effect of trainee performance data on standard-setting judgements using the Mastery Angoff method. J Grad Med Educ. 2018;10(3):301-5.

22. Beeson M, Ankel F, Bhat R, et al. The 2019 Model of the Clinical Practice of Emergency Medicine. J Emerg Med. 2020:S0736-4679.

23. Chapman D. Definitively defining the specialty of emergency medicine: issues of procedural competency. Acad Emerg Med. 1999;6(7):678-81.

24. Petrosoniak A, Herold J, Woolfrey K. Emergency medicine procedural skills: What are residents missing?. CJEM. 2013;15(4):241-8.

25. Hayden S, Panacek E. Procedural competency in emergency medicine: the current range of resident experience. Acad Emerg Med. 1999;6(7):728-35.

26. Gisondi M, Regan L, Branzetti J, et al. More learners, finite resources, and the challenging landscape of procedural training at the bedside. Acad Med. 2018;93(5):699-704.

27. Schroedl C, Frogameni A, Barsuk J, et al. Implementation of simulation-based mastery learning on resident skill managing mechanical ventilators. ATS Sch. 2020;2(1):34-48.

28. Barsuk J, Cohen E, Feinglass J, et al. Residents’ procedural experience does not ensure competence: a research synthesis. J Grad Med Educ. 2017;9(2):201-8.

29. Young T, Tango J, Toomasian C, et al. An inexpensive conceptual training model for transvenous pacemaker placement. West J Emerg Med. 2019;21(1):180-3.

Share on Facebook

Tweet

Share via email