Background
Objectives
Curricular design
Impact/effectiveness
Discussion
Limitations
Conclusion

BACKGROUND

The practice of emergency medicine (EM) requires that physicians be proficient in a vast array of procedures no matter how frequently or rarely they occur. The Accreditation Council for Graduate Medical Education and the American Board of Emergency Medicine have established a series of milestones regarding procedural performance in order to establish a minimum level of physician competency in the field of EM.¹ Procedural skills inevitably degrade over time if not practiced regularly and, thus, maintenance of these skills requires continued practice in the clinical or simulated setting.²

A previously conducted needs assessment looked at current procedural skill practices by emergency physicians (EP) and found a significantly positive correlation between the frequency at which a skill was performed and the perceived confidence by physicians performing that particular skill. Additionally, they found that the vast majority of physicians would like to attend procedural training sessions if offered by their institution.³

In addition to a procedural skills curriculum, “just-in-time” (JIT) refresher training can be particularly useful for procedures that occur rarely in the emergency department (ED). A previous study indicated that JIT training led to improved trainee skills and confidence in performing procedures, from both the resident and supervisor perspective.⁴ A randomized controlled trial that evaluated transvenous pacemaker placement by EPs in a simulated setting found that a JIT intervention, which included both a refresher video and a step-by-step interactive checklist, significantly improved performance.⁵

OBJECTIVES

To our knowledge, there is no currently published literature on the use of JIT procedural training guides in EM. Our first objective was to perform a needs assessment of EPs’ current JIT procedural resource usage. Our second objective was to examine the impact of creating a repository of easily accessible JIT procedural training guides with institution-specific information. Specifically, we sought to evaluate the effectiveness of the guides as a tool to increase knowledge and teach procedures, as well as the impact of the guides on physician confidence and procedural practice.

CURRICULAR DESIGN

All EM faculty, fellows, and residents in the department were eligible to participate in this study.

Survey Development

We used consensus decision-making to create a 12-question needs assessment focused on JIT resource utilization (Appendix A). Questions collected information regarding current resource use on shift to learn and teach procedures, barriers to using particular resources, frequency of specialty consultations for procedures due to lack of physician comfort in performing the procedure, and ideas for new resources. A five-point Likert scale was used to assess the degree to which physicians believed that their comfort level in performing procedures would change if they were provided with a detailed guide for a given procedure.

Five months after the implementation of the JIT procedure guides, we created a 14-question post-intervention survey (Appendix B) to assess the effectiveness of the guides as a tool to increase knowledge and teach procedures, as well as the impact of the guides on physician confidence and procedural practice. Five-point Likert scales were used to assess on average how often physicians required a JIT refresher prior to performing or teaching a given procedure, how often they used the JIT procedure guides, and how helpful they found these guides for both learning and teaching purposes. Physicians were also queried whether their procedural practice patterns had changed after the implementation of the guides. They were asked not only how their confidence had changed in performing procedures in the ED, but also whether their behavior had changed in terms of their decision to consult a specialist for a given procedure. We also assessed the quality of the procedure guides, most helpful features, and suggestions for improvement.

Study Procedure

The study was granted an exemption by the medical center’s institutional review board. The online cross-sectional anonymous needs assessment survey was completed in December 2020 using REDCap (Vanderbilt University, Nashville, TN).^6,7 The survey was disseminated using internal email distribution lists, and participants were given 10 weeks to complete the initial survey.

The needs assessment results directly informed the development of the JIT procedure guides and used Mayer’s principles of multimedia design.⁸ The 54 guides were distributed via the departmental website starting in December 2020 (Appendix C). Procedures covered were based on the Core Emergency Medicine Procedures as defined by the 2019 Model of the Clinical Practice of Emergency Medicine.¹ The guides were created using PowerPoint (Microsoft Corporation, Redmond, WA), with a simple color-coded schema that was visually appealing and easy to follow. The content of the procedure guides included links to high-quality videos curated from the internet or author-created videos, detailed indications/contraindications, an overview of relevant anatomy, standard procedural steps along with (where applicable) steps specific to the procedural kits stocked in our ED, clear visual depictions of our ED kits, complications to anticipate, and aftercare recommendations (Appendix D). Announcements were made via email and at departmental meetings about the availability of the JIT procedure guides.

In May 2021 the anonymous, online post-intervention survey was distributed via internal email distribution lists. Participants were given three weeks to complete the post-survey.

Data Analysis

We performed descriptive statistics and compared resident and faculty responses using Fisher’s exact testing due to the small sample size. All statistical analysis was performed using SPSS 27 (IBM Corp,, Armonk, NY)). Two authors performed thematic analysis of the responses to the two post-implementation survey questions, which asked about the most helpful features of the procedure guides and suggestions for improvement.

IMPACT/EFFECTIVENESS

Needs Assessment Results

The overall response rate for the initial needs assessment survey was 49.5% (53/107): residents 51.3% (20/39) and faculty 48.5% (33/68). The majority of survey respondents reported using online videos (86.8%, 46/53) and educational websites (79.2%, 42/53), with far fewer using personal notes (28.3%, 15/53), textbooks (24.5%, 13/53), or journal articles (11.3%, 6/53) for JIT procedural guidance. There was a statistically significant difference between the percentage of residents and faculty who reported using videos (100% ([20/20] of residents vs 79% [26/33] of faculty, P<0.05) and textbooks (5% [1/20] of residents vs 36% [12/33] of faculty, P<0.05] as JIT resources on shift (Figure 1).

The most common barriers to using JIT resources included limited time (83%, 44/53); resources not being specific to available procedural kits (62.3%, 33/53); and lack of curated, high-quality video resources (62.3%, 33/53). There was a statistically significant difference between the percentage of faculty and residents who identified a lack of a video library as a barrier (85% [17/20] of residents vs 48% [16/33] of faculty, P<0.01) (Figure 2). At least once within the last year, a majority of respondents (58.5%, 31/53) thought about performing a core EM procedure but ultimately did not do so due to lack of comfort in performing the procedure, and 88.7% (47/53) indicated their comfort level in performing a procedure would increase or significantly increase if detailed online procedure guides were available.

Post-Implementation Survey Results

The overall response rate for the post-intervention survey was 29% (31/107): residents 31% (12/39), and faculty 28% (19/68). The majority (58.3%, 14/24) indicated that they sometimes needed a refresher during their clinical shift prior to performing a procedure. When they required a refresher, 25% (6/24) of faculty and residents always used the developed JIT procedure guides: 37.5% (9/24) often used them; 29.2% (7/24) sometimes used them; and only 8.3% (2/24) rarely or never used them. Most respondents used the guides to improve general procedural knowledge (83.3%, 20/24) and to teach procedures in the ED (75%, 18/24).

The vast majority of respondents (95.8%, 23/24) found the procedure guides to be very helpful in increasing their procedural knowledge, while one respondent (4.2%), found them to be somewhat helpful; (faculty 100% [15/15] very helpful; residents 88.9% [8/9] very helpful, and 11.1% [1/9] somewhat helpful). The majority of faculty also found the guides to be very helpful for teaching procedures (86.7%, 13/15), with the remainder neutral (13.3% (2/15). Rating the quality of the guides, faculty unanimously found them to be excellent (100%, 15/15), while the majority of residents (75%, 6/8) found them to be excellent, and some (25%, 2/8) found them to be very good. A minority (22.6%, 7/31) of survey respondents indicated that they had not yet used the procedure guides, and of those respondents 85.7% (6/7) had not yet needed a JIT refresher. Only one respondent (14.2%, 1/7) indicated they had difficulty accessing the guides on the website.

The Table displays all themes, theme frequency, and representative comments to the questions “What are the most helpful (or best) features of the procedure guides?” and “What suggestions do you have for improving the procedure guides?”

Since the implementation of the guides, the majority of residents (75%, 9/12) and faculty (68%, 13/19) felt their confidence in performing procedures had increased or significantly increased. Importantly, 16% (5/31) of respondents indicated they had changed their procedural practice pattern and performed a core EM procedure, covered by our JIT procedure guides, that they previously would have asked a consultant to perform.

DISCUSSION

Our needs assessment indicated there is a clear need to have a readily accessible resource available for the purpose of JIT learning and teaching of EM procedures. While most physicians surveyed use online videos and educational websites, we found that a higher proportion of faculty use textbooks as a JIT resource relative to resident trainees (Figure 1). This could represent a broader pattern between different generations of learners and reinforces that educational interventions need to be targeted to the populations that are going to be using the resources provided.

We attempted to tackle the fact that different groups of learners not only have different learning needs when it comes to JIT procedural refreshers, but they also perceive different barriers. The procedure guides were an attempt to bridge this gap and meet the needs of multiple groups of learners by pulling useful aspects of each type of resource, including succinct text, clear visual depictions, and links to online videos.

One recurring theme that we found from respondents was that the guides were clear and concise. In the interest of keeping these guides concise, we chose to highlight only the primary procedural techniques and not make the guides all-inclusive. An additional theme from respondents was that it was helpful that we displayed and photographed, when applicable, the procedural kits specific to our ED. In some instances, we videotaped how to perform the procedure using our site-specific kits and included links to these videos in the guides. We believe that the ability to make these guides as specific to the institution as possible helped to increase their usefulness.

LIMITATIONS

Limitations of the study include a low survey response rate and a short, five-month intervention period. This decision was made to ensure we maintained the same cohort of faculty, fellows, and residents prior to graduation. We felt, however, that the benefit of keeping the same cohort of physicians in the pre- and post-surveys justified the short implementation timeline. In addition, the project was conducted at a single academic institution, which may limit the generalizability of the results to other institutions. Further helpful areas of study would include data collection on procedure performance and procedural numbers to measure change in practice patterns of individual EPs, as well as consideration of patient outcomes data after the implementation of the JIT procedure guides.

CONCLUSION

There is a clear need to have a readily accessible resource available for the purpose of just-in-time learning and teaching of EM procedures. Our results indicate that having clear, concise, readily accessible, and institution-specific JIT procedure guides can not only increase physicians’ confidence in their ability to perform core EM procedures in the ED, but also change physician behavior by potentially leading to reduced specialist consultation for these procedures, which may lead to a reduction in overall ED length of stay. Our guides represent an innovative resource that may appeal to multiple generations of emergency physicians, as they combine the most useful aspects of multiple resources including textbooks with more modern digital resources such as online videos.

Footnotes

Section Editor: Jeffrey Love, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Kendra Parekh, MD, Vanderbilt University School of Medicine, Department of Emergency Medicine, 1313 21st Avenue South, 703 Oxford House, Nashville, TN 37232. Email: kendra.parekh@vumc.org. 5 / 2022; 23:353 – 357

Submission history: Revision received June 14, 2021; Submitted January 7, 2022; Accepted February 28, 2022

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Beeson MS, Ankel F, Bhat R, et al. The 2019 Model of the Clinical Practice of Emergency Medicine. J Emerg Med. 2020;59(1):96-120.

2. Sawyer T, White M, Zaveri P, et al. Learn, see, practice, prove, do, maintain: an evidence-based pedagogical framework for procedural skill training in medicine. Academic Medicine. 2015;90(8):1025-33.

3. Schonnop R, Stauffer B, Gauri A, et al. Procedural skills training in emergency medicine physicians within the edmonton zone: a needs assessment. AEM Education and Training. 2021;5(2):p.e10495-n/a.

4. Thomas AA, Uspal NG, Oron AP, et al. Perceptions on the impact of a just-in-time room on trainees and supervising physicians in a pediatric emergency department. J Grad Med Educ. 2016;8(5):754-8.

5. Branzetti JB, Adedipe AA, Gittinger MJ, et al. Randomised controlled trial to assess the effect of a just-in-time training on procedural performance: a proof-of-concept study to address procedural skill decay. BMJ Qual Saf. 2017;26(11):881-91.

6. Harris PA, Taylor R, Thielke R, et al. Research Electronic Data Capture (REDCap) – A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-81.

7. Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform. 2019;95:103208-103208.

8. Mayer RE. Applying the science of learning to medical education. Medical Education. 2010;44(6):543-9.