Introduction
Materials and methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Point-of-care ultrasound (POCUS) of the aorta to diagnose abdominal aortic aneurysm (AAA) is a core emergency ultrasound application and an essential component of emergency medicine (EM) residency education.
¹
Emergency department (ED) POCUS has previously been shown to have excellent performance characteristics for the evaluation of AAA.
²
Simulation is a commonly used educational tool for resident ultrasound education, but a scoping review found the majority of prior studies examined changes in confidence, knowledge, and skills rather than objective clinical performance.
³
However, simulation in addition to clinical training has been shown to be effective at translating to clinical performance in other specialties, such as for obstetrics and gynecology residents learning transvaginal ultrasound.
⁴

It is critical that emergency physicians learn to quickly identify AAA at the bedside, as this is a time-sensitive and potentially deadly diagnosis requiring a goal time from presentation to emergency surgery of <90 minutes.
⁵
We aimed to investigate whether the addition of case-based ultrasound simulation to the existing EM residency curriculum was associated with an increase in the quantity and quality of aorta POCUS performed on ED patients.

MATERIALS AND METHODS

Design, Setting, and Intervention

At our academic, Level I trauma center with a three-year EM residency program and advanced practice provider training program, EM residents have existing simulation-based learning built into their monthly academic conference. However, the use of POCUS is not typically incorporated into simulation at our institution. In Spring 2023, we introduced a new simulated case requiring the use of POCUS for diagnosis of AAA in the setting of undifferentiated abdominal pain. During simulation-based learning, residents are divided into groups of 5–8 residents of varied postgraduate year (PGY) and cycle between the simulation session and other educational activities. In addition to verbal prompts regarding case history, physical exam findings, and patient responses to interventions, residents were asked to use a cart-based ultrasound system (Sonosite PX, Fujifilm, Bothell, WA) to examine an ultrasound phantom abdomen (41903–000, Kyoto Kagaku, Japan) with multiple findings including an infrarenal AAA with intermural thrombus, free abdominal fluid, and normal bowels and renal system. The case concluded with a hands-on didactic led by a POCUS fellowship-trained emergency physician and included time at the end of the simulation for hands-on scanning by residents.

Ultrasound Study Review and Outcomes

All ultrasound studies performed in the ED are submitted through a quality assurance workflow for review by a team of POCUS fellowship-trained faculty. All residents who participate receive credit for performing the POCUS. The submitting resident who performed the POCUS completes a worksheet describing the findings, interpretation, and study limitations. Studies are marked complete if residents indicate that a view was obtained of the suprarenal aorta, infrarenal aorta, and iliac bifurcation and incomplete if one or more of these views was not obtained. Images and worksheets are then sent for review and signature to the faculty caring for the patient with the resident. Faculty can either place the study in an educational archive (if they did not use the study for medical decision-making) or a clinical archive (if they used the study for medical decision-making and wish for the images to transfer to the patient’s health record).

Studies are reviewed for quality and assigned a quality score from 1 (worst) to 5 (best) as well as notation of any false positives or false negatives, with EM ultrasound faculty serving as the gold standard. Quality scores 1 and 2 are considered insufficient for diagnosis, with scores of 3–5 considered adequate. We examined cumulative measures of sensitivity and specificity before and after the simulation.

Analysis

We compared the median number of aorta POCUS studies that EM residents performed in the ED in the four months prior to the simulation session to the median number of exams performed in the four months following the simulation session. These quantities were reported as scans performed per 100 shifts per resident and compared using Wilcoxon signed-rank test. We compared mean quality scores over the four months prior to the intervention to the quality scores over the four months after the intervention as described above with the quantity of exams using a paired t-test. Proportions of limited studies and training year distribution were compared with Pearson chi-square. The significance level of all tests was set to 0.05 with Bonferroni correction applied where appropriate. Analysis was performed in SPSS for Macintosh, v 28.0 (SPSS Inc, Chicago, IL). This study received institutional review board approval for waiver of signed informed consent.

RESULTS

Over half of residents 17/32 (53%) participated in the simulation session and had at least one clinical shift before and after the simulation session. The distribution of the participating residents as primary study authors is demonstrated in the Table, with a significant increase in the proportion of aorta POCUS submitted by interns. Overall, there was an 86% increase in total studies and an 80% increase in clinical studies after the session. However, when comparing on a per-resident basis while adjusting for clinical opportunities, there was no significant difference in median total scanning frequency per 100 shifts (4.4 [interquartile range (IQR) = 0–15.8] vs 8.3 [IQR = 3.3–23.6], P = 0.21). There was also no significant change in average total quality scores on a per resident basis (3.2 ± 0.6 vs 3.2 ± 0.5, P = 0.92).

There were no false negative or false positives using faculty review of images as the gold standard. There were no differences in the proportion of studies with agreement vs disagreement with the resident interpretation (100% vs 96%, P = 0.29). There was a decrease in the total number of limited or inadequate studies (12/28 (43%) vs 10/52 (19%), P = .02 [X
²]). There was no significant change in the proportion of clinical studies submitted as “limited” or “inadequate” (2/5 (40%) vs 4/9 (44%), P = 0.87 [X
²]), but the number of educational studies submitted as “limited” or “inadequate” improved (10/23 (44%) vs 6/43 (14%), P = <.001).

DISCUSSION

Overall, the total number of aorta POCUS studies performed in the ED after the simulation increased, albeit without a demonstrable change in quantity or quality rating on a per-resident basis. However, there were a number of positive findings, which support the inclusion of ultrasound simulation in residency training, including a significant increase in the proportion of studies contributed by interns and a significant decrease in the proportion of studies that were incomplete or limited.

It seems unlikely that the significant increase in intern POCUS studies was due to content mastery based on compounding clinical experience alone. Aorta POCUS is one of the applications requiring the most experience to gain proficiency, and with previously demonstrated plateau points in interpretation and acquisition at 66 studies and 84 studies, respectively, which were not approached by anyone in our study.
⁶
The same study found that aorta POCUS quality actually decreases initially with increasing number of scans before it eventually improves above baseline, which may be contributing to the absence of improvement in median quality scores seen in our study.
⁶

Much of the published research regarding POCUS simulation reports outcomes related to the assessment of learner experience and skill performance outside the clinical context.
<sup>7

,

8</sup>
While these outcomes are important, there is a desire to assess more translational outcomes resulting from simulation interventions.
<sup>9

,

10</sup>
There are few translational studies available for direct comparison to the current study. Our simulation experience was delivered as a single session, which is less time-intensive than a prior study of EM interns that found positive clinical effects of simulation-based mastery learning on performance of focused assessment with sonography in trauma.
¹¹
Outside of EM, a randomized trial of ultrasound simulation for obstetrics and gynecology residents that was also more longitudinal than the current study also found positive clinical effects of early simulation training, in addition to clinical practice in first-year residents.
⁴
Further study is needed to determine whether the case-based simulation approach in our study would be more successful if the training were more time intensive, more longitudinal, and most targeted toward junior learners.

LIMITATIONS

Our findings are subject to the limitations of a before-and-after study, including the possibility that other factors may have contributed to the observed changes other than the simulation; however, we are not aware of any other targeted effort to educate our residents about aorta POCUS during the study period, and believe it is likely most changes were associated with the simulation. Second, this was a study of residents whose categorization of study intent was subject to their signing faculties’ preferences. Therefore, we included both educational and clinical archive studies to provide a fair portfolio of each resident’s work, although some educational studies may have not been intended for patient care. Third, EM residents do not get to choose which patients they take care of in the ED (because emergencies are unplanned and unpredictable); so some residents likely had slightly more exposure to patients with indications for aorta POCUS than others. In addition, while we were able to adjust for clinical opportunities based on time spent in the ED, residents often see more patients per hour as they advance through training, and we were unable to account for total patients seen during the study period. Even considering these limitations, we believe the data presented provides an accurate real-world assessment of scanning frequency and quality on ED patients by ED residents.

CONCLUSION

In the four months following a case-based simulation to diagnose abdominal aortic aneurysm using point-of-care ultrasound, the proportion of aorta POCUS studies performed in the ED by interns increased significantly, and the proportion of studies that were incomplete or limited significantly decreased. While there was no overall increase in the median number of scans or mean quality scores when adjusted for clinical opportunities on a per-resident basis, among residents as a whole there was an 86% increase in submitted aorta POCUS studies.

Footnotes

Section Editor: Shane Summers, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Brandon M. Wubben, MD, University of Iowa Hospitals and Clinics, Department of Emergency Medicine, 200 Hawkins Drive, Iowa City, IA 52242. Email: brandon-wubben@uiowa.edu
03 / 2024; 25:205 – 208

Submission history: Revision received August 30, 2023; Submitted December 4, 2023; Accepted December 5, 2023

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. This work was supported by an Educational Innovation and Scholarship Grant provided by the Carver College of Medicine at the University of Iowa.

REFERENCES

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.