Author	Affiliation
Nicholas Pokrajac, MD	Stanford University, Department of Emergency Medicine, Palo Alto, California
Kimberly Schertzer, MD	Stanford University, Department of Emergency Medicine, Palo Alto, California
Cori M. Poffenberger, MD	Stanford University, Department of Emergency Medicine, Palo Alto, California
Al’ai Alvarez, MD	Stanford University, Department of Emergency Medicine, Palo Alto, California
Paloma Marin-Nevarez, MD	Stanford University, Department of Emergency Medicine, Palo Alto, California
Christopher Winstead-Derlega, MD	Stanford University, Department of Emergency Medicine, Palo Alto, California
Michael A. Gisondi, MD	Stanford University, Department of Emergency Medicine, Palo Alto, California

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

The pandemic caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV2) and its resultant clinical illness, coronavirus disease 2019 (COVID-19), has stressed healthcare systems across the world. Nearly 10,000 healthcare workers contracted COVID-19 in the United States (US) alone during the period from February 12–April 9, 2020.¹ SARS-CoV2 spreads by means of surface contamination, exposure to droplets containing viral particles, and through aerosolization, particularly during high-risk procedures.² Healthcare workers are at increased risk for infection given frequent exposure to the virus during routine patient care.

Proper use of personal protective equipment (PPE) by healthcare workers is well established to decrease the rate of infectious disease transmission, including by means of self-contamination.³ However, prior studies have demonstrated that healthcare workers are inconsistent in the proper use of PPE. Contamination rates during donning and doffing of PPE range from 10–100%.^3,4 Deviations from accepted protocols for donning and doffing PPE also occur in 50–87% of healthcare workers.^4-6 Therefore, effective education to improve donning, wearing, and doffing of PPE is critical for healthcare worker safety.

A number of educational interventions to improve correct use of PPE have been performed with varying success.^3,7 A growing body of evidence suggests that simulation-based mastery learning (SBML) is more effective than other educational techniques to attain procedural skill mastery.^8,9 SBML is an educational technique that must include the following: 1) baseline testing of a target skill; 2) discrete learning objectives organized by rising difficulty; 3) attentive learner engagement during the activity; 4) a defined minimum passing standard (MPS); 5) testing during the educational process to direct learning and evaluate achievement of the MPS; 6) advancement after reaching the MPS; and 7) continued practice until the MPS is achieved.¹⁰ This method often is paired with deliberate practice, which requires highly motivated students to engage in focused, repetitive practice toward a specified goal with informative feedback to correct errors. The goal of SBML is to have all participants achieve an expert level of skill with minimal to no variation, which is crucial in patient care environments. Moreover, implementation of a SBML curriculum may improve translational outcomes.^11-13 The goal of this study was to determine whether mastery learning methodology can improve physician ability to correctly don and doff PPE during a simulated clinical encounter with a COVID-19 patient.

METHODS

Study Design and Approval

Physician subjects participated in a mastery learning educational intervention with a simulated clinical encounter of a patient with COVID-19.¹⁴ Prior to the intervention, all participants had received institution-mandated N95 mask-fit testing and training on the proper use of PPE via an online video and a supplemental online infographic demonstrating steps of donning and doffing of PPE. Participants were assessed on their ability to correctly don and doff PPE using a checklist before and after the intervention on the same day. The study was approved by our local institutional review board (IRB #55851).

Participants and Study Setting

Participants included clinical faculty members and emergency medicine (EM) resident physicians in the ED at a large, academic tertiary care center from March 31–April 8, 2020. The assessment was conducted in an administrative office space designed to simulate a medical examination room with a door and no anteroom, with a patient under airborne, droplet, and contact precautions.

Outcome Measures and Measuring Instrument

Participants were assessed individually prior to the intervention using a 21-item checklist of steps for donning and doffing PPE using a double-glove technique (Online Supplement).^15,16 The checklist was developed by one author and adapted from existing best-practices guidelines on PPE use from the US Centers for Disease Control and Prevention (Atlanta, GA) and Stanford University (Palo Alto, CA,).¹⁵ Additional authors with expertise in PPE use, medical education, and checklist design reviewed and modified the checklist. The checklist underwent final review and approval by infection control specialists at our institution to ensure completeness, compliance, and internal consistency within the hospital system. Equipment consisted of Medline isolation gowns, Medline Fitguard nitrile exam gloves (Medline Industries, Inc, Northfield, IL), and DeRoyal SPEyes Eye ShieldZ (DeRoyal Industries, Powell, TN). Due to a national shortage of N95 face masks at the time of this project, a simple substitution of quarter-inch elastic bands stapled to an 8-ounce paper bowl was used (Figure 1).

Adherence to checklist items during testing was assessed by four reviewers. All reviewers underwent a one-hour training on the use of the checklist as a rating instrument, which consisted of checklist review, demonstration of correct PPE use by a study author, deliberate practice, and mock assessments. Reviewers were instructed to give zero points to items not done or performed incorrectly and one point to items performed correctly. All reviewers independently scored at least 10% of participants to determine inter-rater reliability of the instrument.

A MPS for correct completion of checklist items was determined by 16 experts using a combination of Mastery Angoff and Patient Safety approaches.^17-19 All 16 experts were EM clinical faculty members, 10 of whom have advanced training in medical education, three in medical simulation, and one in emergency medical services.

Educational Intervention

Physician subjects individually participated in a mastery learning educational intervention if they did not achieve the MPS on the pretest assessment. The intervention consisted of an in-person expert demonstration of proper donning and doffing of PPE using the 21 steps outlined in the checklist, followed by a 20-minute opportunity for deliberate practice with feedback. If participants again did not achieve the MPS on reassessment, they were given additional opportunities for deliberate practice until the MPS was achieved. Final scores were determined by reviewers unblinded to initial participant assessments.

Statistical Analysis

We performed statistical analysis using SPSS Statistics for Windows, Version 24, (IBM Corp., Armonk, NY). Inter-rater reliability was determined by calculating Cohen’s kappa statistic. We used a two-tailed paired T-test to compare pre-and posttest scores. The difference between faculty and resident physicians’ scores was calculated using a two-tailed Student’s T-test.

RESULTS

A total of 117 physician subjects participated in the study, including 56 faculty members (56/88, 63.6%) and 61 EM resident physicians (61/62, 98.3%). Participant demographic information is summarized in Table 1.

Standard setting using a Patient Safety approach resulted in 19/21 of the checklist items deemed critical for safety. A Mastery Angoff score calculated for the two non-critical items was 90.2%. Requiring completion of all items deemed critical from the Patient Safety approach plus 90.2% of two non-critical items resulted in the final MPS set at 100%.^17-19

Agreement between assessors across checklist items ranged from moderate to strong (κ = 0.70 to 0.87). Two faculty members (3.6%) and one resident physician (1.6%) successfully achieved the MPS on pre-intervention assessment. Mean pretest score among all participants was 73.1% (95% confidence interval [CI], 70.9–75.3%). There was no significant difference between the mean pretest scores of faculty members and resident physicians (75.1% vs 71.3%, p = 0.082) (Figure 2).

Mean pretest doffing scores were significantly lower than donning scores (65.8% vs 82.8%, p<0.001). The items most commonly not completed or incorrectly completed included “adjusts nosepiece of mask,”, “demonstrates mask seal check,” “doffs eye shield in room,” “disposes of eye shield in room,” and “performs hand hygiene on inner gloves” (Table 2).

After our educational intervention, the mean participant score increased 26.9% (95% CI of the difference 24.7–29.1%, p<0.001) (Figure 3). No participants required more than 20 minutes to achieve mastery.

DISCUSSION

This study demonstrated that a (SBML) intervention with deliberate practice led to significant improvement in both faculty and resident physicians’ ability to correctly don and doff PPE. On pretest assessment, faculty and resident participants demonstrated frequent errors during donning and doffing of PPE despite completing comprehensive, institution-mandated online training. Similar to prior studies, errors were more common during doffing of protective equipment, which is when providers are at greatest risk of self-contamination.²⁰ Therefore, these results highlight a critical role for SBML to improve correct PPE use and suggest that the sole utilization of online modules for PPE use may be inadequate for workplace safety.

SBML is a highly effective method of teaching procedures.²¹ The focus of SBML is the achievement of a fixed learning outcome; training time varies between participants to allow adequate opportunities for deliberate practice with feedback. While completion time varied, no participant in our study exceeded 20 minutes of training time and all participants met our predetermined learning outcome on post-intervention testing. This approach is in contrast to more common instructional techniques, in which teaching time is fixed and participant achievement varies. As such, rigorous adherence to SBML principles as used in this study likely represents the gold standard for procedural training.¹³

Previous research demonstrated SBML to be an effective method of teaching both invasive and non-invasive procedures, including lumbar puncture,⁹ central line insertion,^11,12 paracentesis,²² and thoracentesis,^8,23 among others.¹³ Mastery learning also achieves translational outcomes that result in better patient care (T3) as defined by lower complications rates during high-risk procedures.^11,12,23 The proper donning and doffing of PPE for aerosolized infection is similarly high risk and correct completion (fixed achievement) of the don/doff sequence is absolutely necessary to minimize risk to patients and providers.

We designed our mastery learning intervention to simulate patient care performed in locations without anterooms, which comprise the majority of hospital and outpatient clinical spaces. This allows for authenticity in the logistical challenges present when caring for a high frequency of COVID-19 patients requiring airborne, droplet, and contact precautions. In addition, double-glove technique mitigates skin contact with potential surface contaminants while exiting a patient’s room. Providers may also have improved hand comfort with the application of alcohol-based gels over inner gloves rather than skin, given the frequency of required hand sanitization during donning and doffing of PPE.²⁴ The double-glove technique is presumed to increase provider safety for a variety of infectious diseases beyond COVID-19.

Finally, our intervention standardized PPE use among our physicians by allowing participants the opportunity to ask nuanced questions and practice repeatedly. This supportive and psychologically safe training may be especially important to mitigate provider anxiety in anticipation of a COVID-19 surge. A shared mental model also allows observation and direct feedback by faculty-resident pairs during donning and doffing of PPE. The ability to train the majority of our emergency physicians in a one-week time period suggests that comprehensive physician training is feasible. Future study is needed to determine the potential translational impact of our intervention.

LIMITATIONS

This study had several limitations. First, the checklist instrument required use of a double-glove rather than single-glove technique, the latter of which was included as part of the pre-existing, institution-mandated online training for PPE use. As a result, providers may have been less familiar with the double-glove technique on pretest. Double-glove technique was chosen due to the prevalence of patient care areas without anterooms at the institution. This technique was also chosen in an effort to optimize both comfort and personal protection in light of changing guidelines during the pandemic. Second, the strict MPS of the assessment may have decreased the initial pass rate on pretest. Third, reviewers were unblinded to pretest results, which may have influenced posttest scores. Fourth, due to time constraints, participants completed repeat testing immediately following deliberate practice, which limited assessment for skill retention. Finally, while previous studies demonstrated improvements in translational outcomes following SBML, it is still unclear whether this intervention will result in an observable change of behavior during patient care.

CONCLUSION

Mastery learning methodology with deliberate practice is an effective and feasible educational modality for training a large number of physicians in the proper use of PPE in simulated clinical encounters of patients with COVID-19. Before our training intervention, very few providers passed a rigorous assessment of PPE donning and doffing despite institution-mandated, online PPE training. After undergoing the SBML intervention, all participants successfully completed assessment with 100% accuracy. In addition, the double-glove technique may offer additional provider protection when caring for a high volume of COVID-19 patients in treatment areas that lack anterooms. Further study of translational outcomes resulting from our intervention is needed.

Footnotes

Section Editor: Ioannis Koutroulis, MD and Danya Khoujah, MBBS

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Nicholas Pokrajac, MD, Stanford University, Department of Emergency Medicine, 900 Welch Drive, Suite 350, Palo Alto, CA 94304. Email: pokrajac@stanford.edu. 9 / 2020; 21:1089 – 1094

Submission history: Revision received May 11, 2020; Submitted June 25, 2020; Accepted June 24, 2020

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Characteristics of Health Care Personnel with COVID-19, United States, February 12–April 9, 2020. MMWR Morb Mortal Wkly Rep. 2020;69(15):477-81.

2. Van Doremalen N, Bushmaker T, Morris DH, et al. Aerosol and surface stability of SARS-CoV-2 as compared with SARS-CoV-1. N Engl J Med. 2020;382(16):1564-7.

3. Verbeek JH, Rajamaki B, Ijaz S, et al. Personal protective equipment for preventing highly infectious diseases due to exposure to contaminated body fluids in healthcare staff. Cochrane Database Syst Rev. 2020;4(4):CD011621.

4. Kwon JH, Burnham CD, Reske KA, et al. Assessment of healthcare worker protocol deviations and self-contamination during personal protective equipment donning and doffing. Infect Control Hosp Epidemiol. 2017;38(9):1077-83.

5. Mulvey D, Mayer J, Visnovsky L, et al. Frequent and unexpected deviations from personal protective equipment guidelines increase contamination risks. Am J Infect Control. 2019;47(9):1146-7.

6. Zellmer C, Van Hoof S, Safdar M. Variation in health care worker removal of personal protective equipment. Am J Infect Control. 2015;43(7):750-1.

7. Tomas ME, Kundrapu S, Thota P, et al. Contamination of health care personnel during removal of personal protective equipment. JAMA Intern Med. 2015;175(12):1904-10.

8. Barsuk JH, Cohen ER, Williams MV, et al. Simulation-based mastery learning for thoracentesis skills improves patient outcomes: a randomized trial. Acad Med. 2018;93(5):729-35.

9. Barsuk JH, Cohen ER, Caprio T, et al. Simulation-based education with mastery learning improves residents’ lumbar puncture skills. Neurology. 2012;79(2):132-7.

10. McGaghie WC, Siddall VJ, Mazmanian PE, et al. Lessons for continuing medical education from simulation research in undergraduate and graduate medical education: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest. 2009;135(3 Suppl):62S-8.

11. Barsuk JH, McGahie WC, Cohen ER, et al. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med. 2009;37(10):2697-701.

12. Barsuk JH, Cohen ER, Potts S, et al. Dissemination of a simulation-based mastery learning intervention reduces central line-associated bloodstream infections. BMJ Qual Saf. 2014;23(9):749-56.

13. McGaghie WC, Issenberg SB, Barsuk JH, et al. A critical review of simulation-based mastery learning with translational outcomes. Med Educ. 2014;48(4):375-85.

14. Matola I, Devine LA, Chung HS, et al. A critical review of simulation-based mastery learning with translational outcomes. Med Teach. 2013;35(10):e1511-30.

15. Tools for protecting healthcare personnel. 2014. Available at: https://www.cdc.gov/hai/prevent/ppe.html. Accessed March 30, 2020.

16. Casanova LM, Rutala WA, Weber DJ, et al. Effect of single- versus double-gloving on virus transfer to health care workers’ skin and clothing during removal of personal protective equipment. Am J Infect Control. 2012;40(4):369-74.

17. Barsuk JH, Cohen ER, Wayne DB, et al. A comparison of approaches for mastery learning standard setting. Acad Med. 2018;93(7):1079-84.

18. Yudkowsky R, Park YS, Lineberry M, et al. Setting mastery learning standards. Acad Med. 2015;90(11):1495-500.

19. Yudkowsky R, Tumuluru S, Casey P, et al. A patient safety approach to setting pass/fail standards for basic procedural skills checklists. Simul Healthc. 2014;9(5):277-82.

20. Mumma JM, Durso FT, Casanova LM, et al. Common behaviors and faults when doffing personal protective equipment for patients with serious communicable diseases. Clin Infect Dis. 2019;69(Suppl 3):S214-20.

21. McGaghie WC, Issenberg SB, Cohen ER, et al. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011;86(6):706-11.

22. Barsuk JH, Cohen ER, Vozenilek JA, et al. Simulation-based education with mastery learning improves paracentesis skills. J Grad Med Educ. 2012;4(1):23-7.

23. Wayne DB, Barsuk JH, O’Leary KJ, et al. Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice. J Hosp Med. 2008;3(1):48-54.

24. Girard R, Bousquet E, Carré E, et al. Tolerance and acceptability of 14 surgical and hygienic alcohol-based hand rubs. J Hosp Infect. 2006;63(3):281-8.