Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

The emergency physician trainee educational environment of both emergency department (ED) and off-service rotations has changed over the last decade with a rapid increase in staffing by non-physician practitioners (NPP), often nurse practitioners and physician assistants.^1-3 Recently the American Academy of Emergency Medicine Resident and Student Association (AAEM/RSA), as well as several other EM and EM resident societies, published policy statements detailing concerns and best practices for the presence of NPPs—and their postgraduate training programs—in EDs with emergency medicine (EM) residents.^4,5 However, there is scant literature assessing the effects of NPPs on physician resident education across the breadth of medicine and no literature specific to EM.

Most prior studies evaluating the impact of NPPs on physician trainee education are from surgical specialties, which consistently report reduced workload, primarily due to reduced documentation responsibilities. Findings are mixed with respect to the impact on residents’ education, with conflicting reports of better operative experiences because of fewer floor pages vs forfeiting some operative procedures to NPs and PAs.^6-8

One survey across an entire academic institution found that NPs reported contributing positively to the education experience of resident physicians.⁹ Another study found a generally positive impact on intensive care unit fellow education according to fellowship directors.¹⁰ Notably, the residents and fellows whose education was being assessed were not included in either study.

Additionally, an important development across US academic medical centers that has not been captured in any prior studies of resident education to date in any specialty is the increasing number of institutions that now host postgraduate training programs for NPPs.⁴ Early studies have started to evaluate such programs in EM but once again without the perspectives of physician residents, leaving a primary stakeholder unaddressed.^11,12 Our primary objective in this study was to establish how EM residents perceive the effects of NPPs on their education, both while on service in the ED and off service. Our secondary objective was to establish whether those perceptions are associated with the presence vs absence of NPP postgraduate training programs in EM.

METHODS

In-depth paradata are available in Appendix A. Best practices for survey research were followed and reported using recommended reporting guidelines.^13,14 The study was confirmed to be exempt by the Washington Hospital Health System Institutional Review board.

Participants and Eligibility

The sampling frame was EM residents at US (state and territory) Accreditation Council for Graduate Medical Education (ACGME)-accredited programs who are members of the American Academy of Emergency Medicine Resident and Student Association (AAEM-RSA), drawn from its files. Residents in combined programs, such as EM/internal medicine, were included as well. Personal leave was not an exclusion criterion since items assessed perspectives over varying amounts of time.

Survey Method and Validity Evidence

Our primary objective was to capture the physician residents’ perspectives of the effects of NPPs on their education, which is intangible and, thus, best suited for a survey approach.¹⁵ The instrument was built using established best practices including expert and stakeholder involvement, layout recommendations, cognitive interviews, pilot testing, and nonresponse bias analysis.^16,17 Appendix A contains all relevant paradata.

Population Health Research Capsule

What do we already know about this issue?

Non-physician practitioners (NPPs) are a growing part of the workforce in emergency departments with emergency medicine (EM) residency programs.

What was the research question?

What is the impact of NPPs on resident education from the EM resident’s perspective?

What was the major finding of the study?

66.9% of residents reported a detracting impact on their education. Presence of an NPP postgraduate program was associated with 14x increase in resident-forfeited procedures.

How does this improve population health?

Excellent physician education is critical to addressing population health needs throughout the country.

We applied Messick’s validity framework consistent with recommendations by the American Association of Public Opinion Research.¹⁸ Appendix A includes a full validity evaluation, and Appendix B is the final instrument. Educational items were drawn from an existing instrument with good validity evidence,⁸ while gender and race/ethnicity demographics items were drawn from publicly available Association of American Medical Colleges records.¹⁹ The procedure list was drawn from the ACGME requirements,²⁰ and designations of unsupervised NPP practice laws were drawn from a published third-party review study.²¹ A complete list of existing instruments that we considered for use, along with our rationale for inclusion or exclusion in this study, is available in Appendix A.

The survey was open from June 4–July 5, 2021 and distributed electronically via Qualtrics (Qualtrics International, Inc., Provo, UT) with four reminders and an electronic pre-notification the week before the initial invitation, all consistent with best practices.¹⁷ Although the cohort was AAEM/RSA resident physicians, neither AAEM/RSA as an organization, nor its employees or representatives, were involved in any part of this study including instrument creation and analysis, other than simply distributing the instrument to its members.

Statistical Analysis

We used skewness and kurtosis to assess whether items met normal distribution requirements for parametric analyses.²² The Mann-Whitney U test, chi-square, and Fisher exact tests were used to compare the frequencies of events and scale ratings against the presence of an NPP postgraduate training program, postgraduate year status, and the existence of state laws regarding NPP supervision. We used the Spearman correlation coefficient to assess correlations.

Per published recommendations, we determined a priori to assess for nonresponse bias with both a wave analysis and demographics comparison (Appendix A).¹⁷ Qualitative analysis was conducted using a holistic coding approach.²³ Two authors simultaneously developed codes and applied them accordingly (Appendix A). We followed reporting guidelines from Academic Medicine for surveys and qualitative data.^13,24

RESULTS

Respondents

We received 393 partial and complete responses of 1,168 invitations that were confirmed received and viewed, yielding a 34% response rate. Table 1 shows the respondents’ demographics. State representation is in Appendix C. No empirical evidence for nonresponse bias using two independent analyses was found (Appendix A).

General Education and Work Experience

Detailed responses to Likert-type items are in Table 2; histograms are shown in Appendix D. Residents reported a generally unchanged or lighter workload and generally unaffected documentation time due to NPPs in the ED. In contrast, more than two-thirds of residents reported a negative impact of NPPs in the ED on their education. Residents reported having limited confidence in the local and national institutions responsible for ensuring the quality of residents’ medical education with respect to the presence of NPPs in the ED. One-third of residents reported feeling no confidence at all in being able to report concerns about the presence of NPPs in the ED without retribution.

Enhancing and Detracting Educational Impact of Non-physician Practitioners

Responses to the two narrative items evaluating how NPPs in the ED enhance and detract from EM resident education are characterized in Table 3.

Procedure Experiences in the Emergency Department and Off Service

Appendix E shows the complete numeric breakdown of procedure types and the number of each of those procedures forfeited in the ED and off-service, in addition to histograms for the same information. Table 4 describes the reasons why procedures were forfeited. All narrative responses and their final codes are in Appendix F.

Across all procedures, 264 residents (57.2%) reported at least one procedure for their patient being performed by an NPP during an ED rotation and 220 (59.5%) during an off-service rotation. The median number of procedures being performed by an NPP on residents’ patients in the ED was 2.00, while the off-service median was 2.5. The total number of forfeited procedures correlated inversely with the perception of overall impact on education (ie, forfeited procedures were associated with the perception of detraction from education), r_s=.381, P<.001, r²=0.14. Almost one-third (30.5%) of 269 responding residents reported at least one instance during an EM rotation of a patient being preferentially assigned to an NPP. Slightly fewer (25% of 200) reported at least one such instance while off service.

Conversely, 53.3% of 246 respondents reported having at least one patient preferentially assigned to them (physician resident) in lieu of an NPP because of the educational value during an EM rotation. Only 26.1% of 203 respondents reported the same on an off-service rotation. Additionally, of 280 residents who responded to the survey, 15% reported that an NPP taught or supervised them for at least one procedure in the ED, whereas 213 (38%) reported at least one such occurrence while on an off-service rotation.

When asked why procedures were forfeited, unit culture was independently cited significantly more frequently for off-service rotations than for ED rotations (28.0% vs 19.7%, χ² (1, n=132)=10.696, P<.01, Cramer’s V=0.285). “Intimidation” as a theme was expressed only in the ED responses.

Presence of EM Postgraduate Training Programs for Non-physician Practitioners

The presence of a postgraduate training program for NPPs was not significantly associated with residents’ impression of the overall impact of NPPs on their education (P=.26, Fisher exact test) or on their confidence in being able to report concerns about NPPs to local leadership without retribution (χ² (4, n=347)=1.290, P=.87). However, EM residents were significantly more likely to have forfeited at least one procedure on their patients than those without such programs on both EM rotations (69% vs 50%, χ² (1, n=264)=9.160, P<.01, Cramer’s V=.186) and off-service rotations (68.8% vs 54.3%, χ² (1, n=220)=4.422, P=.04, Cramer’s V=.142). Significantly more total procedures in the ED were forfeited as well by residents whose institutions hosted an NPP postgraduate EM program (median = 7.0) compared to not [median =0.5, U(N_with=100, N_without=164)=6,070.5, z=−3.687, P<.001, η²=.052], a factor of 14x, and trended similarly for reports of off-service procedures [median =7.5 with vs 2.0 without, U(N_with=80, N_without=140)=4,769.0, z=−1.894, P=.06].

Twice as many residents reported forfeiting at least one educational ED patient encounter at programs with an associated NPP postgraduate program than those without (43.6% vs 21.3%, χ² (1, n=249)=13.965, P<.001, Cramer’s V=0.237), but the reverse was not true for patients being preferentially assigned to residents (59.3% vs 47.6%, χ² (1, n=229)=2.970, P=.09). The presence of an NPP postgraduate EM program was not significantly associated with the incidence of teaching or supervision by an NPP in the ED (18.8% vs 11.7%, with and without, respectively, χ² (1, n=259)=2.483, P=.12).

EM Resident Postgraduate Training Status

Postgraduate year (PGY) status was not associated with a difference in probability of having forfeited a procedure to an NPP in the ED but was during off-service rotations, with 63.0%, 70.9%, and 47.1% of PGY 1, 2, and 3+, respectively, reporting at least one forfeiture, χ² (2, n=237)=10.101, P<.01, Cramer’s V=0.206. There was no significant difference in the incidence of forfeiting at least one highly educational patient to NPPs across PGY status in the ED [30.0%, 29.1%, 30.4% for PGY 1, 2, 3+, respectively, χ² (2, n=251)=0.036, P=.98] or off-service rotations [20.6%, 9.5%, 9.0% for PGY 1, 2, 3+, respectively, χ² (2, n=199)=1.081, P=.58].

DISCUSSION

A substantial majority of EM residents in AAEM-RSA reported that NPPs in the ED have a detracting or greatly detracting impact on their education. The presence of an EM NPP postgraduate training program was associated with a significantly greater median number of forfeited procedures but not with effects of NPPs on resident perception of education. Additionally, more than one-third of residents reported feeling “not confident at all” that they could approach local leadership about NPP concerns without facing retribution, and almost two-thirds of residents were “not confident at all” that the ACGME would satisfactorily address concerns about NPPs impacting resident education as reported in the oversight body’s year-end survey.

Of further interest are the measures that were not statistically significant. For example, the data do not show that more experienced senior residents are the ones primarily forfeiting procedures; rather, there was no significant difference by PGY status. Additionally, a higher number of forfeited procedures was significantly associated with a negative effect of NPPs on education, and NPP postgraduate programs were significantly associated with a higher number of forfeited procedures. Nonetheless, NPP postgraduate programs were not significantly associated with a difference in the overall perception of NPP effects on resident education. This finding suggests there is at least one mitigating factor of NPP postgraduate programs that balances the loss of procedures.

Of note, forfeited procedures reported here are for those patients the residents were primarily managing, procedures for whom the residents were ostensibly responsible. The number of such forfeited procedures was moderately associated with residents’ perception of educational effects, accounting for 14% of the variance; nevertheless, the confluence of data suggests a phenomenon that is far more complex than frustration over fewer opportunities for procedures.

The narrative responses were telling with respect to the hidden curriculum, which is generally described as a construct for the effects of tacit learning as a confluence of culture, structures, and institutions.²⁵ Intimidation and unit-culture themes suggest a new facet for a hostile learning environment and conditions that appear to leave physician trainees feeling defenseless. The example quotation from Table 4—“They just push their way in and tend to have the support of the administration”—points to a structured, even if unintentional, hidden curriculum that is a hindrance to physician resident education.

The narrative responses also described a loop of exclusion in which residents were told at times that they were required to forfeit their procedure so an NPP could have more experience but also reported times in which they were required to forfeit their procedure because the NPP had more experience than the resident. Findings from the Kang et al study alluded to a similar phenomenon in the operating room for junior residents,⁸ and it is thus not surprising that narrative responses described problematic relationships with NPPs on off-service rotations as well, putting at jeopardy the value-add of off-service rotations for EM residents.

Comparison to the study of surgical residents by Kang and colleagues bears striking contrasts across items that were replicated in our instrument.⁸ A full 88% of their respondents reported that NPPs made their workload lighter or much lighter, compared to 46.5% in our study. Similarly, 86% of surgical residents reported that NPPs enhanced or greatly enhanced care, whereas only 18.5% of emergency physicians shared the same opinion in the ED setting. Finally, 47% of surgical residents felt that NPPs enhanced or greatly enhanced their education (with 47% reporting no impact), whereas 66.9% of EM residents reported that an NPP presence detracted or greatly detracted from their education (with 29.8% reporting no impact), which is essentially the inverse of the surgical findings. The perceived workload and educational benefits found in the surgical specialties are not translated in EM from the perspective of EM residents.^6,8 The specialized and largely procedural nature of surgical education is distinct from the breadth of case exposure required for EM education. The contrast makes clear that surgical and EM resident cohorts are different, and conclusions cannot be inferred across the two groups.

The conflicting findings in the surgical literature of reduced workload on the one hand but reduced procedural opportunities on the other was present in EM residents’ responses as well. One of the most frequently cited educational enhancements provided by NPPs in the ED (17.9%) was reduced workload via fewer lower acuity patients to see, thereby allowing an educational emphasis for residents on more complex patients. By the same token, however, one of the most frequently cited detractions from education as a result of NPPs in the ED (47.7%) was the reduction of cases, including lower acuity cases.

As EM societies grapple with this issue, identifying institutional features of the reported positive interactions will be essential to inform best practices to improve the team relationship.⁴ Within that context, two aspects of structured interaction between NPPs and EM residents must be independently addressed: 1) NPPs in the ED as staff; and 2) NPPs as postgraduate trainees. One resident response in particular was telling with regard to the potential negative impact of the postgraduate programs on physician residents, given the recent report of a novel, parallel track postgraduate physician assistant program:¹¹ “They are in a parallel training environment with different standards and often give [physician residents] sub-par advice or worse, aggressive sub-par advice because they consider themselves more advanced.” It is likely sentiments such as these from physician residents that have led to the AAEM-RSA calling for the cessation of NPP postgraduate programs.²⁶

LIMITATIONS

Our survey asked the survey participants for recall over the course of a full year, which raises the potential for recall bias; however, none was found on the pilot test/retest analysis, supporting item reliability. An additional limitation is that our sampling frame was of a group that did not include every resident in the US and whose members are part of a specific EM society. It is notable that the sampling frame still represents almost three-fifths of all EM residents in the US, a large group indeed. Additionally, most AAEM-RSA members have membership through their programs, suggestive of those programs supporting less involvement of NPPs in resident education if consistent with AAEM and AAEM-RSA position statements. Thus, our findings would be underestimates of the detracting educational effects of NPPs on resident education and of forfeited procedures. It is also worth noting that the other major resident societies, the Emergency Medicine Residents’ Association and the American College of Osteopathic Emergency Physicians’ Resident Student Organization, also signed on to the letter regarding NPP involvement in resident education, underscoring that there is clearly not a bias of our particular cohort.⁴

Importantly, in this study we evaluated how many procedures and patient opportunities were lost but did not count how many were experienced in total, which is an undoubtedly larger and similarly consequential number. Fourteen lost procedures in a year could represent any percentage; the denominator is unknown. Finally, our study focused on the educational aspect of NPPs in the ED from the perspective of EM residents. Staffing models must also account for throughput, cost, and myriad other factors.

Future Study

Although our study focused on physician residents because they were not previously studied, all stakeholders—including physician residents, attendings, staff NPPs and NPPs in postgraduate programs, medical directors, and department administrators—must be included in addressing what residents report to be a hindrance in their education. Additional study and intervention are warranted regarding residents’ lack of confidence in local leadership and the ACGME. Finally, the findings in this study provide tangible evidence of the theoretical concerns raised by the major EM societies.

CONCLUSION

A strong majority of resident members of AAEM-RSA report that non-physician practitioners in the ED have a detracting impact on their overall education and opportunities for learning cases and procedures, at least in part because of preferential treatment of NPPs. Educational enhancement was reported but limited. Residents overwhelmingly do not have confidence in local or national authorities to address potential concerns about NPPs in the ED impacting their education.

Footnotes

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Andrew W. Phillips, MD, MEd, staff physician, DHR Health Research Institute, 5323 S McColl Rd, Edinburg, TX 78539. Phillips.EM.CCM@gmail.com. 5 / 2023; 24:588 – 596

Submission history: Revision received September 4, 2022; Submitted January 28, 2023; Accepted February 4, 2023

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Chekijian SA, Elia TR, Horton JL, et al. A Review of interprofessional variation in education: Challenges and Considerations in the Growth of Advanced Practice Providers in Emergency Medicine. AEM Educ Train. 2021;5(2):e10469.

2. Moote M, Krsek C, Kleinpell R, et al. Physician assistant and nurse practitioner utilization in academic medical centers. Am J Med Qual. 2011;26(6):452-60.

3. Moote M, Krsek C, Kleinpell R, et al. Republished: Physician assistant and nurse practitioner utilization in academic medical centers. Am J Med Qual. 2019;34(5):465-72.

4. Moreno LA, Aintablian H, Sutter RE, et al. Joint Statement Regarding Post-Graduate Training of Nurse Practitioners and Physician Assistants. 2020. Available at: https://www.aaem.org/UserFiles/file/AllEMorgjointstatementrepostgradtrainingofNPandPAs.pdf. Accessed April 4, 2023.

5. Updated position statement on non-physician practitioners. 2020. Available at: https://www.aaem.org/resources/statements/position/updated-advanced-practice-providers. Accessed April 4, 2023.

6. Eaton B, Hessler L, O’Meara L, et al. The impact of advanced practice providers on the surgical resident experience: Agree to disagree?. Am J Surg. 2019;217(6):1107-11.

7. Kahn SA, Davis SA, Banes CT, et al. Impact of advanced practice providers (nurse practitioners and physician assistants) on surgical residents’ critical care experience. J Surg Res. 2015;199(1):7-12.

8. Kang R, Columbo JA, Kunkel ST, et al. Residents’ impressions of the impact of advanced practice providers on surgical training. J Am Coll Surgeons. 2018;226(6):1036-43.

9. Bahouth M, Esposito-Herr MB, Babineau TJ. The expanding role of the nurse practitioner in an academic medical center and its impact on graduate medical education. J Surg Educ. 2007;64(5):282-8.

10. Joffe AM, Pastores SM, Maerz LL, et al. Utilization and impact on fellowship training of non-physician advanced practice providers in intensive care units of academic medical centers: a survey of critical care program directors. J Crit Care. 2014;29(1):112-5.

11. Tsyrulnik A, Goldflam K, Coughlin R, et al. Implementation of a physician assistant emergency medicine residency within a physician residency. West J Emerg Med. 2021;22(1):45-8.

12. Tsyrulnik A, Goldflam K, Coughlin R, et al. Response to “Implementation of a physician assistant emergency medicine residency within a physician residency.”. West J Emerg Med. 2021;22(1):49.

13. Gehlbach H, Artino AR. The survey checklist (manifesto). Acad Med. 2018;93(3):360-6.

14. Phillips AW, Durning SJ, Artino AR. Survey Methods for Medical and Health Professions Education: A Six-Step Approach. 2021.

15. Phillips AW. Proper applications for surveys as a study methodology. West J Emerg Med. 2017;18(1):8-11.

16. Artino AR, Rochelle JSL, Dezee KJ, et al. Developing questionnaires for educational research: AMEE Guide No 87. Med Teach. 2014(36):463-74.

17. Phillips AW, Reddy S, Durning SJ. Improving response rates and evaluating nonresponse bias in surveys: AMEE Guide No. 102. Med Teach. 2016;38(3):217-28.

18. Cook DA, Kuper A, Hatala R, et al. When assessment data are words: Validity evidence for qualitative educational assessments. Acad Med. 2016;91(10):1359-69.

19. Report on residents. 2020. Available at: https://www.aamc.org/data-reports/students-residents/report/report-residents. Accessed April 4, 2023.

20. Emergency medicine defined key index procedure minimums. 2017. Available at: https://www.acgme.org/Portals/0/PFAssets/ProgramResources/EM_Key_Index_Procedure_Minimums_103117.pdf?ver=2017-11-10-130003-693. Accessed April 4, 2023.

21. Rappleye E. 28 States with full practice authority for NPs. Becker’s Hospital Review. 2019. Available at: https://www.beckershospitalreview.com/hospital-physician-relationships/28-states-with-full-practice-authority-for-nps.html?oly_enc_id=4391H9381767E4I. Accessed April 4, 2023.

22. Kim HY. Statistical notes for clinical researchers: assessing normal distribution using skewness and kurtosis. Restor Dent Endod. 2013;38(1):52-4.

23. Saldana J. The Coding Manual for Qualitative Researchers. 2013.

24. O’Brien BC, Harris IB, Beckman TJ, et al. Standards for reporting qualitative research: A synthesis of recommendations. Acad Med. 2014;89(9):1245-51.

25. Martimianakis MA, Michalec B, Lam J, et al. Humanism, the hidden curriculum, and educational reform. Acad Med. 2015;90(11):S5-13.

26. Aintablian H. An open letter to the specialty of emergency medicine. 2021. Available at: https://www.aaemrsa.org/about/position-statements/specialty-of-em. Accessed April 4, 2023.