Author	Affiliation
Ashley C. Rider, MD, MEHP	Stanford University, Department of Emergency Medicine, Palo Alto, California
Sarah R. Williams, MD, MHPE	Stanford University, Department of Emergency Medicine, Palo Alto, California
Vivien Jones, BS	Royal College of Surgeons, Dublin, Ireland
Daniel Rebagliati, BA	Albany Medical College, Albany, New York
Kimberly Schertzer, MD, MS	Stanford University, Department of Emergency Medicine, Palo Alto, California
Michael A. Gisondi, MD	Stanford University, Department of Emergency Medicine, Palo Alto, California
Stefanie S. Sebok-Syer, PhD	Stanford University, Department of Emergency Medicine, Palo Alto, California

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Patient care in the emergency department (ED) depends on highly effective interprofessional teams. ED teams are dynamic, complex to train, and subject to the preparedness of individual team members while caring for critically ill patients. Although team training has been championed by the National Academy of Medicine to reduce adverse events, the fluid nature of ED teams makes such training complex.¹ Additionally, individual team member contributions can influence the readiness of an ED team. Previous research has shown that individual performance and communication failures are substantial contributors to adverse events,^2,3 affecting the interdependent nature of team-based care.^4,5 Therefore, evaluating how well existing team performance assessments are at capturing individual and team-based performance is necessary to ensure accurate measurement of teams under the direst circumstances.

The Agency for Healthcare Research and Quality and the US Department of Defense rigorously developed measures that evaluate teamwork.¹ The most widely used tool for assessing team performance and patient safety is the Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS) Team Performance Observation Tool (TPOT), now with a second version, TPOT 2.0. This 23-item instrument integrates five areas of competence: team leadership; team structure; situation monitoring; mutual support; and communication.⁶ The Team Emergency Assessment Measure (TEAM) is an alternate 12-item tool that also measures team performance, but it was designed specifically for team assessment in the ED setting.⁷ The Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS) is another tool that assesses crisis resource management skills of leader-team member interactions.⁸ Although these tools have some validity evidence,^8–13 the extent to which they reliably and accurately measure team-based performance in various contexts warrants further investigation to understand how to best assess an individual’s contributions to ED team performance.

Further complicating team assessment is the critical role of dyads¹⁴ and the interdependence of individuals within teams.^4,15 Interprofessional members of the ED team are inseparably tied to one another, and often there is no choice whether someone becomes part of the team. In the ED, teams are formed out of necessity to provide acute care for critically ill patients. These circumstances essentially require immediate entrustment among individual team members, which is not always feasible or realistic. Underlying the theory of interdependence is the idea that some pairings of team members will be more effective than others; therefore, identifying key factors that influence team member dynamics is critical.^14,15 This conceptual framing has implications for how ED team-based performance (ie, where teams rapidly form to meet the emergent needs of patients) is assessed.

Due to our incomplete understanding of the important elements that contribute to individual and team performance, we set out to explore the effectiveness of current team-based performance assessment tools in the ED setting. Using an interprofessional, in situ simulation, we aimed to do the following in this study: 1) evaluate the effectiveness of TPOT 2.0, TEAM, and Ottawa GRS team-based assessment tools in the ED setting; and 2) identify characteristics of effective teams that are attributable to individuals and may not be captured within existing team-based assessments.

METHODS

We used mixed-methods case study methodology in the context of a team-based, in situ simulation to explore the effectiveness of team-based assessments and explore the relationship between team dynamics and individuals’ team-based performance.^16,17 We used a constructivist paradigm, which holds that an individual’s perspective is the basis for reality and that multiple, socially constructed realities can exist at once for this research.^18,19 We chose case study methodology to understand the various perspectives of team participants and observers in the context of an ED-based simulation.

All ED nurses, technicians, pharmacists, and postgraduate year (PGY) 2–4 emergency medicine residents within one academic health system were eligible to participate in this study. We excluded PGY-1 residents due to their limited experience leading resuscitations. The study took place in a large, suburban, academic ED at a tertiary care facility. We conducted simulations twice per week during low-volume hours; strict policies for cancellation were followed based on ED volume and patient care needs. The Stanford University Institutional Review Board approved this study (#55327).

Quantitative Study Design and Data Collection

Using convenience sampling, we solicited volunteers to participate in simulations held over a four-month period (January–April 2021). Each simulation included a nurse, a resident, a pharmacist and, in some cases, an ED technician. Attendings were not included to ensure that patient care would not be disrupted. Prior to the simulated case, the 63 participants received a two-minute pre-brief from an in-person facilitator on expectations for the simulation, goals of the session, and confidentiality. We obtained written consent for study participation and video recording of the simulations, and participants could opt out of the study at any time.

We conducted a simulated case of a patient presenting with sepsis and an arrhythmia using the high-fidelity HAL patient simulator (Gaumard Scientific Company Inc, Miami, FL) and equipment props that are typically available in an ED patient room. The ED pharmacist supplied simulated critical care medications for use during the scenario. We recorded the simulation for asynchronous rating. The case was followed by a 10–15 minute debrief with all team members, which was not recorded to protect the psychological safety of participants. After the debrief, the participants completed a self-rating for the entire team using the TeamSTEPPS TPOT 2.0 to increase familiarity with the components of TeamSTEPPS, as well as a brief demographic form that included training year/years of work experience, age, and gender. We omitted items 2d, 5c, and 5d on the TPOT 2.0, as these were not relevant to our study protocol.

We recruited two board-certified emergency physicians from outside institutions to assess the simulation video recordings. The two raters underwent a two-hour training session where they were introduced to the project and the three instruments. The facilitator also reviewed an example case, which the reviewers independently scored and were then calibrated against each other. The raters subsequently watched an example video and deliberated each item on the scoring sheet until they arrived at a consensus. Raters then independently reviewed all recorded simulations for which consent was provided by all team members. Raters assessed team performance with the TPOT 2.0 and TEAM. They assessed leadership by completing the leadership categories on TPOT and TEAM, as well as the Ottawa GRS. Only the TPOT 2.0 assessment by the raters was used for comparative data analysis to use objective third-party ratings rather than the self-assessment from participants.

Qualitative Study Design and Data Collection

We invited all volunteer staff participants via email who completed the simulation component of the study to participate in an individual, semi-structured interview via Zoom (Zoom Video Communications, San Jose, CA). A total of 10 ED staff members volunteered to participate in the semi-structured interview, including five nurses, four pharmacists, and one ED technician, and each participant received a $25 gift card as compensation for their time. We also conducted two focus groups with five resident team leaders. Each session lasted 30–60 minutes. A single female interviewer (AR) conducted all interviews and focus groups with predetermined questions that were then allowed to progress to open dialogue.

Data Analysis – Quantitative

We collected demographic information and calculated measures of central tendency for each group. We also analyzed rater’s average scores and standard deviations for each of the tools. We performed a correlation analysis of the within-rater and between-rater scores on each tool. We also compared team-based leader performance based on the Ottawa GRS with the leadership subset on the TEAM and TPOT 2.0. We generated validity evidence^20,21 for the TPOT 2.0 using content validity, internal structure, and relationship to other variables. Content validity was assessed by examining which performance measures participants thought should be included in an assessment tool. We examined internal structure by assessing correlations between the inter-rater reliability and self vs rater scores. Relationship to other variables was manifested as concurrent validity by comparing the tools. We performed data analysis using IBM SPSS v 27 (SPSS Inc, Chicago, IL) and Microsoft Excel v 16.6 (Microsoft Corporation, Redmond, WA).

Data Analysis – Qualitative

Of the 63 participants, 15 (24%) agreed to the interview. We transcribed and anonymized the interviews using the HIPAA-compliant software TranscribeMe! (TranscribeMe Inc, Oakland, CA) program. Two coders (VJ and DR) who were not involved in either the simulation or interview process underwent qualitative training consisting of pre-reading on thematic analysis and completion of a Dedoose webinar v 9.0.17 (Dedoose, Manhattan Beach, CA) webinar. Coders completed a one-hour training session using an excerpt of a transcript to demonstrate the coding process. A second excerpt was done in real time. The coders were then given five days to code the first transcript. This was reviewed by both coders and other members of the research team to discuss and identify patterns. Coders then read all transcripts prior to starting the first coding round. In accordance with Braun and Clarke’s six phases of analysis,²² after complete read-through of the coded transcripts, coders then generated initial codes on the second review.

After the initial round, two researchers (VJ and DR) discussed and refined all independently created codes. Consensus was achieved with review of each transcript on a unified code list. Two other members of the research team (AR and SW) then reviewed the transcripts and codes to develop themes. Investigator triangulation of themes, with attention to the quantitative findings, was performed by a third member of the research team (SS). Initial code and excerpt to theme categorization resulted in 67% independent agreement between the two secondary reviewers. Coding was then revised, consolidated, and modified based on consensus. Two researchers (AR and SW) performed a round of focused re-coding and theme generation, and a final reviewer (AR) performed the last round of code review and edits within existing themes.

Regarding reflexivity, both coders (DR and VJ) had significant experience with healthcare teams and crisis resource management as prior simulation technicians, but were not employed full-time in the ED. While this limited their context for some of the qualitative analysis, it allowed them to focus on teamwork and leadership features without preconceived notions. The code reviewers (AR and SW) are emergency physicians who practice at the academic health center where the study was conducted. Both code reviewers have been involved in residency program leadership. AR facilitated all the interviews but was blinded to the identity of residents and staff during coding.

RESULTS

Quantitative Analysis

We completed 18 simulations with 63 participants from January–April 2021. Some cases had a pharmacist who had participated in multiple simulations (due to the limited number of clinical pharmacists employed in the ED); otherwise, participants were part of a scenario only once. Participant demographics are listed in Table 1 along with the mean self-rated TPOT 2.0 score. Missing data points were omitted from the analysis.

The descriptive statistics of rater scores on each scenario were on the upper end of the scale for each of the tools. The two raters’ scores clustered high for the five-point TPOT 2.0 (R1 4.90, SD 0.17; R2 4.53, SD 0.27), the four-point TEAM tool (R1 3.89, SD 0.19; R2 3.58, SD 0.39), and the seven-point Ottawa GRS tool (R1 6.6, SD 0.56; R2 6.2, SD 0.54). All three scales were noted to have scores that crowded around the maximum. There were high correlations of total score for a given case reviewed within the same rater, particularly for TEAM and Ottawa GRS. Inter-rater correlations were 0.46, 0.68, and 0.72, respectively, for the TPOT 2.0, Ottawa, and TEAM (Table 2). Year in residency (PGY-2, PGY-3, PGY-4) was not correlated to raters’ scores on each of the tools.

Qualitative Analysis

We identified six themes related to the individual and team-based performance (Table 3), including the following: 1) team member entrustment; 2) interdependent energy; 3) leadership tone; 4) optimal communication; 5) strategic staffing; and 6) simulation empowering team performance.

The concept of entrustment stems from the competency-based medical education literature.²³ In the setting of team-based performances, team member entrustment means trusting that a team member will be able to complete a role-specific task without oversight or specific direction. Such entrustment decisions may need to be made rapidly in the setting of ad hoc ED teams and is critical for building relationships that drive team dynamics.

Within our data, characteristics such as age and gender of team members were not perceived to impact entrustment. Our participants noted that personality and previous experience with someone managing a critically ill patient was important for team member entrustment. In the following quote, one participant comments that witnessing a leader’s ability to manage critically ill patients inspired entrustment in their leadership role. “I don’t think it’s necessarily a number of shifts. What I think it is, it’s severity of cases. So, you might have one shift with someone and just have a killer of a day with ESI [Emergency Severity Index]1s and 2s and watched this person rock it, and you’re like, “Okay, I know they’re on it.” (Participant G, RN) As this team member describes, familiarity was an indicator used by participants to make quick entrustment decisions in the ED setting.

Interdependent energy was described as the influence of confidence and demeanor that an individual team member has during a performance that appeared to alter team dynamics and impact team synergy. Several participants also mentioned the importance of tone-setting for a collaborative environment and finding a balance of humility and confidence, as highlighted by this comment about leadership tone. “I have never worked with that doctor before. But I can tell just by his demeanor and his tone that he knew. He was pretty confident on what was going on. So that made me relax and kind of confident as well.” (Participant F, RN)

Optimal communication was also noted as a key factor. This includes appropriate timing, directed toward a specific individual, execution using a reasonable tone, and facilitation of a shared mental model. Strategic staffing, specifically small teams, was described by participants to optimize performance, with examples such as keeping the noise level low and allowing for direct communication to individuals. Finally, simulation empowering team performance reflects that the simulation was described by participants as a way to practice skills and subsequently reflect upon the experience during an interprofessional team debrief. The session allowed team members to foster relationships, provide feedback, and build entrustment.

DISCUSSION

We used an interprofessional, in situ simulation to evaluate team performance using multiple instruments. A mixed-methods approach allowed us to gather quantitative ratings of performance and qualitatively identify features of optimal interprofessional team performance. We found the two team assessment tools, TPOT 2.0 and TEAM, poorly discriminated when teams were assessed as functioning well together. This leaves little opportunity for capturing individual contributions to team performance for subsets of individuals within the team. Our qualitative findings also suggest that these performance measures do not capture some of the dynamic interdependent team features that drive team functionality.⁵ Moving forward, finding a way to capture dynamic features of team relationship building and interdependence can comprehensively provide a more accurate assessment of team performance.

Our findings suggest that the TPOT 2.0 lacks sufficient validity evidence for use in the ED. The overall clustering of high scores may suggest either strong performers within our sample, items that are too easy, or vague anchor points that made it difficult for raters to discriminate. Alternatively, this tool may not be optimized for differentiating individual performance within high-performing teams. The inter-relater reliability IRR of the TPOT was low at 0.46 (Table 2), which may reflect limited rater agreement and, therefore, reliability of the tool. Finally, we identified several features of team performance that participants felt were not sufficiently captured in the assessment tool, mainly entrustment features related to anticipation and automaticity, leadership tone, and interdependent energy.

Additionally, our qualitative analysis provided insight on the features of team dynamics that may be important for optimizing performance. Entrustment among fellow team members occurs when individuals serving in various interprofessional roles are trusted to function within the scope of their practice. Entrustment in our qualitative analysis was largely driven by strong role competence, anticipation, and automaticity. While competence may come from training and experience, anticipation and automaticity are uniquely important for each member of a rapidly forming ad hoc team in high-stakes situations like the ED. Because every resuscitation is slightly different, automaticity and anticipation cannot be based on an algorithm but rather on pattern-recognition and creation of shared understanding, an innately interdependent process. While anticipation is reflected in the TEAM tool, neither is explicitly represented in the TPOT. Other features that are highly important to ED teams to emphasize in performance tools included interdependent energy and tone of communication.

Situating these findings in the broader literature, ED teams are interdisciplinary action teams that task multiple, highly specialized professionals with a critical situation.²⁴ Fernandez et al proposed a robust model for EM teamwork taxonomy to capture the process as well as the outcome.²⁵ This includes the stages of planning processes, action processes, reflection processes, and supporting mechanisms. According to this model, teams will go back and forth between action processes focused on goals and transition processes that allow for planning. Both stages are highly dependent upon interpersonal factors between team members.

Two of the action processes—“backup behavior” of managing team members’ tasks and “coordination” of the inherently interdependent order of activities—are fundamentally dependent on this described construct of team member entrustment.^25,26 This idea resonates with the concept of collaborative interdependence¹⁵ in which team members come together and leverage the strengths of one another. Entrustment may be a necessary step toward establishing a team’s collaborative interdependence as its absence may lead to a breakdown in team functioning. Our study helped discern team member actions and factors that may contribute to rapid entrustability and guide these action processes, even in ad hoc teams, including demonstration of role competency, automatic fulfillment of duties, and anticipation of next actions. To improve interprofessional team performance assessment, we need more granular resuscitation-specific performance measures that capture team member entrustment,²³ leadership tone, and interdependence.^4,23

Educational Implications

Our finding that postgraduate year (PGY) level did not correlate with team performance scores highlights the challenge of assessing resuscitation leadership due to the interdependent nature of team performance.^5,27–28 In the move toward competency-based education and implementation of Entrustable Professional Activities in the workplace,²⁹ this is critically important. A PGY-2 may, for instance, be falsely assessed as fully entrustable based on the resuscitation of a patient in the clinical setting, when in fact their performance was highly influenced by other experienced team members. This underscores the inherent challenges of resident assessment in the clinical setting, due to the constant interdependent workflows with other team members. We propose that future team assessment skills involve leadership tone and energy as played out in the interdependent workflow of the team. This can only be accurately assessed in the context of interprofessional teams in the workplace through collection of both observations and gathering team member experience of tone, energy, and entrustment.

LIMITATIONS

We performed this study at a single academic institution and, thus, the findings represent the culture and characteristics of that setting. Further research is needed to assess the transferability of our findings to other contexts. The study participants were from a convenience sample, which may limit the generalizability of these results. Furthermore, the nursing staff was noted to be very experienced with a median of eight years in practice; this may have positively influenced team performance and contributed to the high scores we observed across the tools. It is also possible that filming the scenarios may have contributed to a Hawthorne effect. While all participants were offered an opportunity to participate in the qualitative interviews, only a smaller subset did, which limits the transferability of our findings as those choosing to participate may be different than those who did not. Finally, the case used a mannequin instead of a real patient, which offers a blanket of psychological safety that a real clinical scenario does not.

CONCLUSION

This mixed-methods study identified limitations of current tools for assessing team-based performance and offers opportunities for improvement. Future tools assessing team performance should focus on capturing entrustment, leadership tone, and interdependence.

Footnotes

Section Editor: Laura Walker, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Ashley C. Rider, MD MEHP, Stanford University, Department of Emergency Medicine, 900 Welch Rd., Suite #350, Palo Alto, CA 94304. Email: ashrider@stanford.edu
07 / 2024; 25:557 – 564

Submission history: Revision received March 31, 2023; Submitted January 21, 2024; Accepted March 1, 2024

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. This research was generously supported by grant funding from the Emergency Medicine Foundation/Council of Residency Directors in Emergency Medicine Education Scholarship Starter Grant. There are no other conflicts of interest or sources of funding to declare.

REFERENCES

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.