Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

As self-assessment can be flawed, feedback is a valuable input for physician performance improvement.¹ Conventionally, feedback is obtained from residents’ attending physicians. However, physicians work in close contact with other members of the care team, most prominently the nursing staff. Nurses may have more opportunity to directly observe residents performing patient care, including aspects of patient care that attending physicians do not routinely observe. Additionally, as nurses approach patient care from a different perspective, they may observe and remark on different behaviors and attitudes of the residents with whom they interact.²

Recognizing the importance of performance input from a variety of sources,³ many of the Accreditation Council for Graduate Medical Education (ACGME) Milestones recommend multi-source feedback, which is further supported by a mandate from the Emergency Medicine (EM) Residency Review Committee. Nursing perspective is particularly relevant to several of the EM milestones.⁴ These include the following: 1) effective communication; 2) working effectively as part of a healthcare team; 3) professionalism; and 4) systems-based practice, including the ability to work in interprofessional teams to enhance patient safety and quality.

Previous studies in other specialties have found that nursing assessments of residents are reliable and may provide information that is different from that provided by attendings.^2,5-10 One study showed that nursing assessments of residents mirror patient assessments, unlike attending, peer, or self-assessments.⁷ Another study demonstrated that nurses were able to assess the humanistic qualities of residents, such as respect and integrity.⁶ Nurses may be less lenient than attendings, although still correlated.⁸ Additionally, nursing assessments of interpersonal skills correlated better with faculty measures, whereas assessments of medical knowledge did not correlate as well.¹¹ These studies demonstrate the unique and concordant assessment domains compared to standard faculty assessments.

The literature demonstrates that the nursing perspective is both valid and at times correlated with other forms of assessment. Previous studies have not explored the specific behaviors nurses may observe and upon which they may comment. All of the previous studies included an assessment form with quantitative data points, rather than narrative, qualitative data points. The objective of this study was to examine the nurse’s perspective of resident performance in the emergency department (ED) by a quantitative analysis of assessments and content analysis of narrative comments. This project will contribute to our understanding of nursing narrative feedback to residents.

METHODS

This was a retrospective, cross-sectional, mixed-method study of nursing assessments submitted from July 2010 to October 2013. The setting was an academic, four-year EM program with over 50 residents. Participants were nurses who worked in the ED and completed resident assessments using an online instrument with quantitative and narrative components. The quantitative component asked nurses to score residents on eight items (Table 1), rating them from 1 (unsatisfactory) to 5 (outstanding). Additionally, nurses were invited to provide narrative comments. The assessments were completed online (Medhub^TM) and were not mandatory. Resident leadership periodically spoke at nursing staff meetings to encourage completion of the assessments. The residents receive these de-identified, aggregated, nurse assessments at each mandatory semi-annual review. The institutional review board determined the study exempt.

Population Health Research Capsule

What do we already know about this issue?

Multi-source feedback is important because self- assessment is flawed. Specifically nursing feedback can be meaningful to guide resident behavior.

What was the research question?

What is the nursing perspective on emergency medicine (EM) resident behaviors? This study examined nursing assessments of EM residents.

What was the major finding of the study?

Nurses provided feedback and valued EM residents when they were kind, efficient, team players and communicated well.

How does this improve population health?

Nurses can provide meaningful feedback to residents to help improve patient care and teamwork.

Quantitative analysis included descriptive statistics of 1,506 assessment forms to support the findings of the qualitative analysis. The narrative comments were de-identified and analyzed using content analysis.^12-16 The analysis was informed by the literature on multisource feedback and by the expertise of the coding group (two nurses, four residents, and two faculty, including a qualitative expert). We started with team immersion review of the data. From this, we developed an initial set of codes. Given the different perspectives, coding was then done iteratively as a group over multiple sessions using the constant comparative method of analysis and grouping of data chunks. When the team disagreed on how a comment should be coded, this was resolved through dialogue. We recorded and refined emergent themes. Saturation was achieved, as no new themes emerged after the first 150 comments (assessment questionnaires) were coded. We coded an additional 60 for a total of 210 to ensure no new themes. Themes are presented using the nurse’s written voice.

RESULTS

Qualitative Data

For all comments two themes were determined. The first was whether the comment provided feedback that was positive or negative. Based on content analysis, 50% of comments were positive, 50% were negative, and 10% were coded as both. The second theme was whether feedback from the comment was actionable or not. Actionable comments were those that were specific enough that the resident could conceivably choose to change behavior to act upon the comment. The majority of comments (80%) were determined to be actionable. An example of actionable is “This MD might improve by being better aware of the patient care that is completed by the registered nurse (RN) and the timeline it takes to accomplish some tasks.”

We identified nine additional themes (Table 2). The most common themes were nice/kind, communication with nurses, and work ethic/efficiency (Table 1). Nurses described the residents both in positive and negative behaviors for each of these themes. The following section will describe the most common themes with direct quotations to demonstrate and clarify the theme.

Nurses frequently commented on communication with nurses. This included updating and informing nurses on the plan of care, new orders or tasks, and being responsive to pages. For example, one positive comment mentioned, “He works great with the nurses and keeps them informed of the treatment plan.” On the other hand, nurses noticed when this did not occur: “Does not initiate conversations regarding patient care/updates with staff.” Multiple nurses commented specifically on residents not responding to pages promptly. “He rarely responds to pages. It is very difficult to get in touch with him regarding questions and requests. I usually have to give up and go find him, which can be frustrating.”

Another common theme was the work ethic and efficiency of the resident. Nurses frequently commented on whether residents were able to pick up, evaluate, and disposition patients in a timely manner, as well as the ability to multitask, prioritize, and balance patient load: “Very unorganized. Takes long time to dispo[sition] patients.” Many nurses commented on whether residents completed orders in a timely and efficient fashion, specifically whether they placed all necessary orders at one time or staggered them. “Gives verbal orders but doesn’t follow through with written orders.” “MD should order appropriately per patient condition and relay to RN. There were many instances with Dr. [ ] where RN had to request imaging orders on a critical patient because she failed to write them – putting patient at risk.”

The third theme was nice/kind. Nurses referred to these residents as being generally enjoyable to work with and having qualities such as being approachable, friendly and taking time to address questions and concerns of ED staff. The opposite was being rude, brusque, and unapproachable. Some examples include “Very approachable and great to work with,” and “She is not very pleasant to work with […] and also is very short with the nursing staff.”

Resident judgment included comments on residents’ knowledge base and decision-making capabilities as well as on their procedural skills. Nurses commented on whether residents had the judgment to recognize critical patients and give appropriate guidance to nursing. “I had a patient who was very hypotensive and hypoxic, and he left me with the patient to go to another patient in resus[citation]…he did not give any direction or send in another physician.” The nurses also made comments on residents’ skills such as, “Awesome pt (patient) positioning when it comes to suturing!” and “Does not know how to administer eye meds.”

Nurses are often present as residents communicate with patients and families. The nurses noted whether residents had good bedside manner, developed rapport with patients, and updated patients on the plan of care. In a positive example, a nurse commented, “Dr. [ ] has the ability to communicate with his patient and family in a way that informs, encourages, and teaches [… ] and asks if there is anything more that the patient or family needs.” On the other hand, nurses noticed when residents were not communicating with their patients, such as, “I have been put in situations where my patients have wanted to leave because she had not seen them in hours after ordering exams for them.” As nurses often go through a resident’s discharge instructions with patients, they were able to comment on those as well.

Nurses also commented on whether the resident was a team player. This included how the resident worked with all the staff and whether he or she did tasks outside of the usual job of the doctor to help patient care. “He is a fantastic MD – he helped me start a difficult IV – he was helpful and respectful of my time.” Another positive example included, “He is one of the few MDs who will help a [patient] walk to the restroom or get them a blanket. He genuinely seems to be a team player, and I appreciate the help he has given me in my patient care.” On the other hand, they note when residents are not working well with other members of the team, as evidenced by “does not work with other staff well, just tells them what to do in a strict ‘I am better then you’ attitude,” and another, “Would like the resident to be more of a team player and supportive of the nurses with combative patients.”

Nurses note leadership and confidence. Specific attributes that the nurses commented upon included decisiveness in voicing orders and plan of care, staying calm in difficult situations, and answering questions with certainty. On the other hand, a lack of confidence included those who were anxious, appeared stressed and unsure of themselves, and were not specific in voicing orders and plan of care. This category also included leadership, and when this was commented upon, it was often in the context of running resuscitations. “Dr. [ ] continues to appear/act in a passive manner while working in the resus[citation] bays. He does not direct well or take on a leadership role during critical times.”

There were multiple comments on the manner in which the residents communicated. These were categorized as respect for other team members. For example, “Dr. [ ] is condescending to staff and rolls her eyes constantly,” and “demonstrates too much arrogance. Does not appear like he wants to listen to nursing staff, not important [sic].” Many of these comments were closely tied to comments about the resident being a team player.

A final category included comments about acknowledging nursing clinical judgment. The nurses wanted residents to be open to their suggestions or opinions on patient care and to listen when they expressed concerns. Positive assessments included statements such as “Seems to respect the information that the RN brings to the patient,” and “Able to accept questioning of orders from nursing staff…listens to suggestions when offered.” In contrast, nurses were aware of and commented about residents who did not acknowledge nursing concerns. “Would like it if he would take the nurse’s views, observation into consideration instead of acting solely.” This category also included comments about recognizing the nurses’ patient load, time constraints and having a good understanding of what nurses in the ED are supposed to do.

Quantitative Data

The quantitative results are found in Table 2. Generally, nurses scored physicians above expectation, especially for the categories of “How would you rate this resident’s attitude?” and “Is this resident a team player?” Residents were scored lowest on “How would you rate this resident’s clinical efficiency and ability to maintain patient flow?”

DISCUSSION

The nurse’s voice in assessment of residents provided unique perspectives and feedback for residents. Their comments suggest that nurses note good communication and the relationship between nurses and doctors (kindness). The advantage to the qualitative analysis of the comments is that they provide a deeper understanding of what nurses observe in the behavior of residents. For example, while we may feel that we understand what “efficiency” means, the specific comments help to enrich our understanding (e.g., putting in all orders at one time so that the nurse does not have to duplicate work by redrawing blood or contacting the laboratory).

The quantitative numbers were primarily positive; however, the narrative comments included a number of negative comments. It is possible that for the survey the nurses may have been providing a socially desirable response by scoring the residents highly. While there is some overlap between the narrative comments and the quantitative questions, nurses also provided additional information to the residents through their voice. Further, there is likely the recognition in the comments that there are still areas of improvement for the residents.

When providing feedback, Tekian et al. noted the importance of linking feedback to an action plan.¹⁷ We found that the majority of nursing assessments comments are actionable and can be as simple as entering lab orders at the same time so that nurses do not have to redraw blood, to more thoughtful behaviors such as getting a patient a warm blanket when the resident recognizes that the nurse is currently busy with multiple pressing tasks. There was a consensus among nursing comments in terms of the specific behaviors directly observed in the clinical setting. The most salient qualities that we found nurses to note were the following: 1) placing orders promptly and at the same time; 2) communicating the plan of care directly with nurses; 3) communicating results and plan of care with patients; and 4) responding to nursing concerns and pages in a timely manner.

Additionally, nurses may also identify patterns of physician behavior that could potentially be detrimental to residents’ professional advancement (e.g., speaking to nurses and patients in a condescending tone). The residents may not be aware that this perception or their behavior is negatively affecting others. By highlighting what is important – attitude, teamwork and efficiency – nursing comments could provide stimulus through which residents can inform their own self-assessment and make positive changes. Bringing nurses into the conversation helps physician providers understand domains of performance of which they may not be aware and promotes an interdisciplinary approach to the assessment of residents in the clinical setting that may lead residents to improved self-assessment and team dynamics.

The Joint Commission’s report on sentinel events demonstrated that in the majority of events, issues with communication were one of the major root causes.¹⁸ In a culture of safety, attention is focused on effective teamwork and communication between healthcare providers. Therefore, as nurses provide actionable feedback through their comments provided to residents, they are instructing the residents how to become better members of the team. Residents who can incorporate this feedback may have improved interactions with the team and be able to provide improved patient care. The use of nursing feedback in resident assessment by residency programs also indicates the importance of our nursing partners and their role in patient care and the team.

Conventionally, resident feedback comes predominantly from the attending physicians. They are appropriately situated to assess a resident’s procedural skills and medical knowledge; however, the resident’s learning environment is broader than the attending-resident interaction. To be an effective physician, residents must also display characteristics such as interpersonal and communication skills, professionalism, and systems-based practice.¹⁵ The interactions in which residents display these characteristics may occur more often with other members of the care team as well as with the patients—interactions that attending physicians do not frequently observe. Nursing assessments add color, depth, and context to resident assessments and, when used in conjunction with conventional attending provider feedback, may provide a more holistic picture of a resident’s ability to provide effective patient care. Further studies need to compare the comments and scores of the nurses, faculty, and peers for each resident. In addition, it will be important to examine the design of assessments specifically for the purpose of providing feedback.¹⁷ While this paper examined nursing feedback for residents, it is also important to include patient feedback in the multisource feedback for residents.

Future studies should examine if residents’ reviews of actionable nursing assessments influence a change in their behavior. From what we know about poor self-assessment, the nursing comments should be part of informed self-assessment.^19,20 If residents were to identify nurses as a respected, “trusted source,” nurses could then conceivably help coach those residents with problematic behavior. For example, when a resident does not understand how s/he might be perceived as arrogant, the nursing coach could help provide specific examples and better approaches.

LIMITATIONS

There were several limitations associated with this study. First, the nursing survey contained the eight questions on specific characteristics of physicians before the comment, free-text portion. These specific questions may have influenced the free-text responses. Because we do not have a response rate for nurses completing evaluations there may be some bias. In addition, there may be some social desirability response in the textual comments. Second, this was a single site, which may constrain generalizability. This study did not examine the changes over time for the responses, as the nursing assessments of residents started over a decade ago. Finally, this study is an initial step toward understanding nurses’ feedback to residents; however, there are limitations implicit in our qualitative methods. Qualitative studies are not intended to test inferences about causation or associations.

CONCLUSION

Nurse-physician relationships form the basis of effective interprofessional practice and patient care. Nurses’ comments suggest that they remark on communication and the relationship between nurses and doctors as well as teamwork and efficiency. Nurses’ assessments can provide feedback and direction for resident professional development.

Footnotes

Section Editor: Chris Merritt, MD, MPH

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Sally A. Santen, MD, PhD, Virginia Commonwealth University School of Medicine, Department of Emergency Medicine, Box 980565, VCU SOM, Richmond, VA 23298-0565. Email: sally.santen@vcuhealth.org. 1 / 2019; 20:23 – 28

Submission history: Revision received July 14, 2018; Submitted October 12, 2018; Accepted October 15, 2018

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Davis DA, Mazmanian PE, Fordis M, et al. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296(9):1094-102.

2. Tintinalli JE. Assessment of emergency medicine residents by nurses. Acad Med. 1989;64(1):49-50.

3. Perry M, Linn A, Munzer B, et al. Programmatic assessment in emergency medicine: implementation of best practices. J Grad Med Educ. 2018;10(1):84-90.

4. The Emergency Medicine Milestones Project. 2012. Available at: http://www.acgme.org/acgmeweb/Portals/0/PDFs/Milestones/EmergencyMedicineMilestones.pdf. Accessed November 12, 2013.

5. Linn LS, Oye RK, Cope DW, et al. Use of nonphysician staff to evaluate humanistic behavior of internal medicine residents and faculty members. J Med Educ. 1986;61(11):918-20.

6. Wenrich MD, Carline JD, Giles LM, et al. Ratings of the performances of practicing internists by hospital-based registered nurses. Acad Med. 1993;68(9):680-7.

7. Woolliscroft JO, Howell JD, Patel BP, et al. Resident-patient interactions: the humanistic qualities of internal medicine residents assessed by patients, attending physicians, program supervisors, and nurses. Acad Med. 1994;69(3):216-24.

8. Kaplan CB, Centor RM. The use of nurses to evaluate house officers’ humanistic behavior. J Gen Intern Med. 1990;5(5):410-4.

9. Butterfield P, Mazzaferri E. A new rating form for use by nurses in assessing residents’ humanistic behavior. J Gen Intern Med. 1991;6(2):155-61.

10. Butterfield PS, Mazzaferri EL, Sachs LA. Nurses as evaluators of the humanistic behavior of internal medicine residents. J Med Educ. 1987;62(10):842-9.

11. Davis JD. Comparison of faculty, peer, self, and nurse assessment of obstetrics and gynecology residents. Obstet Gynecol. 2002;99(4):647-51.

12. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis qualitative. Qual Health Res. 2005;15(9):1277-88.

13. Sullivan GM, Sargeant J. Qualities of qualitative research: part I. J Grad Med Educ. 2011;3(4):449-52.

14. Sargeant J. Qualitative research part II: participants, analysis, and quality assurance. J Grad Med Educ. 2012;4(1):1-3.

15. Corbin J, Strauss A. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. 2008.

16. Glaser B, Strauss A. The Discovery of Grounded Theory: Strategies for Qualitative Research. 1967.

17. Tekian A, Watling CJ, Roberts TE, et al. Qualitative and quantitative feedback in the context of competency-based education. Med Teach. 2017;39(12):1245-49.

18. Sentinel event statistics. 2014. Available at: http://www.jointcommission.org/assets/1/18/Root_Causes_by_Event_Type_2004-2014.pdf. Accessed April 20, 2014.

19. Sargeant J, Eva KW, Armson H, et al. Features of assessment learners use to make informed self-assessments of clinical performance. Med Educ. 2011;45(6):636-47.

20. Sargeant J, McNaughton E, Mercer S, et al. Providing feedback: exploring a model (emotion, content, outcomes) for facilitating multisource feedback. Med Teach. 2011;33(9):744-9.