Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Although effective communication with patients is an integral part of the role of all physicians and has gained the spotlight over the last decade, there is no established standard on how it should be taught and assessed during traditional medical training.^1-6 The urgency to address this gap is evident, as the literature indicates that deficiencies in communication skills can lead to higher malpractice rates, patient dissatisfaction, and adverse patient outcomes.^7-9 The Accreditation Council for Graduate Medical Education (ACGME) endorsed “Interpersonal and Communication Skills” and “Professionalism” as two of the six core competencies. These competencies, and newly mandated ACGME Milestones, are challenging to assess in the clinical setting due to varying faculty frames of reference and the influence of factors external to resident performance.¹⁰ Faculty may use themselves, other doctors, or patient outcomes as frames of reference when assessing residents. In addition, faculty report that they often use “gut feeling” or “gestalt” to translate their observations to numerical assessment scores.¹⁰

Standardized patient (SP) encounters with validated tools are an established method of assessing learners¹¹ and may offer a more consistent way to assess residents. The literature supports a correlation between patient surveys and SP-based assessments of learners,¹² as well as the use of SP feedback for training and assessment of residents.^13-16 Using this established method can offer a more reliable assessment of these residency Milestones. This project aimed to pilot an innovative SP-based model to assess the interpersonal communication skills and professionalism Milestones of emergency medicine (EM) residents.

METHODS

In 2005, the University of Illinois-College of Medicine at Chicago (UIC-COM) Clinical Performance Center (CPC) developed an institution-based competency tool to provide resident performance data to program directors (PD). The Communication and Interpersonal Skills Objective Structured Clinical Exam (CIS-OSCE) was administered and analyzed across specialties including internal medicine, family medicine, surgery, pediatrics, neurology, and obstetrics-gynecology.¹⁷ Later, a many-faceted Rasch measurement model was used to further analyze each item on the scale and the results of this analysis were used to create an improved communication rating scale.¹⁸ We used the new Revised Communication and Interpersonal Skills (RUCIS) scale, a four-category behaviorally anchored rating scale (Table).

Using a mapping method, four EM academic residency faculty integrated the Milestones into this existing RUCIS scale. Already being familiar with the Interpersonal and Communication Skills and Professionalism Milestones, the faculty members were given a chance to review the RUCIS Scale (with no modification to the anchors) and the details of the six communication tasks. They were then asked to individually map each of the behaviorally-anchored ratings on the RUCIS scale to a specific Milestone and level. Then the mapping was reviewed as a group and consensus was reached through iterative discussion until consensus was reached among all faculty (Table 1). Eleven of the 13 RUCIS items were successfully mapped to specific levels of the two EM Milestones (Milestone 20 PROF1 and Milestone 22 ICS1). In the 11 items, Levels 1 through 4 of both Milestones were represented, with Milestone 20 measured five times and Milestone 22 measured six times.

In parallel to the Milestone mapping, the communication challenges were identified and developed. The six communication tasks were identified based on the previously developed patient-centered communication competency assessment implemented in the CPC in 2003. The tasks were originally identified based on the communication literature and their salience to clinical practice. As noted by the authors, “they were designed to allow residents to demonstrate their skills across a range of patient ages, genders, and problems”.⁹ For our Milestone assessment initiative, the tasks used were the following: giving bad news; obtaining informed consent; patient education (addressing medication non-compliance); medical error; treatment refusal; and advanced directives. The cases for each of these tasks were either adapted to the EM setting from previous cases used by other specialties or were newly created and validated by EM faculty through iterative review. Each case was designed to present a communication task with an underlying communication challenge. For example, the communication task in the “giving bad news” case was for the resident to appropriately deliver a death notification and the communication challenge was for the resident to address the need for an autopsy of the deceased.

Each SP encounter was 10 minutes, with the SP completing the RUCIS scale immediately following the encounter. This was followed by 10 minutes for SP-to-resident debriefing. Professional actors were trained by an EM faculty member and an experienced SP trainer on each of the six cases. During the rigorous training, the actors reviewed and practiced the standardized scripting of each case and were tested on their accuracy and standardized portrayal of the patient. The SPs completed rater training for the RUCIS scale, which entailed discussing examples of each item and score with the trainers and watching video examples. The SPs were also trained in techniques of providing feedback to the residents according to the CPC protocol. A convenience subset of encounters were observed by an EM faculty member. At each session six residents rotated through the encounters and concluded the half day with an individual survey of their experience and a group debriefing.

Piloting consisted of 18 residents representing all levels of EM residents or combined EM/Internal Medicine residents in the University of Illinois at Chicago Program (seven postgraduate year [PGY]-1 residents, six PGY-2, and five PGY-3). The six cases were new to all resident participants. Residents were assigned a simulation time slot during which they were excused from clinical duties. At the end of the academic year, the data were forwarded to the EM PD for use during the annual evaluation process for individual residents overseen by the Clinical Competency Committee (CCC) meeting. The scores were averaged across the six cases using the mapped Milestone level, and the resident’s level on Milestone 20 and Milestone 22 were reported separately.

This study was included under the Clinical Performance Center Institutional Review Board (or human subjects committee) approval.

RESULTS

As this was intended as a pilot of an innovative Milestone assessment method, the sample size was small and collected data was limited. An individualized score report was provided to the CCC for each resident that included the Milestone score for each of the two Milestones. See Table 2 as an example. The score report was included for faculty to review as part of the resident’s file; but as this was a pilot, it was not incorporated in any specific numerical way into the resident’s overall Milestone score.

Additionally, in the individual survey 94% of residents agreed that verbal feedback from the SP was helpful and 100% of residents felt the cases allowed them to demonstrate their communication and professionalism skills. In the faculty debriefing, residents uniformly agreed the SP encounter and feedback would improve the quality of care for future patients.

DISCUSSION

In this program, essential communication skills were assessed and EM residents received feedback from the SP as well as an EM faculty member in a simulated setting. This paper demonstrates the utilization of an established OSCE method for Milestone assessment that could provide useful, quantitative performance data to a residency CCC. Although the CCC did not use this pilot data in a structured way, there is potential for standardized incorporation of these scores in the future. Using a larger sample size, it would be important to look at the correlation of individual resident OSCE scores with other assessments in their file.¹⁹ Other possibilities would be to correlate resident scores to level of experience and comfort. Although in this pilot variability in resident scores across experience level was noted, the sample size was too small to report any statistically significant correlation data. Also, SPs were able to offer targeted feedback to individual learners and a difficult-to-obtain patient perspective. This method of assessment is reported in the literature,^13,14 but further study is needed to assess resident communication skills Milestone improvement after SP debriefing.

In this pilot, a previously created tool was mapped to the Milestones. Since patient-centered care was the conceptual framework for the RUCIS scale and not the ACGME EM Milestones, it is necessarily limited in directly assessing ACGME EM Milestones. Further work is needed to create a new, targeted assessment tool that can be used in conjunction with the established OSCE methodology to specifically assess interpersonal communication and professional values Milestones. Using the Milestones as a conceptual framework, a behaviorally anchored tool could be created, similar to the CIS-OSCE, to assess specific behaviors as outlined by the ACGME Milestones. This type of tool could provide consistent, reliable, quantitative data to residency PDs and enhance the instruction and assessment of residents throughout their training with the ultimate goal of improving these skills in patient interactions.

LIMITATIONS

As this was a pilot study, there was an anticipated limitation in sufficient data collection to perform definitive quantitative analyses. Residents knew they were scheduled for communication OSCE encounters and may have focused on demonstrating strong communication. Thus, like many standardized assessments these can be best thought of as “maximal performance” assessments, which may not reflect learners’ typical performance or “worst-case” performance.²⁰ Also, as with many OSCEs, it is possible that residents who were scheduled earlier discussed some case content with later-scheduled residents although the scores of later residents did not reflect this to be the case.

Of note, when the Milestones were mapped to the RUCIS scale, level 5 of both Milestones was not represented in the current tool, although it is notable that residents are not expected to reach level 5 during residency training. This may speak to the need to develop specific assessment methods to measure higher level Milestones with OSCE assessments used for early level Milestones. In addition, the scores provided to the CCC reported a number as a continuous variable as opposed to an ordinal variable as required by the Milestone scale, which may have limited their usefulness. In the future, qualitative feedback from the PD or the CCC on the value of the mapped OSCE score could inform score report structure. This program was piloted at one institution, which would limit its generalizability. Due to these limitations, it would be worthwhile to explore creating a new assessment tool with the EM Milestones as the underlying conceptual framework.

CONCLUSION

As competency-based medical education has come to the forefront, there is a need for reliable and valid methods of assessing communication and professionalism skills. This pilot supports the potential use of an established method to conduct a more rigorous assessment of interpersonal communication skills and professional values Milestones of EM residents. Future studies may also compare SP assessment to standard simulation assessment of these skills to further expand the Milestone assessment toolbox.

Footnotes

Section Editor: Douglas Ander, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Samreen Vora, MD, MHAM, Children’s Minnesota, Department of Simulation, 2525 Chicago Ave, Minneapolis, MN 55404. Email: samreen.vora@childrensmn.org. 11 / 2018; 19:1019 – 1023

Submission history: Revision received December 7, 2017; Submitted July 1, 2018; Accepted August 23, 2018

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Wolff M, Macias CG, Garcia W, et al. Patient safety training in pediatric emergency medicine: a national survey of program directors. Acad Emerg Med. 2014;21(7):835-8.

2. Gaiser RR. he teaching of professionalism during residency: why it is failing and a suggestion to improve its success. Anesth Analg. 2009;108(3):948-54.

3. Morgan ER, Winter RJ. Teaching communication skills: an essential part of residency training. Arch Pediatr Adolesc Med. 1996;150(6):638-42.

4. Klein EJ, Jackson JC, Kratz L, et al. Teaching professionalism to residents. Acad Med. 2003;78(1):26-34.

5. Lee AG, Beaver HA, Boldt HC, et al. Teaching and assessing professionalism in ophthalmology residency training programs. Surv Ophthalmol. 2007;52(3):300-14.

6. Hochberg MS, Kalet A, Zabar S, et al. Can professionalism be taught? Encouraging evidence. Am J Surg. 2010;199(1):86-93.

7. Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. A report of the Committee on Quality of Health Care in America, Institute of Medicine. 2000.

8. Commission J. Communication: a critical component in delivering quality care. Joint Commission: The Source. 2005;3:1-10.

9. Cox LM, Logio LS. Patient safety stories: a project utilizing narratives in resident training. Acad Med. 2011;86(11):1473-8.

10. Kogan JR, Conforti L, Bernabeo E, et al. Opening the black box of clinical skills assessment via observation: a conceptual model. Med Educ. 2011;45(10):1048-60.

11. Downing SM, Yudkowsky R. Assessment in health professions education. 2009.

12. Dine CJ, Ruffolo S, Lapin J, et al. Feasibility and validation of real-time patient evaluations of internal medicine interns’ communication and professionalism skills. J Grad Med Educ. 2014;6(1):71-7.

13. Leeper-Majors K, Veale JR, Westbrook , et al. The effect of standardized patient feedback in teaching surgical residents informed consent: results of a pilot study. Curr Surg. 2003;60(6):615-22.

14. Barry CT, Avissar U, Asebrook MI, et al. Use of a standardized patient exercise to assess core competencies during fellowship training. J Grad Med Educ. 2010;2(1):111-7.

15. Doyle Howley L, Martindale J. The efficacy of standardized patient feedback in clinical teaching: a mixed methods analysis. Med Educ Online. 2004;9(1):4356.

16. Turan S, Üner S, Elçin M. The Impact of standardized patients’ feedback on the students’ motivational levels. Procedia Soc Behav Sci. 2009;1(1):9-11.

17. Yudkowsky R, Downing SM, Sandlow LJ. Developing an institution-based assessment of resident communication and interpersonal skills. Acad Med. 2006;81(12):1115-22.

18. Iramaneerat C, Myford CM, Yudkowsky R, et al. Evaluating the effectiveness of rating instruments for a communication skills assessment of medical residents. Adv Health Sci Educ Theory Pract. 2009;14(4):575-94.

19. Duffy FD, Gordon GH, Whelan G, et al. Assessing competence in communication and interpersonal skills: the Kalamazoo II Report. Acad Med. 2004;79(6):495-507.

20. DuBois CL, Sackett PR, Zedeck S, et al. Further exploration of typical and maximum performance criteria: definitional issues, prediction, and white-black differences. J Appl Psychol. 1993;78(2):205.