Introduction
Methods
Results
Discussion
Limitations
Conclusion
Supplementary information

INTRODUCTION

Breaking bad news (BBN) is considered to be one of the most important, stressful, and challenging responsibilities of a physician.^1-6 Trainees and experienced physicians alike report being uncomfortable with this task, notably due to a lack of prior training.^6-8 For patients, the acknowledgment of this information and their comprehension and perception are of paramount importance to facilitate their psychological adjustment and a long-term quality relationship with medical caregivers.^9-12

The BBN process has changed drastically over the past decades, moving from a paternalistic medical approach to one of greater patient empowerment, which acknowledges the need for information^13-14 and results in a greater awareness and clearer understanding of their diagnosis and prognosis.¹³ Patients prefer to receive individualized, comprehensive information communicated with warmth and honesty.^15-18 Patient and family expectations regarding the exact content of news have been shown to be highly variable,¹³ making it difficult for healthcare professionals to tailor the information to suit each patient.¹⁹

Bad news in an emergency department (ED) may consist in announcing that a relative has been admitted to the ED or in sharing with patients or their families news concerning the need for hospitalization or conditions that might lead to a life-threatening situation sooner or later.²⁰ BBN in the ED is a particular challenge because the patient is generally meeting the emergency physician (EP) for the first time and neither of them enter into the relationship by choice. A recent survey²¹ revealed that 78.1% of BBN occurred without previous contact between the patient and the physician. Moreover, history taking, diagnosis, and the acknowledgment of bad news are usually accomplished within a very short time frame²² during which the physician is confronted with distractions, stress, or time constraints.²³

EP training in communication skills to notify family members of a patient’s death has been reported to be poor at best,²⁴ leading medical students, residents and young physicians to adopt inappropriate communication behaviors,^4,25 which in turn significantly increase their stress levels.² Inappropriate communication behavior does not take into account the needs of patients or their families. Several guidelines have been developed in oncology to help physicians deliver bad news.^4,26-31 One of the most widespread BBN protocols is the SPIKES (Setting, Perception, Invitation, Knowledge, Emotions and Summary) protocol.²⁸

BBN training in the ED has scarcely been studied to date.³² The studies undertaken have included a limited number of participants,^33-34 no validated assessment tools³⁵ or control group,^34,35 or were limited to death notification only.^21,24 In this study, we assessed the effects of incorporating a four-hour ED BBN simulation-based training (BBNSBT) on self-efficacy, the BBN process, and communication skills among medical students and junior residents who rotated in the ED. We hypothesized that BBNSBT has the potential to increase self-efficacy, the BBN process, and communication skills.

Population Health Research Capsule

What do we already know about this issue?

A lack of training in breaking bad news (BBN) leads to inappropriate communication behaviors. BBN protocol has been developed to help physicians to deliver bad news.

What was the research question?

What are the effects of a role-play on the BBN process among students who rotated in the emergency department?

What was the major finding of the study?

A short BBN simulation-training has the potential to improve the BBN process and communication skills.

How does this improve population health?

A four-hour, simulation-based training is a good way for trainees to master BBN and better inform the patients and their families on diagnosis and prognosis

METHODS

The ethics committee approved the study (reference number 2015/235). Only authorized individuals had access to the data and materials. The researchers did not participate in the training program.

Training Program

Control Group

The control group (CG) followed the traditional 160-hour ED rotation. Trainees cared for ED patients under the supervision of EPs. The CG did not receive any formal BBN training.

Training Group

The training group (TG) received a traditional 156-hour ED rotation and four hours of BBNSBT.

BBNSBT

Participants were split into small groups up to six members. The BBNSBT involved two components: 1) a one-hour theoretical course on BBN, SPIKES and communication skills with a 15-minute video illustrating SPIKES components; and 2) a three-hour simulation including six role-plays. Three participants were included in each role-play (one playing the physician and two playing family members) while the three other participants watched the simulation. Each one took 10–15 minutes plus 20–25 minutes for a debriefing. The debriefings followed the framework for Promoting Excellence and Reflective Learning in Simulation, using the advocacy-inquiry technique.^36-40 The debriefings focused on the SPIKES protocol and effective communication behaviors.

The following steps ensured the consistency of the BBNSBT: 1) the International Nursing Association for Clinical and Simulation Learning (INACSL) Standards of Best Practice for Simulation^SM41,42 were used to design the BBNSBT; 2) six experts including psychologists, EPs, and simulation instructors validated the scenarios and simulation design; 3) the same facilitators, a psychologist and an EP trained in BBN and certified as basic simulation instructors conducted training; 4) PowerPoint slides with major theory points accompanied the theoretical part of the BBNSBT; and 5) prewritten scripts were used for the role-play explanations and the debriefings.

Recruitment

Medical students and first-year residents specializing in emergency medicine (EM) who had recently graduated were included in the study for one academic year, between September 2017–June 2018. A convenience sample was invited to participate in the study. It included medical students (n = 64) following a one-month ED internship and first-year EM residents (n = 9) beginning their first month of internship. Each participant gave his or her signed informed consent on a voluntary basis. Five students did not complete the rotation and were excluded from the study; therefore, a total of 68 participants were included. The TG and the CG had, respectively, 37 and 31 members.

Study Design

The feasibility study used cluster randomization to reduce contamination bias.⁴³ Each month, a group of 10 to 12 medical students and first-year EM residents was randomly assigned either to the TG or the CG. Demographic data such as gender, age, BBN experience, and study year were collected. During the first week of internship, participants underwent a pre-test of their ED BBN self-efficacy and skills. During the second week, participants assigned to the TG participated in the BBNSBT. Post-testing took place four weeks later and included a self-efficacy and ED BBN assessment (Figure 1).

Assessment Tools

Self-efficacy

We assessed the BBN self-efficacy of participants using a seven-item questionnaire (supplemental material 1 specifically developed for this study, corresponding to seven skills (eg, “To manage your nonverbal communication during the BBN”). Participants rated each item on a Likert scale from 1 (“not at all”) to 5 (“entirely”) in three separate areas: knowledge about the skill; ability to manage the skill; and applying the skill in practice. The experts placed the content validity index of the questionnaire at 0.92.⁴⁴

BBN Skills Assessment

BBN skills were assessed in simulation exercises involving two standardized family members played by actors. A randomly selected BBN scenario was used to assess each participant in both pre- and post-test. The scenarios were as follows: 1) a life-threatening situation after a motorcycle accident; 2) a life-threatening cardiogenic shock; and 3) brain damage after a fight. Each trainee performed in one random scenario. The scenarios for the pre-test and the post-test were different in order to avoid memorization bias. The BBN skills assessments were video recorded and anonymized.

Two blinded raters assessed participants by using two assessment tools. The SPIKES competence form,²⁸ with 14 items, assessed the participants’ compliance with the SPIKES protocol. Each item was scored as “yes” or “no,” resulting in an overall score (range 0–14). The experts determined a cut-off score using the modified Angoff method.⁴⁵ A passing score was 11 and above, and a failing score was below 11. We used the modified Breaking Bad News Assessment Schedule (mBAS) to evaluate communication.⁴⁶ Rather than allocating points proportionally according to the results obtained, the mBAS is reversed, going from 1 (very good) to 5 (very poor). Overall scores ranged from 5–25. A passing score of 14 or lower was also set by the experts. A failing score was above 14.

Assessments were made in two rounds. In the first round, raters independently rated the video. If items were adjacent raw disagreements between raters (more than a one-point difference), they watched the video together, discussed it, and scored it again.

Statistical Analysis

The investigators entered the data collected into the R software, version 3.4.1 (the R Foundation). The statistician used SAS version 9.4 (Cary, NC). We compared the homogeneity of the CG and of the TG at pre-test with χ2 test and Fisher’s exact test for qualitative variables and with the Mann-Whitney U test for quantitative parameters.

A generalized linear mixed model⁴⁷ (GLMM) measured changes before and after the BBNSBT in self-efficacy, the SPIKES competence form and the mBAS. We adjusted the effects of time, group, and group-by-time by the study year as a confounding factor. GLMMs were performed with a covariance matrix of the compound symmetry type.⁴⁸ We performed the McNemar’s test to compare the proportion of students who passed the SPIKES competence form and the mBAS cut-offs between pre-test and post-test within the groups.

Furthermore, two further analyses were considered. First, we calculated the relative gains between pre-and post-test within the two groups by means of the following formula: [(post-test – pre-test) / pre-test]. A Mann-Withney U test was used to compare relative gains. Second, we tested whether the BBNSBT could help fill the performance gap between participants with limited clinical experience (less than one year) in the TG and participants with clinical experience (more than one year) in the CG by means of a Mann-Whitney U test. Results were considered statistically significant at the 5% critical level (p< 0.05).

RESULTS

Participants’ Sociodemographic Data

Table 1 summarizes the sociodemographic parameters for gender, age, BBN experience, study year, training before BBN simulation training, and pre-test assessment scores. No statistically significant difference was found between the TG and the CG for gender (p = 0.24) and BBN experience (p = 0.44). No participants had attended a communication skills training workshop before the BBNSBT. A statistically significant difference was found for study years (p<0.001). In the CG, participants were predominantly in the third or fourth year of medical school whereas in the TG, they were predominantly in their second year. There was also a statistically significant difference in mean age (p = 0.02), although all students were between 22 and 26 years old.

Pre-test Assessment

Pre-test assessment results are reported in Table 1. We found no statistically significant difference between the TG and the CG with regard to self-efficacy (p = 0.74). However, at baseline the CG had better scores in the SPIKES Competence Form (p = 0.03) and for BBN skills according to the mBAS (p = 0.02).

Post-test Assessment

Table 2 presents the results at pre-test and post-test for each group, time effect, group effect, and group-by-time effect. These effects were adjusted by study year. There was a significant group-by-time effect of the training on participants’ self-efficacy (p<0.001). Self-efficacy improved significantly over time, with a 55% enhancement for the TG (p<0.001), while it fell slightly in the CG (2.6% reduction; p = 0.5) (Table 3). The difference between these gains in the two groups was highly significant (p<0.001).

A significant group-by-time effect (p<0.001) of the training on the BBN process was also found for the SPIKES competence form. There was a 33.3% improvement (p<0.001) between pre-test and post-test in the TG while there was no significant gain for the CG. The difference between these gains in the groups was statistically significant (p<0.001). With regard to the measurement of communication skills with the mBAS during BBN, we found a significant group-by-time effect (p<0.001) of the training on communication skills. There was a 23.53% reduction in the non-effective communication skills of the TG participants, but 0% in the CG. The difference between the groups was statistically significant (p<0.001).

Cut-off Scores

Table 4 shows the proportion of students deemed competent when cut-off scores were applied to the SPIKES and the mBAS between groups at pre-test and post-test, as well as between the times within the same group. There was no statistically significant difference at pre-test between the CG and TG for the level of either SPIKES-competent students (CG 15/31, 48.4%; and TG 11/39, 29.7%) or mBAS-competent students (CG 10/31, 32.3%; and TG 4/37, 10.8).

At post-test, we found a statistically significant (p = 0.02) difference for the SPIKES cut-off score: the TG had a higher number of participants passing the cut-off score (27 students passed; 73.0%) than the CG (14 passed; 45.2%). More TG students (23 of the 37; 62.2%) passed the mBAS cut-off score than CG students (11 of the 31; 35.5%), but without a statistically significant difference (p = 0.07). While there was a statistically significant improvement in pre- and post-test scores for the TG in SPIKES and the mBAS (p<0.001 for both), we didn’t find a significant change in the CG scores (SPIKES: p = 0.92; mBAS: p=0.74).

Clinically Inexperienced Participants Benefiting from Simulation Vs Clinically Experienced with No Simulation Sessions

Further analysis compared participants with limited clinical experience (less than one year) in the TG (n=25) and participants with clinical experience (more than one year) in the CG (n=23) (Table 5). We found no statistically significant difference between subgroups with respect to perceived self-efficacy in pre-test (p=0.13). In post-test, the difference between the two subgroups was highly significant (p<0.001). The self-efficacy of students with limited clinical experience benefiting from simulation was higher than in the more experienced group. While BBN skills were statistically higher in students with clinical experience in the CG at pre-test, (p=0.049), we observed no differences post-test in the two subgroups (p=0.34). The results showed that participants with limited clinical experience made up the difference. Analyses showed the same for communication skills during BBN.

DISCUSSION

Medical educators aim to identify the best methods to prepare students for clinical practice. Traditional training is the common pedagogical method for learning clinical skills.⁴⁹ Trainees rarely learn BBN in real clinical practice due to the paucity of opportunities^32,50 and the fact that clinical preceptors are rarely available to give feedback.^3,6,32,50 At pre-test, our study shows a low level of participant experience and a lack of BBN skills, especially in the TG. Chiniara et al.⁵¹ define the “simulation zone” as areas in which simulation education may be better suited than other methods. BBN is an example of the HALO quadrant: high impact on the patient and low opportunity to practice.

This feasibility study assessed the impact of a four-hour ED BBNSBT compared to clinical internship. It was hypothesized that BBNSBT would have the potential to increase participant self-efficacy in BBN communication and management, adherence to BBN stages and processes, and to improve communication skills during BBN. Our results revealed that this training increased self-efficacy perception. Participants had a low level of self-efficacy in pre-test. After the BBNSBT, the TG reported being more confident about their knowledge and application of BBN and about their ability to perform BBN compared to the CG. This confirms the results of another, smaller study (n = 20), which showed an improvement in confidence and self-efficacy.⁵²

These findings may be explained by Bandura’s social cognitive theory,⁵³ which suggests four ways to enhance self-efficacy that we identify in the BBNSBT: 1) enactive attainment (performing the action during the role-play simulation); 2) vicarious experience (observing the video and watching other participants in the role-plays); 3) verbal persuasion (facilitators support the students during the debriefings); and 4) psychological safety during the simulations. Moreover, the perceived self-efficacy of students in the CG with more clinical experience decreased. This result could have different potential explanations, notably that the pre-test may have led to introspection and reflection about their BBN and communication skills.

Communication with patients and their families is one of the Accreditation Council for Graduate Medical Education Milestones for EM residents, specifically the fourth level of BBN.^54,55 Our research used two validated assessment tools that allow for standardization of the evaluation and training. The results demonstrate that BBNSBT using role-playing and debriefing enhances participant BBN learning and performance compared with the traditional learning paradigm and direct immersion in acute clinical situations. BBNSBT offers the opportunity to teach BBN and communication skills to students and young residents in a psychologically safe environment, preventing harm to patients and family members. It allows each participant to announce bad news and observe several BBN simulations with debriefings.

By contrast, in the traditional curriculum role modeling at the bedside could have a negative impact on patients and relatives when medical students or residents engage in inappropriate communication behaviors,^56,57 such as not keeping patients or family members adequately informed or using medical words they do not understand. More students in the TG reached the cut-off scores: 73% for SPIKES and 62.2% for the mBAS vs 45.2% and 35.5% in the CG. These results demonstrate the relevance of BBNSBT in communicating bad news in the ED. However, the difference between the groups for the mBAS cut-off score is not significant. BBNSBT probably focuses more on SPIKES than on communication behavior. It may be necessary to create an advanced course centered on communication skills rather than on SPIKES.

Despite this, BBNSBT offers experiential learning for participants. From the simulation experience, the debriefing process leads students to explore their frames, incorporate new frames such as SPIKES skills, and re-practice these new skills. This process allows knowledge to be acquired through experience.⁵⁶ Moreover, participants had access to ED BBN experts for four hours, which, unfortunately, is unlikely to happen in real clinical practice.

Additional data analyses allowed us to address a new question: Is BBNSBT more useful for students with less than one year of clinical experience? We found a statistically significant difference in the pre-test. Students with limited clinical experience reached the same level of BBN skills as students with more clinical experience after the BBNSBT. The gap between these groups could be filled by simulation training, without the pitfalls of stress and discomfort of direct clinical exposure. No study has previously focused on this question. In fact, BBNSBT used a step–by-step process involving novice participants to bring them to a higher level. The first step involved theoretical explanations given via video, discussions, and lectures. Each simulation, and especially each debriefing, further enhanced the participants’ skills.

One strength of the study is that we paid special attention to the theoretical background upon which the training and evaluation were based, using the widespread SPIKES²⁸ theoretical model and the INACSL Standards of Best Practice for Simulation^SM.^41,42 Moreover, the simulations were well designed, the debriefings were standardized, and the facilitators were trained and experienced. We believe that it is mandatory to meet the INACSL Standards of Best Practice, as well as work with simulation experts to obtain positive results with simulation training.

The next steps for research and pedagogical method improvement can be identified based on these results. Further research is needed to investigate the role of an advanced course in BBN. As BBN is not a required skill for EPs, it would be interesting to investigate whether BBNSBT is feasible and effective in other areas such as obstetrics, intensive care units, etc. Finally, we think that e-learning preparation before BBNSBT, as described for a training on managing low urine output,⁵⁸ could replace some of the in-person time.

LIMITATIONS

According to Kirkpatrick’s four-level training evaluation model,⁵⁹ the self-efficacy and skills assessment used in the simulation are categorized at level 2, which is a low level.⁶⁰ Moreover, we assessed the impact of BBNSBT just after training. Skills transfer to a real clinical setting is not guaranteed and does not allow for any definite conclusion with regard to the actual impact on patients or family members.⁵⁰ Future studies could assess the impact of this training in the workplace and on skills retention over time.⁶⁰ Despite these limitations, the results are very encouraging given that training is only four hours long, a significantly shorter period than other programs previously described.²⁵

While the three scenarios used for the assessments share similarities, they were different before and after the training, as in real life. Cluster randomization resulted in an inequitable distribution of participants. Despite this heterogeneity, statistical analyses adjusted by study year seem to prove that BBN training has an impact on students. This study assessed the impact of a four-hour BBN training, but we cannot be sure that this duration would be more effective than two or six hours. Finally, we did not assess the emotional impact of BBNSBT and BBN assessment on trainees. It would be interesting to know whether BBNSBT elicits a different response than traditional internships.

CONCLUSION

Training programs aspire to produce competent emergency physicians including excellence in the domains of professionalism and communication. According to the EM Milestones, the target for a trainee ready to graduate for “patient-centered communication (ICS1)” specifically includes being able to deliver bad news. The results of this study revealed that a short, simulation-based training with a debriefing session may improve the self-efficacy, BBN skills, and communication skills of medical students and young residents in the ED. Role-playing appears to be an effective and feasible way for trainees to master BBN and acquire patient-centered skills. Further studies should assess the transfer and retention of these skills as well as when to implement the simulation training in the curriculum.

Supplementary Information

Footnotes

Section Editor: Danya Khoujah, MBBS

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Jean-Christophe Servotte, RN, MSc, University of Liège, Département des Sciences de la Santé publique Nutrition, Quartier Hôpital, Avenue Hippocrate, 13, Bâtiment B23, 4000 Liège, Belgium. Email: jcservotte@uliege.be. 11 / 2019; 20:893 – 902

Submission history: Revision received April 17, 2019; Submitted August 6, 2019; Accepted August 7, 2019

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Bragard I, Etienne AM, Libert Y, et al. Efficacy of a communication and stress management training on medical residents’ stress to communicate, self-efficacy and burnout: a randomized controlled study. J Health Psychol. 2010;15(7):1075-81.

2. Brown R, Dunn S, Byrnes K, et al. Doctors’ stress responses and poor communication performance in simulated bad-news consultations. Acad Med. 2009;84(11):1595-1602.

3. Deep KS, Griffith CH, Wilson JF. Communication and decision making about life-sustaining treatment: examining the experiences of resident physicians and seriously-ill hospitalized patients. J Gen Intern Med. 2008;23(11):1877-82.

4. Fallowfield L, Jenkins V. Communicating sad, bad, and difficult news in medicine. Lancet. 2004;363(9405):312-9.

5. Larson DG, Tobin DR. End-of-life conversations: evolving practice and theory. JAMA. 2000;284(12):1573-8.

6. Orgel E, McCarter R, Jacobs S. A failing medical educational model: a self-assessment by physicians at all levels of training of ability and comfort to deliver bad news. J Palliat Med. 2010;13(6):677-83.

7. Barnett MM, Fisher JD, Cooke H, et al. Breaking bad news: consultants’ experience, previous education and views on educational format and timing. Med Educ. 2007;41(10):947-56.

8. Dosanjh S, Barnes J, Bhandari M. Barriers to breaking bad news among medical and surgical residents. Med Educ. 2001;35(3):197-205.

9. Hancock K, Clayton JM, Parker SM, et al. Truth-telling in discussing prognosis in advanced life-timing illnesses: a systematic review. Palliat Med. 2007;21(6):507-17.

10. Campbell TC, Carey EC, Jackson VA, et al. Discussing prognosis balancing hope and realism. Cancer J. 2010;16(5):461-6.

11. Sardell AN, Trierweiler SJ. Disclosing the cancer diagnosis. Procedures that influence patient hopefulness. Cancer. 1993;72(11):3355-65.

12. Last BF, van Veldhuizen AM. Information about diagnosis and prognosis related to anxiety and depression in children with cancer aged 8–16 years. Eur J Cancer. 1996;32(2):290-4.

13. Liénard A, Konings S, Hertay A, et al. Le processus d’annonce d’un diagnostic. Service Public Fédéral Santé Publique, Sécurité de la Chaîne Alimentaire et Environnement. 2016:1-315. Available at: https://www.health.belgium.be/sites/default/files/uploads/fields/fpshealth_theme_file/processusannoncediagnostic.pdf. Accessed August 3, 2019.

14. Garderet L, Ollivier MP, Najman A, et al. L’annonce d’une mauvaise nouvelle. Revue de la littérature. Oncol. 2006;8(12):HS126-31.

15. Dagenais L, Hall N, Majnemer A, et al. Communicating a diagnosis of cerebral palsy: caregiver satisfaction and stress. Pediatr Neurol. 2006;35(6):408-14.

16. Fujimori M, Uchitomi Y. Preferences of cancer patients regarding communication of bad news: a systematic literature review. Jpn J Clin Oncol. 2009;39(4):201-16.

17. Janglang E, Gunningberg L, Carlsson M. Patients’ and relatives’ complaints about encounters and communication in health care: evidence for quality improvement. Patient Educ Couns. 2009;75(2):199-204.

18. Seifart C, Hofmann M, Bär T, et al. Breaking bad news–what patients want and what they get: evaluating the SPIKES protocol in Germany. Ann Oncol. 2014;25(3):707-11.

19. Hagerty RG, Butow PN, Ellis PA, et al. Cancer patient preferences for communication of prognosis in the metastatic setting. J Clin Oncol. 2004;22(9):1721-30.

20. Pastor DK, Cunningham RP, White PH, et al. We have to talk: results of an interprofessional clinical simulation for delivering bad health news in palliative care. Clin Simul Nurs. 2016;12(8):320-7.

21. Toutin-Dias G, Daglius-Dias R, Scalabrini-Neto A. Breaking bad news in the emergency department: a comparative analysis among residents, patients and family members’ perceptions. Eur J Emerg Med. 2018;25(1):71-6.

22. Knopp R, Rosenzweig S, Bernstein E, et al. Physician-patient communication in the emergency department, part 1. Acad Emerg Med. 1996;3(11):1065-9.

23. Park I, Gupta A, Mandani K, et al. Breaking bad news education for emergency medicine residents: a novel training module using simulation with the SPIKES protocol. J Emerg Trauma Shock. 2010;3(4):385-8.

24. Shoenberger JM, Yeghiazarian S, Rios C, et al. Death notification in the emergency department: survivors and physicians. West J Emerg Med. 2013;14(2):181-5.

25. Liénard A, Merckaert I, Libert Y, et al. Is it possible to improve residents breaking bad news skills? A randomised study assessing the efficacy of a communication skills training program. Br J Cancer. 2010;103(2):171-7.

26. Back AL, Arnold RM, Baile WF, et al. Approaching difficult communication tasks in oncology. CA Cancer J Clin. 2005;55(3):164-77.

27. Back AL, Anderson WG, Bunch L, et al. Communication about cancer near the end of life. Cancer. 2008;113(7):1897-1910.

28. Baile WF, Buckman R, Lenzi R, et al. SPIKES-A six-step protocol for delivering bad news: Application to the patient with cancer. Oncologist. 2000;5(4):302-11.

29. Bragard I, Razavi D, Marchal S, et al. Teaching communication and stress management skills to junior physicians dealing with cancer patients: a Belgian interuniversity curriculum. Support Care Cancer. 2006;14(5):454-61.

30. Brown RF, Butow PN, Dunn SM, et al. Promoting patient participation and shortening cancer consultations: a randomised trial. Br J Cancer. 2001;85(9):1273-9.

31. Harrison ME, Walling A. What do we know about giving bad news? A review. Clin Pediatr (Phila). 2010;49(7):619-26.

32. Lamba S, Pound A, Rella JG, et al. Emergency medicine resident education in palliative care: a needs assessment. J Palliat Med. 2012;15(5):516-20.

33. Quest TE, Otsuki JA, Banja J, et al. The use of standardized patients within a procedural competency model to teach death disclosure. Acad Emerg Med. 2002;9(11):1326-33.

34. Chumpitazi CE, Rees CA, Chumpitazi BP, et al. Creation and assessment of a bad news delivery simulation curriculum for pediatric emergency medicine fellows. Cureus. 2016;8(5):e595.

35. Benenson RS, Pollack ML. Evaluation of emergency medicine resident death notification skills by direct observation. Acad Emerg Med. 2003;10(3):219-23.

36. Eppich W, Cheng A. Promoting Excellence and Reflective Learning in Simulation (PEARLS). Development and rationale for a blended approach to health care simulation debriefing. Simul Healthc. 2015;10(2):106-15.

37. Rudolph JW, Raemer DB, Simon R. Establishing a safe container for learning in simulation. Simul Healthc. 2014;9(6):339-49.

38. Szyld D, Rudolph JW. Debriefing with Good Judgment. The Comprehensive Textbook of Healthcare Simulation. 2013:85-93.

39. Rudolph JW, Simon R, Rivard P, et al. Debriefing with good judgment: combining rigorous feedback with genuine inquiry. Anesthesiol Clin. 2007;25(2):361-76.

40. Rudolph JW, Simon R, Dufresne RL, et al. There’s no such thing as “nonjudgmental” debriefing: a theory and method for debriefing with good judgment. Simul Healthc. 2006;1(1):49-55.

41. Lioce L, Meakim CH, Fey MK, et al. Standards of Best Practice: Simulation Standard IX: simulation design. Clin Simul Nurs. 2015;11:309-15.

42. Standards of Best Practice: Simulation. Clin Simul Nurs. 2013;9(6S):S1-32.

43. Dreyhaupt J, Mayer B, Keis O, et al. Cluster-randomized studies in educational research: principles and methodological aspects. GMS J Med Educ. 2017;34(2).

44. Shelestak D, Voshall B. Examining validity, fidelity, and reliability of human patient simulation. Clin Simul Nurs. 2014;10(5):e257-60.

45. Verhoeven BH, van der Steeg AF, Scherpbier AJ, et al. Reliability and credibility of an angoff standard setting procedure in progress testing using recent graduates as judges. Med Educ. 1999;33(11):832-7.

46. Wand S, Schildmann J, Burchardi N, et al. The “bad news consultation assessment scale” (Aufklärungsgesprächbewertungsskala, AGBS): a tool for assessing communication competencies when breaking bad news to cancer patients. Z Arztl Fortbild Qualitatssich. 2007;101(10):645-51.

47. Littell R, Henry P, Ammerman C. Statistical analysis of repeated measures data using SAS procedures. J Anim Sci. 1998;76(4):1216-31.

48. Wolfinger R. Covariance structure selection in general mixed models. Commun Stat Simul Comput. 1993;22(4):1079-106.

49. Steinert Y, Mann K, Centeno A, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach. 2006;28(6):497-526.

50. Jacques AP, Adkins EJ, Knepel S, et al. Educating the delivery of bad news in medicine: preceptorship versus simulation. Int J Crit Illn Inj Sci. 2011;1(2):121-4.

51. Chiniara G, Cole G, Brisbin K, et al. Simulation in healthcare: a taxonomy and a conceptual framework for instructional design and media selection. Med Teach. 2013;35(8):e1380-95.

52. Hobgood C, Harward D, Newton K, et al. The educational intervention “GRIEV_ING” improves the death notification skills of residents. Acad Emerg Med. 2005;12(4):296-301.

53. Bandura A. Social Foundations of Thought and Action: A Social Cognitive Theory. 1986:1-617.

54. Beeson MS, Carter WA, Christopher TA, et al. The development of the emergency medicine milestones. Acad Emerg Med. 2013;20(7):724-9.

55. The emergency medicine milestone project. The Accreditation Council for Graduate Medical Education and The American Board of Emergency Medicine. Available at: https://www.acgme.org/Portals/0/PDFs/Milestones/EmergencyMedicineMilestones.pdf. Accessed July 7, 2019.

56. Downar J, McNaughton N, Abdelhalim T, et al. Standardized patient simulation versus didactic teaching alone for improving residents’ communication skills when discussing goals of care and resuscitation: A randomized controlled trial. Palliat Med. 2017;31(2):130-9.

57. Roter DL, Hall JA, Kern DE, et al. Improving physicians interviewing skills and reducing patients emotional distress – a randomized clinical-trial. Arch Intern Med. 1995;155(17):1877-84.

58. Szyld D, Uquillas K, Green BR, et al. Improving the clinical skills performance of graduating medical students using “WISE OnCall,” a multimedia educational module. Simul Healthc. 2017;12(6):385-92.

59. Kirkpatrick DL. Evaluating Training Programs: The Four Levels. 1994:1-229.

60. Adamson KA, Kardong-Edgren S, Willhaus J. An updated review of published simulation evaluation instruments. Clin Simul Nurs. 2013;9(9):e393-400.