| Author | Affiliation |
|---|---|
| Matthew M. Hall, MD | Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts |
| Jason J. Lewis, MD | Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts |
| Joshua W. Joseph, MD, MD, MBE | Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts |
| Andrew R. Ketterer, MD | Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts |
| Carlo L. Rosen, MD | Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts |
| Nicole M. Dubosh, MD | Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts |
Background
Objectives
Results
Impact
Limitations
Conclusion
Supplementary information
ABSTRACT
The Standardized Video Interview (SVI) was developed by the Association of American Medical Colleges to assess professionalism, communication, and interpersonal skills of residency applicants. How SVI scores compare with other measures of these competencies is unknown. The goal of this study was to determine whether there is a correlation between the SVI score and both faculty and patient ratings of these competencies in emergency medicine (EM) applicants. This was a retrospective analysis of a prospectively collected dataset of medical students. Students enrolled in the fourth-year EM clerkship at our institution and who applied to the EM residency Match were included. We collected faculty ratings of the students’ professionalism and patient care/communication abilities as well as patient ratings using the Communication Assessment Tool (CAT) from the clerkship evaluation forms. Following completion of the clerkship, students applying to EM were asked to voluntarily provide their SVI score to the study authors for research purposes. We compared SVI scores with the students’ faculty and patient scores using Spearman’s rank correlation. Of the 43 students from the EM clerkship who applied in EM during the 2017–2018 and 2018–2019 application cycles, 36 provided their SVI scores. All 36 had faculty evaluations and 32 had CAT scores available. We found that SVI scores did not correlate with faculty ratings of professionalism (rho = 0.09, p = 0.13), faculty assessment of patient care/communication (rho = 0.12, p = 0.04), or CAT scores (rho = 0.11, p = 0.06). Further studies are needed to validate the SVI and determine whether it is indeed a predictor of these competencies in residency.
BACKGROUND
In 2017, the Association of American Medical Colleges (AAMC) developed the Standardized Video Interview (SVI) score as an additional way to assess the professionalism, communication, and interpersonal skills of residency applicants.1 The SVI is composed of six questions answered via a video-recorded, computerized interface and centered on two core competencies of the Accreditation Council for Graduate Medical Education (ACGME): knowledge of professional behavior and interpersonal and communication skills.2 Responses are scored by third-party reviewers using a 1–5 point system with a composite score of 6–30 (Appendix A).3 This score was provided in the 2017–2018 and 2018–2019 Electronic Residency Application Service (ERAS) application packets for emergency medicine (EM) residencies.
The AAMC and several leading EM organizations have sought to assess the validity the SVI. The AAMC found that SVI scores did not correlate with United States Medical Licensing Examination scores and speculated that they would add an additional element to the application.4 The decision was made to proceed with a pilot administration during the 2017–2018 application period.5 While the SVI may add additional information to the residency application, it is unclear how it correlates with other measures of professionalism and communication. Previous work has shown that the SVI does not correlate with faculty gestalt of communication and professionalism.6 We sought to investigate whether correlations exists between the SVI and two other measures of these competencies in EM applicants: faculty end-of-shift ratings of patient care/communication and professionalism, and patient ratings of communication skills.
OBJECTIVES
The goal of this study was to determine whether a correlation exists between the SVI and faculty and patient ratings of these competencies in EM applicants. This was a retrospective analysis of a prospectively collected dataset including fourth-year medical students who enrolled in the EM clerkship at our institution and applied to EM residencies in 2017–2018 and 2018–2019. We collected self-reported SVI scores, end-of-shift faculty evaluations on professionalism and patient care/communication, and scores on the Communication Assessment Tool (CAT), a questionnaire assessing communication skills from the patient perspective that has validity evidence.7-9 We compared scores on all three tools using Spearman’s rho. Statistical analyses were performed with Python 3.6 (Python Software Foundation, Fredericksburg, VA).10-11 A p-value of <0.05 with a Bonferroni correction for multiple comparisons was considered statistically significant. This study was determined to be exempt by our institutional review board.
RESULTS
Forty-three students from our EM clerkship applied to EM during the study period. The response rate of SVI scores was 86.7% (36/43). Fifty-eight faculty members completed evaluations. Faculty ratings were available for 36 students, and CAT scores were available for 32 students. Median scores are shown in Table 1. None of the three tools had a normal distribution (p<0.01). SVI scores did not correlate with CAT scores (rho = 0.11, p = 0.06), nor with faculty evaluation of professionalism (rho = 0.09, p=0.13) or patient care/communication (rho = 0.12, p = 0.04). Faculty professionalism and patient care/communication scores were highly correlated (rho = 0.86, p<0.05).
| Characteristics of Medical Students (n = 36) | Number |
| Male | 20 |
| Female | 16 |
| Medical schools represented | 28 |
| Evaluation Scores of Medical Students | Median, IQR (min) |
| Median number of faculty evaluations per student | 9, 8–10 (6) |
| Median number of evaluations completed per attending | 5, 2–8 (1) |
| Median faculty rating: professionalism | 4, 4–5 (2) |
| Median faculty rating: patient care/communication | 4, 4–5 (2) |
| Median CAT score | 69, 66–70 (44) |
| Median SVI score | 20,18–24 (14) |
IQR, interquartile range; CAT, Communication Assessment Tool; SVI, Standardized Video Interview.
Population Health Research Capsule
What do we already know about this issue?
The SVI does not correlate with faculty gestalt of communication and professionalism.
What was the research question?
Is there a correlation between the SVI and faculty and patient ratings of professionalism and communication skills?
What was the major finding of the study?
We found no correlation between the SVI and ratings of professionalism and communication skills.
How does this improve population health?
This project suggests further research into the SVI is warranted before full implementation of this assessment tool.
IMPACT
We found no significant correlation between students’ SVI scores and faculty ratings of professionalism and patient care/communication skills or CAT scores. To the best of our knowledge, this is the first study comparing SVI scores with existing measures of communication and professionalism in the clinical setting.
Assessing communication and professionalism skills is essential in medical training, and the ACGME has identified both as core competencies.12 A recent review demonstrates that EM program directors value strong interpersonal and humanistic qualities in applicants.13 While it is important to understand applicants’ professionalism and communication abilities, there is currently no “gold standard” assessment method. The ACGME suggests multi-source feedback and multiple evaluators for assessing trainees’ competencies.14 While validated tools are still needed, the use of multi-source assessment including patient feedback in the clinical setting has been shown to be successful.15,16 The SVI scenarios are neither real-time clinical scenarios nor interactions with patients, and it is unclear whether an artificial testing environment is the ideal method of evaluating these competencies. The lack of correlation between the SVI and real-time evaluation of patient interactions raises questions about the SVI’s validity. While the SVI is no longer being considered for use in EM, understanding the concerns surrounding its validity is essential if it is to be reconsidered in the future or used in other specialties.
LIMITATIONS
Our study has several limitations. The SVI scores are self-reported; thus, it is possible students did not provide the correct score. We used this methodology given proprietary restrictions regarding the use of ERAS data. Second, as a single-center study with a small sample size, generalizability is limited. Larger studies are needed to confirm these findings. While students worked 14 clinical shifts during their clerkship, the median number of faculty evaluations completed for each student was nine. This faculty response rate may have introduced bias to these scores. While faculty at our institution are offered individualized training by the clerkship directors on completing evaluations, it is possible that not all faculty participated in a training session and inter-rater reliability may be limited.
Additionally, the faculty evaluation tool groups patient care and communication together (Appendix B), and it is possible some faculty may have weighed this domain more heavily on the patient care aspect and not communication. Four of the students’ CAT scores were lost and not included in the analysis; however, there were no demographic differences between these students and the analyzed population, and thus we do not expect this to have skewed the results. Neither the CAT nor our faculty evaluation system has been validated in terms of predicting success in residency; therefore. we cannot draw conclusions about the SVI’s utility at assessing residency success based on our data. However, there is evidence evaluating the validity of similar tools based on direct observation in the clinical setting.17 Finally, the three scoring systems are all based on different scoring scales and comparison across scoring methods is limited.
CONCLUSION
While this was a small pilot study, we found no significant correlation between SVI scores and neither faculty nor patient ratings of communication competencies. This raises concern about the validity of the SVI. Further, larger scale studies are needed to determine the best methods for assessing trainees’ communication skills and professionalism.
Supplementary Information
Footnotes
Section Editor: Jonathan Fisher, MD, MPH
Full text available through open access at http://escholarship.org/uc/uciem_westjem
Address for Correspondence: Matthew M. Hall, MD, Beth Israel Deaconess Medical Center, Department of Emergency Medicine, One Deaconess Road, Rosenberg 2, Boston, MA 02189. Email: mmhall@bidmc.harvard.edu. 1 / 2020; 21:145 – 148
Submission history: Revision received June 10, 2019; Submitted October 25, 2019; Accepted November 13, 2019
Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.
REFERENCES
1. EMSVI announcement to the EM community. 2017. Available at: http://www.saem.org/home/misc/emergency-medicine-standardized-video-interview. Accessed January 5, 2018.
2. Exploring the ACGME core competencies. 2018. Available at: https://knowledgeplus.nejm.org/blog/exploring-acgme-core-competencies/. Accessed February 28, 2018.
3. How the SVI is scored. Available at: https://students-residents.aamc.org/applying-residency/article/how-svi-scored/. Accessed February 26, 2018.
4. AAMC standardized video interview update. 2017. Available at: aamc-orange.global.ssl.fastly.net/production/media/filer_public/63/7f/637f699e-dd03-4ac3-ae29-bff95f0a1bfe/svi-gsa-2017.pdf. Accessed January 5, 2018.
5. Bird S, Blomkalns A, Deiorio NM, et al. Stepping up to the plate: emergency medicine takes a swing at enhancing the residency selection process. AEM Educ Train. 2017;2(1):61-65.
6. Schnapp BH, Ritter D, Kraut AS, et al. Assessing residency applicants’ communication and professionalism: standardized video interview scores compared to faculty gestalt. West J Emerg Med. 2019;20(1):132-7.
7. Makoul G, Krupat E, Chang CH. Measuring patient views of physician communication skills: development and testing of the communication assessment tool. Patient Educ Couns. 2007;67(3):333-42.
8. Stausmire JM, Cashen CP, Myerholtz L, et al. Measuring general surgery residents’ communication skills from the patient’s perspective using the communication assessment tool (CAT). J Surg Educ. 2015;72(1):108-16.
9. Myerholtz L, Simons L, Felix S, et al. Using the communication assessment tool in family medicine residency programs. Fam Med. 2010;42(8):567-73.
10. Oliphant TE. Python for scientific computing. Comput Sci Eng. 2007;9(3):10-20.
11. McKinney W. Data structures for statistical computing in Python. :51-6.
12. ACGME program requirements for graduate medical education in Emergency Medicine. 2017. Available at: https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/110_emergency_medicine_2017-07-01.pdf. Accessed February 27, 2018.
13. Katzung KG, Ankel F, Clark M, et al. What do program directors look for in an applicant?. J Emerg Med. 2019;56(5):e95-101.
14. Advancing educations in interpersonal and communication skills: and educational resource from the ACGME Outcome Project. Available at: http://www.acgme.org/outcome/assess/toolbox.asp. Accessed June 14, 2018.
15. Brinkman WB, Geraghty SR, Lanphear BP, et al. Effect of multi-source feedback on resident communication skills and professionalism: a randomized controlled trial. Arch Pediatr Adolesc Med. 2007;161(1):44-9.
16. Lockyer J, Vioato C, Fidler H. Likelihood of change: a study assessing surgeon use of multisource feedback data. Teach Learn Med. 2003;15(3):168-74.
17. Chan TM, Wallner C, Swoboda TK, et al. Assessing interpersonal and communication skills in emergency medicine. Acad Emerg Med. 2012;19(12):1390-402.