Author	Affiliation
Brendan W. Munzer, MD	University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan
Jeffery Love, MD	Georgetown University Hospital/Washington Hospital Center, Department of Emergency Medicine, Washington, D.C.
Barbara L. Shipman, MA	University of Michigan, Alfred Taubman Health Sciences Library, Ann Arbor, Michigan
Brendan Byrne, MD	University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan; Naval Medical Center Portsmouth, Department of Emergency Medicine, Portsmouth, Virginia
Stephen J. Cico, MD, MEd	Indiana University, Department of Emergency Medicine, Indianapolis, Indiana
Robert Furlong, MD	University of Michigan, Department of Emergency Medicine, Ann Arbor, Michigan
Sorabh Khandelwal, MD	Ohio State University, Department of Emergency Medicine, Columbus, Ohio
Sally A. Santen, MD, PhD	University of Michigan, Department of Learning Health Sciences, Ann Arbor, Michigan

Introduction
Methods
Results
Discussion
Limitations
Conclusion
Supplementary information

INTRODUCTION

Dissemination of educational research evidence is critical to improving medical education and ultimately patient care. One reasonable measure of the impact of a publication is the number of citations a particular work receives. This number is indicative of the dissemination of its results and serves as a measure of the work’s service as a foundation for supporting further research.

Publications and the number of citations also serve as important criteria on which faculty are promoted at some institutions, though other institutions do not use number of citations as a promotional criterion. The number of citations does not solely reflect the work of an individual or team, but also serves as a metric for evaluating the research performed at a departmental, institutional, or even national level.¹ The number of citations of publications may provide one measure by which to determine the impact of work. Additionally, other factors, such as funding, are often used to assess productivity in the research community.

Examination of citations is rarely performed except for the purpose of putting together a promotion package. Azer performed bibliometric analyses evaluating the top-cited articles in medical education;² however, there is currently no literature describing the top-cited education articles within the field of emergency medicine (EM). This type of intentional examination can have a number of benefits. First, when examining which articles have received the most citations, it becomes possible for researchers to more easily familiarize themselves with the landmark articles within a field. Second, it provides researchers with information on which topic areas, journals, and research methods tend to be more highly cited. This indicates not only the quality of the research but also the translational impact of the work.³ Third, as funding for medical education research is difficult, by evaluating the funding sources of highly cited articles, this type of examination allows researchers to identify sources of potential funding.

The objective of this study was to identify the top 25 cited education articles in the EM literature and report on their characteristics, as well as the top 25 cited EM education articles in all other indexed journals. We sought to provide clinicians, educators, and researchers with resources for identifying the highest-impact literature in emergency medical education and a database of options to explore when looking to publish within the field of medical education.

METHODS

Within two distinct searches in the Web of Science index, we identified the top 25 articles related to education and EM. The EM journal search was limited to 19 EM journals (Appendix 1). EM-related journals eligible for inclusion were English-language journals indexed within PubMed. Exclusion criteria for these journals included non-English language journals, prehospital journals, and journals with a non-physician focus. We ran the second search within all other indexed journals, excluding the EM journals that were excluded in the first search in Web of Science.

Inclusion criteria for each individual article were the following: it had to be primarily EM-related or include emergency physician subjects and be relevant to EM education; its content was deemed educational; it had to be published in English and it had to have subjects that included physicians or future physicians. Among the exclusion criteria for individual articles were these: the subjects did not include EM residents or physicians or medical students in an emergency department setting; or research subjects were non-hospital based (such as EMS or community-based teaching).

From May 18-June 2, 2016, the authors used keywords and search tools within the Web of Science database to retrieve the top-cited articles in both categories. The aim of this search was to identify not only the most highly cited education articles published in EM journals but also the most highly cited EM-based articles related to education that been published in other literature. The keywords were for the large part those used by Azer.² (See Appendix 1 for search strategy)

Articles were placed in descending order of number of citations in an Excel spreadsheet. Two of the authors then independently assessed both search categories and applied inclusion and exclusion criteria Inter-rater reliability among assessors for selection of the top-cited EM articles, calculated using Cohen’s kappa, was acceptable (0.69). The top 25 cited articles involving EM and education were identified and placed into a final list (“Top Cited Education Articles in EM Literature” and “Top Cited EM-related Education Articles”).

Articles were assigned for review and divided evenly between the author group. Two authors independently reviewed the full text for each article and recorded the following information: (1) article name; (2) first author; (3) source journal; (4) year of publication; (5) number of citations; funding source (if applicable); (6) journal impact factor; (7) journal discipline; (8) article type; (9) educational content; (10) subjects; and (11) research method (if applicable). Each author-pair discussed the outcomes of this data collection to create a consensus. If any discrepancies arose, a third author evaluated the article in question and provided a tiebreaker. Finally, findings were discussed in conference with all authors.

RESULTS

Table 1 summarizes the 25 top-cited medical education articles in EM journals.^4-28 Table 2 summarizes the 25 top-cited medical education articles involving EM in all other journals.^29-52 Articles are listed in descending order with a rank from 1–25 based upon the number of citations, as found in Web of Science at the time of the search.

With regards to EM journals, the greatest number of articles were classified as articles/reviews. The most frequently cited article was “Deliberate Practice and Acquisition of Expert Performance: A General Overview” by Ericsson, published in Academic Emergency Medicine in 2008 and based on a consensus preconference.¹² It had been cited 266 times. Six articles were research papers and seven were curriculum, four of which included a research methodology. One article was a needs assessment. The topics included simulation; learning theory; ultrasound; assessment; learner retention; and interprofessional education. The top three most-cited articles all exhibited a focus on learning theory. Table 3 summarizes these results.

Articles were published most commonly in Academic Emergency Medicine (n = 18; 72%), Annals of Emergency Medicine (n = 4; 16%); and Resuscitation (n = 2; 8%). The majority of articles (n = 19; 76%) listed no funding, while five articles (20%) received external funding alone. One article received both internal and external funding.

With regards to the articles in other journals, the majority were research papers. There were seven curriculum articles and two articles/reviews. The topics included simulation; professionalism; management practice; ultrasound; assessment; cross-cultural care; error in diagnosis; learner retention; specialty choice; and supervision.

Articles in the second literature search were from a wide variety of journals including Academic Medicine (n = 5; 20%); Medical Education (n = 3; 12%); British Medical Journal (n = 2; 8%); Journal of the American Medical Association; and Pediatrics (n = 2; 8%). The most frequently cited article was “Prospective Analysis of a Rapid Trauma Ultrasound Examination Performed by Emergency Physicians” by Ma, published in Journal of Trauma in 1995 and cited 193 times. Eight articles had no funding (32%), 13 (52%) were funded by external awards, one article (4%) was funded solely through internal grants, while three (12%) received both internal and external funding. Table 4 summarizes these results.

We calculated the Pearson correlation coefficient (r) to determine if the age of the article was correlated to the number of citations received. For the articles in the non-EM journals, there was a negative correlation between the year of publication and the number of citations (r = 0.42), meaning that the more recently published articles were cited less often. For the EM journals, however, this correlation was not seen (r = 0.2). Articles from Table 2 (Non-EM) published in higher impact journals were cited more often (r = 0.46). This was not the case for articles in the EM journal search (r = 0.03).

DISCUSSION

This study identified the top 25 most frequently cited EM education articles in both EM literature as well as the remainder of journals based in the Web of Science index. The findings of this study provide information regarding pertinent trends and topics in EM education, as noted in Table 3 and Table 4, while providing an accessible location to identify some of the highest-impact literature within this field. Additionally, it allows us to take note of the journals in which EM education is most often recognized and published, serving as an historical perspective for those seeking to publish work.

It is apparent that there are trends both with regards to the overall field of EM education, as well as the journals in which these articles are published. Non-EM journals have, on average, a higher impact factor (up to 35 for JAMA), indicating that they have a higher number of cited articles and therefore are likely distributed to a wider audience. It then makes sense that, when appropriate, authors would seek to submit articles to a wider-reaching journal. For instance, the largest number of highly cited articles in both groups of journals was simulation. This suggests that simulation is a topic that has both specialty-specific and wide-reaching interest. The top two cited articles focused on simulation located in non-EM journals (Shapiro et al. and Barsuk et al.) are both more highly cited than the top ranked simulation article in EM journals. An author looking to publish an article involving simulation would therefore need to balance the benefits of publishing within the field of EM, with associated peer recognition, against the benefits of publishing in a journal with higher impact. It is interesting to note that the top two cited articles in this study are located in EM journals. This confounds the idea that a wider audience will provide a greater number of citations overall and is possibly related to specialty association and peer recognition.

Table 1 and Table 2 provide information regarding the type of article, with the finding that the vast majority of articles in the non-EM journals were found to be research articles, while EM journals tended to have more articles/reviews that were highly cited. This suggests that there is a preference in publication for research-driven articles in non-EM journals. In contrast, Ericsson’s theory-based article was significant in EM, as was Croskerrv’s article indicating that publishing a key learning theory paper in EM may also provide a meaningful foundation.^{11, 12}

The most common EM journal in which highly cited medical education articles were published was Academic Emergency Medicine. Medical education articles previously considered for publication in Academic Emergency Medicine will now be directed to submit to the new journal Academic Emergency Medicine Education and Training. It should be noted that because this journal will not be indexed for several years, articles published in this journal would not have been considered for this ranking list.

Within non-EM journals, roughly half were specialty focused (i.e., Pediatrics, Journal of Trauma: Injury, Infection, and Critical Care) and half were general medical journals. Articles included within specialty journals tended to have a focus that was less specific to the specialty of EM. It was of note that some of these articles chose a generalizable topic such as simulation or depression and used a subject population that included EM residents as well as other specialties.

The articles published within non-EM journals had a larger number of authors who received funding, whether internal or external funding, than articles published in EM journals. It should be noted, however, that the top-cited articles in both the EM journals and the non-EM journals did not have any funding. While it can be helpful to have the support that funding provides, this finding suggests that unfunded work is worthwhile and can still be impactful.

One goal of education research is to disseminate educational practices.³ Many articles have been widely disseminated despite not being highly cited. For instance, all EM residencies use the “Standardized Letter of Evaluation” (SLOE), as one way of reviewing potential applicants; however, the paper describing its predecessor, the “Standardized Letter of Recommendation (SLOR) and subsequent SLOE papers would not appear on the top 25 cited list in this article.^{53, 54} This suggests that citation numbers alone do not provide all information regarding the reach of research being performed.

This study provides a repository for some of the most impactful literature in EM medical education. For example, the articles on deliberate practice and cognitive strategies for de-biasing are important foundations for EM education. Additionally, some research articles form the basis for further research and educational development. By collecting these articles in one location, it allows others to discover landmark articles within the field of medical education. It also allows others to identify trends in EM education research, note common funding sources, and advance the field of medical education.

LIMITATIONS

Limitations of this study include the fact that articles were searched in only one database, the Web of Science. It is possible that a search performed in a different database, such as SCOPUS, may have provided additional articles or slightly different search findings. For instance, the EM journals were identified a priori of the search. The Journal of Trauma – Injury Infection and Critical Care was not originally identified in that list but could be considered an EM journal. We chose to leave it in the non-EM list based on the a priori listing of journals. Additionally, we excluded articles and journals if they were not English language. This may have skewed search results to favor a Western viewpoint while neglecting articles that may have had additional global influence. Another limitation was our attempt to define what constitutes education research as related to education and training. This may have added a measure of subjectivity, although our inter-rater kappa was acceptable. One final limitation is that this article did not identify where articles were cited, and the subsequent reach of these articles, as well as self-citations.

As related above, the ability to determine impact based upon citation count alone is difficult as there are widely read articles that are not highly cited. Citation counts do provide a foundation; further research could identify what qualities make an article more likely to be disseminated.

CONCLUSION

This study identified the most frequently cited medical education journals in the field of emergency medicine, published in EM journals as well as all other journals indexed in Web of Science. The results identify impactful articles that are collected in one location, providing a resource to others while identifying trends that may be used to guide emergency medicine educational research and publishing efforts.

Supplementary Information

 

Footnotes

Section Editor: David A. Wald, DO

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Brendan W. Munzer, MD, University of Michigan, Department of Emergency Medicine, 3440 Wexford Ct., Ann Arbor, MI 48108. Email: Bmunzer@med.umich.edu. 1 / 2017; 18:60 – 68

Submission history: Revision received July 8, 2016; Accepted October 27, 2016

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. The authors disclosed none.

REFERENCES

1. Perry M, Hopson L, House JB, et al. Model for Developing Educational Research Productivity: The Medical Education Research Group. West J Emerg Med. 2015;16:947-51.

2. Azer SA. The top-cited articles in medical education: a bibliometric analysis. Acad Med. 2015;90:1147-61.

3. Santen SA, Deiorio NM, Gruppen LD. Medical education research in the context of translational science. Acad Emerg Med. 2012;19:1323-7.

4. Blaivas M, Brannam L, Fernandez E. Short-axis versus long-axis approaches for teaching ultrasound-guided vascular access on a new inanimate model. Acad Emerg Med. 2003;10:1307-11.

5. Bond WF, Lammers RL, Spillane LL, et al. The use of simulation in emergency medicine: a research agenda. Acad Emerg Med. 2007;14:353-63.

6. Burdick WP, Schoffstall J. Observation of emergency medicine residents at the bedside: how often does it happen?. Acad Emerg Med. 1995;2:909-13.

7. Campbell JC, Coben JH, McLoughlin E, et al. An evaluation of a system-change training model to improve emergency department response to battered women. Acad Emerg Med. 2001;8:131-8.

8. Cooper S, Cant R, Porter J, et al. Rating medical emergency teamwork performance: development of the Team Emergency Assessment Measure (TEAM). Resuscitation. 2010;81:446-52.

9. Counselman FL, Sanders A, Slovis CM, et al. The status of bedside ultrasonography training in emergency medicine residency programs. Acad Emerg Med. 2003;10:37-42.

10. Croskerry P. Achieving quality in clinical decision making: cognitive strategies and detection of bias. Acad Emerg Med. 2002;9:1184-204.

11. Croskerry P. Cognitive forcing strategies in clinical decisionmaking. Ann Emerg Med. 2003;41:110-20.

12. Ericsson KA. Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med. 2008;15:988-94.

13. Gisondi MA, Smith-Coggins R, Harter PM, et al. Assessment of resident professionalism using high-fidelity simulation of ethical dilemmas. Acad Emerg Med. 2004;11:931-7.

14. Jabbour M, Osmond MH, Klassen TP. Life support courses: are they effective?. Ann Emerg Med. 1996;28:690-8.

15. Jones AE, Tayal VS, Kline JA. Focused training of emergency medicine residents in goal-directed echocardiography: a prospective study. Acad Emerg Med. 2003;10:1054-8.

16. Kovacs G, Croskerry P. Clinical decision making: an emergency medicine perspective. Acad Emerg Med. 1999;6:947-52.

17. Kuhn GJ. Diagnostic errors. Acad Emerg Med. 2002;9:740-50.

18. Mandavia DP, Aragona J, Chan L, et al. Ultrasound training for emergency physicians–a prospective study. Acad Emerg Med. 2000;7:1008-14.

19. Mateer J, Plummer D, Heller M, et al. Model curriculum for physician training in emergency ultrasonography. Ann Emerg Med. 1994;23:95-102.

20. McLaughlin SA, Doezema D, Sklar DP. Human simulation in emergency medicine training: a model curriculum. Acad Emerg Med. 2002;9:1310-8.

21. Mower WR. Evaluating bias and variability in diagnostic test reports. Ann Emerg Med. 1999;33:85-91.

22. Perkins GD. Simulation in resuscitation training. Resuscitation. 2007;73:202-11.

23. Reznek M, Smith-Coggins R, Howard S, et al. Emergency medicine crisis resource management (EMCRM): pilot study of a simulation-based crisis management course for emergency medicine. Acad Emerg Med. 2003;10:386-9.

24. Rudolph JW, Simon R, Raemer DB, et al. Debriefing as formative assessment: closing performance gaps in medical education. Acad Emerg Med. 2008;15:1010-6.

25. Santora TA, Trooskin SZ, Blank CA, et al. Video assessment of trauma response: adherence to ATLS protocols. Am J Emerg Med. 1996;14:564-9.

26. Small SD, Wuerz RC, Simon R, et al. Demonstration of high-fidelity simulation team training for emergency medicine. Acad Emerg Med. 1999;6:312-23.

27. Swing SR. Assessing the ACGME general competencies: general considerations and assessment methods. Acad Emerg Med. 2002;9:1278-88.

28. Vozenilek J, Huff JS, Reznek M, et al. See one, do one, teach one: advanced technology in medical education. Acad Emerg Med. 2004;11:1149-54.

29. Baernstein A, Fryer-Edwards K. Promoting reflection on professionalism: a comparison trial of educational interventions for medical students. Acad Med. 2003;78:742-7.

30. Baraff LJ. Management of the febrile child: a survey of pediatric and emergency medicine residency directors. Pediatr Infect Dis J. 1991;10:795-800.

31. Barsuk JH, Cohen ER, Feinglass J, et al. Use of simulation-based education to reduce catheter-related bloodstream infections. Arch Intern Med. 2009;169:1420-3.

32. Bond WF, Deitrick LM, Arnold DC, et al. Using simulation to instruct emergency medicine residents in cognitive forcing strategies. Acad Med. 2004;79:438-46.

33. Harvey A, Nathens AB, Bandiera G, et al. Threat and challenge: cognitive appraisal and stress responses in simulated trauma resuscitations. Med Educ. 2010;44:587-94.

34. Hobgood C, Hevia A, Tamayo-Sarver JH, et al. The influence of the causes and contexts of medical errors on emergency medicine residents’ responses to their errors: an exploration. Acad Med. 2005;80:758-64.

35. Isaacson JH, Fleming M, Kraus M, et al. A national survey of training in substance use disorders in residency programs. J Stud Alcohol. 2000;61:912-5.

36. Kendall JL, Hoffenberg SR, Smith RS. History of emergency and critical care ultrasound: the evolution of a new imaging paradigm. Crit Care Med. 2007;35:S126-30.

37. Kennedy TJ, Lingard L, Baker GR, et al. Clinical oversight: conceptualizing the relationship between supervision and safety. J Gen Intern Med. 2007;22:1080-5.

38. Kennedy TJ, Regehr G, Baker GR, et al. Point-of-care assessment of medical trainee competence for independent clinical work. Acad Med. 2008;83:S89-92.

39. Kennedy TJ, Regehr G, Baker GR, et al. Preserving professional credibility: grounded theory study of medical trainees’ requests for clinical support. BMJ. 2009;338:b128.

40. Kennedy TJ, Regehr G, Baker GR, et al. ‘It’s a cultural expectation…’ The pressure on medical trainees to work independently in clinical practice. Med Educ. 2009;43:645-53.

41. Larsen DP, Butler AC, Roediger HL. Repeated testing improves long-term retention relative to repeated study: a randomised controlled trial. Med Educ. 2009;43:1174-81.

42. Ma OJ, Mateer JR, Ogata M, et al. Prospective analysis of a rapid trauma ultrasound examination performed by emergency physicians. J Trauma. 1995;38:879-85.

43. Papp KK, Stoller EP, Sage P, et al. The effects of sleep loss and fatigue on resident-physicians: a multi-institutional, mixed-method study. Acad Med. 2004;79:394-406.

44. Revicki DA, Whitley TW, Gallery ME. Organizational characteristics, perceived work stress, and depression in emergency medicine residents. Behav Med. 1993;19:74-81.

45. Shapiro MJ, Morey JC, Small SD, et al. Simulation based teamwork training for emergency department staff: does it improve clinical team performance when added to an existing didactic teamwork curriculum?. Qual Saf Health Care. 2004;13:417-21.

46. Stiell I, Wells G, Laupacis A, et al. Multicentre trial to introduce the Ottawa ankle rules for use of radiography in acute ankle injuries. Multicentre Ankle Rule Study Group. BMJ. 1995;311:594-7.

47. Thomas EJ, Williams AL, Reichman EF, et al. Team training in the neonatal resuscitation program for interns: teamwork and quality of resuscitations. Pediatrics. 2010;125:539-46.

48. Vaidya NA, Sierles FS, Raida MD, et al. Relationship between specialty choice and medical student temperament and character assessed with Cloninger Inventory. Teach Learn Med. 2004;16:150-6.

49. Wallin CJ, Meurling L, Hedman L, et al. Target-focused medical emergency team training using a human patient simulator: effects on behaviour and attitude. Med Educ. 2007;41:173-80.

50. Weissman JS, Betancourt J, Campbell EG, et al. Resident physicians’ preparedness to provide cross-cultural care. JAMA. 2005;294:1058-67.

51. Weller JM. Simulation in undergraduate medical education: bridging the gap between theory and practice. Med Educ. 2004;38:32-8.

52. Wright RJ, Wright RO, Isaac NE. Response to battered mothers in the pediatric emergency department: a call for an interdisciplinary approach to family violence. Pediatrics. 1997;99:186-92.

53. Keim SM, Rein JA, Chisholm C, et al. A standardized letter of recommendation for residency application. Acad Emerg Med. 1999;6:1141-6.

54. Love JN, Smith J, Weizberg M, et al. Council of Emergency Medicine Residency Directors’ standardized letter of recommendation: the program director’s perspective. Acad Emerg Med. 2014;21:680-7.