|Abbas Husain, MD||Staten Island University Hospital – Northwell Health, Department of Emergency Medicine, Staten Island, New York|
|Zachary Repanshek, MD||Lewis Katz School of Medicine at Temple University, Department of Emergency Medicine, Philadelphia, Pennsylvania|
|Manpreet Singh, MD||University of California, Los Angeles Medical School, Department of Emergency Medicine, Los Angeles, California|
|Felix Ankel, MD||University of Minnesota Medical School, Department of Emergency Medicine, Minneapolis, Minnesota|
|Jennifer Beck-Esmay,||Mount Sinai St. Luke’s-West, Department of Emergency Medicine, New York, New York|
|Daniel Cabrera, MD||Mayo Clinic College of Medicine, Department of Emergency Medicine, Rochester, Minnesota|
|Teresa M. Chan, MD, MHPE, FRCPC||McMaster University, Department of Medicine, Hamilton, Ontario|
|Robert Cooney, MD, MSMedEd||Geisinger Medical Center, Department of Emergency Medicine, Danville, Pennsylvania|
|Michael Gisondi, MD||Stanford University School of Medicine, Department of Emergency Medicine, Stanford, California|
|Michael Gottlieb, MD||Rush University Medical Center, Department of Emergency Medicine, Chicago, Illinois|
|Jay Khadpe, MD||University of Florida College of Medicine, Department of Emergency Medicine, Jacksonville, Florida|
|Jennifer Repanshek, MD||Lewis Katz School of Medicine at Temple University, Department of Emergency Medicine, Philadelphia, Pennsylvania|
|Jessica Mason, MD||University of San Francisco-Fresno, Department of Emergency Medicine, Fresno, California|
|Dimitrios Papanagnou, MD||Sidney Kimmel Medical College at Thomas Jefferson University, Department of Emergency Medicine, Philadelphia, Pennsylvania|
|Jeff Riddell, MD||University of Southern California Keck School of Medicine, Department of Emergency Medicine, Los Angeles, California|
|N. Seth Trueger, MD||Stanford University School of Medicine, Department of Emergency Medicine, Stanford, California;
Lewis Katz School of Medicine at Temple University, Department of Emergency Medicine, Philadelphia, Pennsylvania
|Fareen Zaver, MD||Rush University Medical Center, Department of Emergency Medicine, Chicago, Illinois;
University of California, Los Angeles Medical School, Department of Emergency Medicine, Los Angeles, California
|Emily Brumfield, MD||Oschner Clinic Foundation, Department of Emergency Medicine, New Orleans, Louisiana|
As scholarship moves into the digital sphere, applicant and promotion and tenure (P&T) committee members lack formal guidance on evaluating the impact of digital scholarly work. The P&T process requires the appraisal of individual scholarly impact in comparison to scholars across institutions and disciplines. As dissemination methods evolve in the digital era, we must adapt traditional P&T processes to include emerging forms of digital scholarship.
We conducted a blended, expert consensus procedure using a nominal group process to create a consensus document at the Council of Emergency Medicine Residency Directors Academic Assembly on April 1, 2019.
We discussed consensus guidelines for evaluation and promotion of digital scholarship with the intent to develop specific, evidence-supported recommendations to P&T committees and applicants. These recommendations included the following: demonstrate scholarship criteria; provide external evidence of impact; and include digital peer-review roles. As traditional scholarship continues to evolve within the digital realm, academic medicine should adapt how that scholarship is evaluated. P&T committees in academic medicine are at the epicenter for supporting this changing paradigm in scholarship.
P&T committees can critically appraise the quality and impact of digital scholarship using specific, validated tools. Applicants for appointment and promotion should highlight and prepare their digital scholarship to specifically address quality, impact, breadth, and relevance. It is our goal to provide specific, timely guidance for both stakeholders to recognize the value of digital scholarship in advancing our field.
The promotion and tenure (P&T) process requires the appraisal of individual scholarly impact in comparison to scholars across institutions and disciplines. Comparative metrics such as the journal impact factor and the h-index are used to quantify and compare the quality of an individual’s scholarship and, therefore, his or her academic merit.1 As knowledge dissemination methods evolve in the digital era, we must adapt traditional P&T processes to include emerging forms of digital scholarship.2 In this paper, we aim to first situate our readers within the literature on the topic of academic scholarship, after which we will describe the process by which we derived and refined our consensus guideline. Finally, we will outline the recommendations for the use of digital scholarship for academic promotion made by this particular guideline group.
The Evolution of Scholarship
Scholarship is persistently dynamic. Analog technologies progressed from tablet and stone to pen and paper; modern digital scholarship is evolving with blogs, podcasts, and digital journals. Still, the standards for evaluation are consistent and focus predominantly on impact and quality of the scholarship.3 In 1990, Ernest Boyer of the Carnegie Foundation originally redefined scholarship for the professoriate as belonging to one of four types.4 A decade later, Charles Glassick followed up this work by describing criteria for evaluating scholarship.5,6 To further develop nuances around the scholarship of teaching and learning, Lee Shulman and Patricia Hutchings further clarified specific criteria for this subtype of scholarship (to differentiate it from high-quality, scholarly, and evidence-based teaching).7 These foundational concepts are summarized in Table 1 below.
Table 1Foundations of education and teaching scholarship.
|Boyer’s Scholarly Domains 1990||Hutchings and Shulman Criteria for Scholarship of Teaching 1999||Glassick’s Criteria for Evaluating Scholarship 2000|
|Scholarship of discovery
Scholarship of integration
Scholarship of application
Scholarship of teaching
Available for peer review & critique according to the standards of a field
Able to be reproduced and extended by other scholars
Traditionally, peer-review processes of academic journals served as a safeguard to ensure overall quality, with evaluators deferring to experts and peers within a scholar’s domain to provide an appraisal for quality and an estimate of impact. Similarly, bibliometrics of journals (eg, journal impact factor)8 and number of citations are surrogates for scholarly reach and proof of impact.3 Despite well-described limitations, these metrics are quantifiable and defined processes that are easily compared. Thus, they are highly relied upon by P&T committees to compare scholars from disparate disciplines. When scholarship using new media is produced, it is reasonable to scrutinize the methodology, content, impact, and quality of these new forms of scholarship, such as digital scholarship. Our use of the term “digital scholarship” in this paper reflects original content that is disseminated digitally, whether that content is research, teaching materials, enduring resources, commentaries, or other scholarly work.
It is unsurprising that as the world becomes more digital, so do scholarly contributions.9 Online-only journals, pre-print archives,10 and post-production, peer-review journals (eg, Cureus) are rapidly changing the landscape of peer-reviewed publication.11,12 Similarly, with the advent of peer-reviewed blogs,13 self-published peer-reviewed books,14 and educational resource repositories, we see an increased breadth of expression from those engaging in the scholarship of teaching. These varied forms increasingly mirror the rigor required by Glassick’s criteria and Shulman’s paradigms.15,16
Quantity vs Quality
Judging these new forms of scholarship is different. In many ways, with advanced web analytics, it is easier to quantify the reach and attention (eg, pageviews, podcast downloads, IP addresses that have accessed the content, and time on page) of these digital assets. (See Table 2 for common analytics available for new digital scholarship.) For example, the PubMed-indexed repository MedEdPortal provides download analytics of the published resources that aid in describing entries as fitting within the scholarship of teaching.
Table 2Summary of metrics used to demonstrate digital scholarship impact, role and quality, with a sample scholarly work.
|Promotion metric||Supporting data||Example with metrics|
|ImpactDemonstration of impact shows your work reaches your intended audience||PageviewsTime Spent on PageLikesImpressionsDissemination (Shares)Unique UsersGeographic ReachFollowers on Professional Social Media AccountsSocial Media IndexDigital Object Identifier (DOI)Alexa RankingAltmetrics||Thoma B, Chan T, Benitez J, Lin M. Educational Scholarship in the Digital Age: A Scoping Review and Analysis of Scholarly Products. The Winnower. 2014. doi:10.15200/winn.141827.77297Pageviews 4137Altmetric Score 61202 tweets from 86 users, with an upper bound of 263,362 followers|
|RoleDemonstration of your “brand” or role within digital scholarship helps establish your area of expertise||EditorAuthorCuratorReviewerInvited CommentariesPodcast Guest or Editor||[Invited Commentary] Berg A, Weston V, Gisondi MA. Journal Club: Coronary CT Angiography Versus Traditional Care. NUEM Blog. http://www.nuemblog.com/blog/cta-for-chest-pain/ Published online 4/12/16.|
|QualityWhile also demonstrating commitment to scientific rigor in your work, you may also highlight novel quality assurance methods unique to digital scholarship.||METRIQ-5 and -8, rMETRIQALiEM AIR ScoreSAEM Online Academic Resources (SOAR)Social Media Index (SMi)The Quality Checklists for Health Professions Blogs and Podcasts||[Peer-reviewed blog] Long, B. “Myths in Heart Failure: Part I – ED Evaluation” emDOCs.net http://www.emdocs.net/myths-in-heart-failure-part-i-ed-evaluation/ published online 7/23/2018.Selected as ALIEM AIR Cardiovascular, Non-ACS module 2019.39 This post was deemed to be of an acceptable score within the ALiEM AIR Scoring tool, and was granted the designation “AIR Approved” by the adjudicating group of educators. There is a second tier below, known as “honorable mention” for posts of moderate quality that did not meet the threshold for inclusion.|
ALiEM, Academic Life in Emergency Medicine; AIR, approved instructional resources; SAEM, Society for Academic Emergency Medicine
However, since many disciplines both within and outside of medicine have not yet fully embraced digital scholarship as enthusiastically as emergency medicine (EM) and critical care,17 it is no surprise that P&T committees do not yet have specific or universal standards for presentation or evaluation of digital scholarship.18 Those without digital scholarship experience may grapple with understanding the nuances of determining impact and quality in this new era, and their lack of understanding may even result in general skepticism of novel products. Thus, fields that have already established robust methods for determining the quality of digital scholarship can lead the way. Since digital scholarship has matured in EM,19,20 it is appropriate for our field to call for the identification of best practices for evaluating digital scholarship and for consensus in the inclusion of such items in promotions decisions.
Specific guidelines for P&T are lacking despite robust digital contributions proliferating among academicians. In this work, we provide a guiding framework for the presentation and evaluation of digital scholarship for the applicant for promotion, referees for the candidate, and members of P&T committees.
We conducted a blended, expert consensus procedure using a nominal group process to create a consensus document.21 Invited participants met at the Council of Emergency Medicine Residency Directors (CORD) Academic Assembly on April 1, 2019 (Seattle, WA), to discuss recommendations for evaluation and promotion of digital scholarship with the intent to develop specific, evidence-supported recommendations to P&T committees and applicants. We began with a live, brainstorming event. The meeting notes were compiled by a leadership team and formatted into a collaborative working document. All authors continued formulating this document via a collaborative online authorship using Google Docs (Google LLC, Mountain View, CA).22
The participants were selected by the leadership of the CORD Social Media and Digital Scholarship Committee (EB, ZR, AH). Participants were selected based on criteria of known interest or scholarship in the area, national and international level contributions to EM digital scholarship, and availability to attend the conference in person or by phone. Supplemental Digital Appendix A lists original invitation list and individual selection rationale. The complete list of attendees of the in-conference proceedings is listed in the acknowledgments.
As a large group, the consensus conference participants democratically developed the discussion and brainstorming procedures. Based on suggestions from the floor about previous consensus procedures at other similar conferences,23,24 our group decided to engage in small- group brainstorming discussions aligned with the expertise and interests of the participants, which was then discussed as a large group and vetted by the rest of the participants. Consensus was defined as universal agreement of the participants.
Ideation and refinement
The participants self-identified their areas of expertise or interest, and then separated into three groups based on these content areas using an iterative process to formulate specific recommendations. The three discussion groups were tasked with formulating recommendations for the following:
The P&T applicant for promotion of one’s digital scholarship;
P&T committee members for evaluation of quality of digital scholarship;
P&T committee members for evaluation of the impact of digital scholarship.
Small groups presented preliminary recommendations to the entire group and made further revisions via iterative discussion. Participants transcribed an outline of the discussion and final recommendations and agreed upon them in a democratic fashion. Participants self-selected areas of the manuscript to prepare based on expertise, interest and group approval. All members developed the manuscript from the outline via collaborative authorship. All participants contributed to the manuscript, and CORD Social Media and Digital Scholarship committee members (AH, MS, ZR, EB) served as final editors of the manuscript.
Recommendations for Presenting Digital Scholarship for Promotion and Tenure
Demonstrate Scholarship Criteria
When presenting digital scholarship to a P&T committee, begin by ensuring and demonstrating that it meets the criteria of scholarship as defined by Glassick and expanded upon by Sherbino and colleagues with regard to social media.25,26 The adapted criteria are as follows: 1) create original content; 2) advance the field of health professions education by building on theory, research or best practice; 3) be archived and disseminated, and 4) provide the health professions education community with the ability to comment on and provide feedback in a transparent fashion that informs wider discussion. In addition, consider providing evidence of archival and dissemination, such as Google Scholar indexing or inclusion of a digital object identifier (DOI).
Provide External Evidence of Impact
Ensure that your digital scholarship is reflected consistently throughout your promotions dossier. Dissemination metrics are important to include as measures of impact. For example, some blog editors will provide information about how many times a post has been accessed and the locations of its readers, if requested for P&T purposes. Such metrics of dissemination and impact should be presented in the dossier as evidence of your professional reputation as a scholar in your field.
Additional metrics include pageviews, downloads, and geographic reach. Other programs assessing the reach of scholarship, such as altmetrics, may also be valuable.27 The Social Media Index is a relatively newer technique to assess the impact of websites and could be used as a surrogate for impact, much the same as a journal’s impact factor.28 See Table 2.
Other measures of impact could include letters of support and awards. If permitted by your institution, consider obtaining letters of support with regard to your digital scholarship. You may also consider inviting both peer letters and letters from non-collaborators discussing the dissemination metrics and impact of specific pieces of scholarship, or simply your overall impact. There are also a number of digital scholarship-based awards, which may be of value for demonstrating scholarly impact.29
Include Digital Peer-review Roles
Include editor or peer-reviewer roles for digital scholarly content in your curriculum vitae (CV) in a similar manner as you would for traditional print literature. It is important to highlight these supporting components of digital scholarship and they should be factored into the P&T decisions.25
Citing Digital Scholarship
Cite scholarly work on your CV using a consistent format, whether that work was published in a hard-copy journal or as digital content. Reorganize the categories of scholarly publications on your CV to include a section for “Digital Scholarship,” which is the appropriate subheading for items such as blog posts, podcasts, and videos. See Table 3 below for example subheadings for the scholarly bibliography of your CV. Include only those items that reflect true scholarship and relate to the health professions or sciences. Do not list citations for personal website posts or other digital content that is unrelated to your academic position.
Table 3Subheadings for “Scholarly Bibliography” section of curriculum vitae.
|1||Original Research Articles – Peer Reviewed|
|2||Editorials, Reviews, Case Reports, Letters, Commentaries – Peer Reviewed|
|3||Textbooks, Textbook Chapters|
|4||Proceedings and Non-Refereed Papers|
|7||Exhibits, Audiovisuals, Teaching Materials|
|8||Media Appearances and Quotes – Print, Television, Online|
Consistently format your scholarship across all subheadings on your CV following the American Medical Association (AMA) Manual of Style, 10th Ed.30 The AMA Manual describes the methods for citing scholarship in most of the categories listed. Examples of each citation type are provided above, and selected citations are adapted in Table 4. Digital scholarship is best formatted using the AMA Manual instructions for “Internet Documents.”33 Academic Life in Emergency Medicine (ALiEM) also offers guidelines for citing digital scholarship, with examples.31
Table 4Suggested examples of digital scholarship citations and qualifier use.
|Last Name, First Initial. “Title of Submission.” Name of Publisher. URL as hyperlink. Published online XX/XX/XX.|
|Gisondi MA, Stefanac L. “The Feedback Formula: Part 1, Giving Feedback.” International Clinician Educators Blog. https://icenetblog.royalcollege.ca/2018/10/02/the-feedback-formula-part-1-giving-feedback/. Published online 10/02/18.|
|Example Qualifiers for Curriculum Vita:|
|[Blog Post] Gisondi MA. “Leadership in Medical Education: Addressing Sexual Harassment in Science and Medicine.”International Clinician Educators Blog. https://icenetblog.royalcollege.ca/2019/01/15/leadership-in-medical-education-addressing-sexual-harassment-in-science-and-medicine/ Published online 1/15/19.|
|[Podcast Guest] Kellogg A, Gisondi MA. “Sex and Why Episode 10: How to Give Better Feedback.” In: seX & whY Podcast (Wolfe J, Editor-in-Chief.) https://www.sexandwhy.com/sex-why-episode-10-how-to-give-better-feedback/ Published online 1/29/19.|
|[Peer-Reviewed] Schnapp B, Fant A, Powell E, Richards C, Gisondi M. “8 Tips for How-to-Run an Awesome Works-in-Progress Meeting.”Academic Life in Emergency Medicine. http://www.aliem.com/8-tips-works-progress-meeting/ Published online 11/1/15.|
|[Commentary, Invited] Berg A, Weston V, Gisondi MA. Journal Club: Coronary CT Angiography Versus Traditional Care. NUEM Blog. http://www.nuemblog.com/blog/cta-for-chest-pain/ Published online 4/12/16.|
|[Video] Mason J. Placing a Transvenous Pacemaker. Emergency Medicine: Reviews and Perspectives. October 1, 2018. https://www.emrap.org/episode/transvenous/transvenous. Accessed November 21, 2018.|
|[Traditional Paper with Altmetrics] Chan TM, Gottlieb M, Sherbino J, Cooney R, Boysen-Osborn M, Swaminathan A, Ankel F, Yarris LM. The ALiEM faculty incubator: a novel online approach to faculty development in education scholarship. Academic Medicine. 2018 Oct 1;93(10):1497–502. Altmetrics data: https://wolterskluwer.altmetric.com/details/43542602|
Digital scholarship is often criticized for lack of peer review, which leads to confusion about the quality and integrity of articles published in exclusively online journals. Peer review is a requirement for all journals to be indexed and available on PubMed, including online journals. Research articles published in online-only journals that have a PubMed unique identifier (PMID)32 should not be listed under “Digital Scholarship,” but rather alongside similar scholarly work published in peer-reviewed print journals. Regardless of the mode of publication, all peer-reviewed research should be listed under the same CV subheading in the “highest” possible category.
Blog posts that are cited under a “Digital Scholarship” CV subheading can be peer reviewed as well. For example, some blogs offer a peer-review process for authors and identify which posts have undergone peer review.33 Therefore, use qualifiers to identify any digital scholarship citation on your CV that was peer-reviewed or invited. These qualifiers may add additional credibility to your scholarship when a P&T committee reviews your CV.
Crafting a Digital Scholarship Mission Statement
A digital scholarship mission statement can provide a framework for your P&T committee to understand and interpret your digital scholarship.34 Akin to the educational philosophy statement of a teaching portfolio, the digital scholarship mission statement provides a lens through which the committee can interpret the congruence and value of your scholarship.35,36 This narrative should articulate the beliefs that drive your digital work in ways that give perspective to your activities and provide consistency with the academic and social media strategies of each institution. Table 5 below lists specific considerations to include. Please see Supplemental Digital Appendix B for a sample narrative.
Table 5Specific elements to consider within a mission statement.
|1||Reinforce why your digital scholarship exists and is important to the field.|
|2||Explain your digital scholarship’s broad goals and objectives.|
|3||Explain your perception of needs in the modern learning environment, and how that affects your methods.|
|4||Explain how your approach to digital scholarship/teaching has changed over time.|
|5||Explain the niche that you are filling, specifically highlighting how your role/expertise at your institution gives you a reputable voice.|
|6||Describe how your digital scholarship complements your other, more traditional forms of scholarship.|
|7||Explain how digital scholarship aligns with your overall career objectives.|
|8||Name your intended target audience and describe other collateral audience groups that may benefit from your public academic work.|
|9||Describe best practices for ensuring quality during the content creation process:
Highlight team-based and interdisciplinary scholarship as markers of quality
Preview external validation processes of your digital scholarship (below).
|10||Highlight the ancillary benefits that have arisen because of your digital scholarship presence, such as invited lectures or collaborations on additional scholarship.|
Use Traditional Frameworks: Harnessing the Teaching Portfolio
We recommend using traditional frameworks to describe digital scholarly activity and support for academic promotion. One such example of this is the teaching portfolio. As not all institutions require a separate educational portfolio, we recommend that you present your digital scholarship alongside traditional scholarship according to your institutional requirements. Refer to your respective institutional guidelines for requirements and formatting of teaching portfolios. Regardless, to facilitate appraisal by P&T committees you should create a dossier that includes a digital mission statement, demonstrates alignment with overall career development goals, and describes the scholarly significance of your digital work.25
Digital scholarship should not replace materials that are typically included in a teaching portfolio, such as course evaluations or other traditional measures of teaching effectiveness. Teaching portfolios should summarize teaching effort and quality that meet the criteria of Boyer’s scholarship of teaching.4,37 Within the teaching portfolio, you may reflect and provide exemplars of digital works and curricula that you have created or curated for learners, but you will not actually list item-by-item the digital scholarship you produce; this should take place in the CV. An entry in a portfolio would holistically describe the pedagogical principles behind a digital educational program or innovation (eg, if you are the creator of a popular podcast, you would explain how you developed the podcast, how you engaged stakeholders to develop the podcast, and, if possible, share data to convey its impact at large through analytics). In contrast, entries of digital scholarship on a CV would be entered individually. Table 6 provides some common examples of digital scholarship, and how they might align best with previous descriptions of traditional academic scholarship (as per Boyer, Glassick, Hutchings and Shulman).
Table 6How types of digital scholarship might be described using traditional descriptions of academic scholarship.
|Example of digital scholarship||Blog post providing a new insight into a novel teaching technique, with a recipe for helping students learn about social justice by meeting patient partners.||Podcast synthesizing the role of human factor engineering in the emergency department.||Tweetorial reviewing and appraising the latest evidence on a topic|
|Does this meet the criteria for scholarship per Hutchings and Shulman?
Available for peer review and critique according to the standards of a field
Able to be reproduced and extended by other scholars
|1) Is it public? Yes2) Is it available for peer review? Yes, some blogs have pre-publication peer review, others have comments enabled to allow for post-publication peer review)3) Able to be reproduced and extended by other scholars? Yes, since it is available for review and extendibility since it is openly published on the internet.||1) Is it public? Yes2) Is it available for peer review? Yes, listeners can leave comments on most podcast hosting sites.3) Able to be reproduced and extended by other scholars? Yes, since it is available for review and extendibility since it is openly published on the internet.||1) Is it public? Yes2) Is it available for peer review? Yes, tweetworials can be found by searching Twitter.3) Able to be reproduced and extended by other scholars? Yes, since it is available for review and extendibility since it is openly published on the internet.|
|What type of Boyer’s scholarship is this?||Scholarship of teaching||Scholarship of integration (merging of engineering and medicine)||Scholarship of application (helping others to determine if evidence might be applied in their context)|
There are no hard and fast rules for determining impact. Cabrera and his colleagues have previously suggested scale-based assessments of social media-based impact in their 2017 paper.34 They provide ample guidance to promotions committees for comparing size and scale of various media within a specific subtype (eg, international blog vs a local blog). We highly recommend that readers review this article for further guidance.
Another tool is the Social Media Index, which seeks to create an “impact factor”-like metric based on social media followership. This tool would be best used to judge the impact of an entire digital media collection, such as an entire website or podcast. This tool is available online ( https://www.aliem.com/social-media-index/) and has been revised and validated against quality metrics within emergency medicine Free Open Access Medicine resources.38
Due to lower barriers of entry allowing digital scholarship to be more easily produced, general skepticism due to less serious, nonmedical online content, as well as pseudoscientific and/or predatory online content, groups have sought to scaffold and support end-users and educators in seeking high-quality online resources.39,40 The online medical education community has worked to quell skepticism by establishing methods to appraise the quality of digital scholarship.3 See below for a list of critical appraisal tools for rating online secondary resources. For those who have been asked to review files as external referees, these tools may be very useful in guiding us toward high-quality educational content from an educator’s CV or portfolio.
Some scholars in this space have proposed that we move beyond bibliometrics and surrogates for quality (eg, impact factor, citations, altmetrics), and that P&T committees consider applying direct quality assessments to items of interest (eg, applying the revised METRIQ41 or ALiEM Approved Instructional Resources (AIR series) scores39 to a few choice works of digital scholarship from a faculty member’s CV, or applying the PRISMA42 reporting guidelines to a few systematic reviews). Equitably applying both descriptive bibliometrics (eg, citation rate, h-index, etc.) and quality audits to all works of scholarship (digital or otherwise) would go a long way to augment P&T processes. Table 2 contains suggested critical appraisal tools to facilitate secondary resource evaluation.
The live conference was limited to invited participants who could join in person or by phone. Those with scheduling conflicts were therefore excluded from the live session, perhaps limiting valuable insights and contributions. However, those that could not attend the live conference were still heavily involved in the organization and creation of the recommendations post-conference via a collaborative writing process. Additionally, all authors participated robustly in the asynchronous editing of this manuscript, reducing the potential that important viewpoints were excluded. Conference participants were selected by the committee members, and important contributors may have been overlooked. To reduce this possibility, invited members were requested to suggest additional invitees. Finally, as digital scholarship participants and creators, there may be bias toward legitimizing our own work over less-familiar scholarship.
We attempted to ground our recommendations using best available evidence in order to reduce this bias. However, there is certainly a paucity of literature on how social media is viewed upon (or accepted) as a form of scholarship by the academy. Thus, further explorations of the acceptability or evaluation of digital by P&T committees may be a useful program of research going forward. A paper has recently been published about perceptions in the librarian sciences world that is quite interesting, and worthy of replication within academic medicine.43
As traditional scholarship continues to evolve within the digital realm, academic medicine must also adapt how that scholarship is evaluated. P&T committees in academia are at the epicenter for supporting the changing paradigm in scholarship. Unlike traditional academic products, where reach and impact were difficult to quantify, web-based metrics allow us to track unique users and their locations. The authors suggest that committees critically appraise digital scholarship using the methods outlined in this paper. Applicants for appointment and promotion should highlight and prepare their digital scholarship in a way that specifically addresses quality, impact, breadth, and relevance. It is our goal to provide specific, timely guidance for both stakeholders to recognize the value of digital scholarship in advancing our field.
Section Editor: Section Editor: Jeffrey Druck, MD
Full text available through open access at http://escholarship.org/uc/uciem_westjem
Address for Correspondence: Abbas Husain, MD, Staten Island University Hospital, Department of Emergency Medicine, 475 Seaview Avenue, Staten Island, NY 10305. Email: firstname.lastname@example.org. 7 / 2020; 21:883 – 891
Submission history: Revision received January 6, 2020; Submitted April 4, 2020; Accepted April 9, 2020
Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.
1. Azer SA, Holen A, Wilson I, et al. Impact factor of medical education journals and recently developed indices: Can any of them support academic promotion criteria?. J Postgrad Med. 2016;61(1):32-9.
2. Schimanski LA, Alperin JP. The evaluation of scholarship in academic promotion and tenure processes: past, present, and future. F1000Res. 2018;7:1605.
3. Chan TM, Kuehl D. On lampposts, sneetches, and stars: a call to go beyond bibliometrics for determining academic value. Acad Emerg Med. 2019;26(6):688-94.
4. Boyer EL. Scholarship Reconsidered: Priorities of the Professoriate. 1990.
5. Glassick CE, Huber MT, Maeroff GI. Scholarship Assessed: Evaluation of the Professoriate. 1997.
6. Glassick CE. Boyer’s expanded definitions of scholarship, the standards for assessing scholarship, and the elusiveness of the scholarship of teaching. Acad Med. 2000;75(9):877-80.
7. Hutchings P, Shulman LS. The scholarship of teaching: new elaborations, new developments. Change: The Magazine of Higher Learning. 1999;31(5):10-5.
8. Cabrera D, Roy D, Chisolm MS. Social media scholarship and alternative metrics for academic promotion and tenure. J Am Coll Radiol. 2018;15(1Pt B):135-41.
9. Thomas B, Chan T, Benitez J, et al. Educational scholarship in the digital age: a scoping review and analysis of scholarly products. The Winnower. 2014.
10. bioRxiv Homepage. Available at: https://www.biorxiv/org/. Accessed June 21, 2019.
11. An introduction to publishing with Cureus. Available at: https://www.cureus.com/author_guide. Accessed June 17, 2019.
12. Adler JR, Chen TM, Blain JB, et al. #OpenAccess: free online, open-access crowdsource-reviewed publishing is the future; traditional peer-reviewed journals are on the way out. CJEM. 2019;21(1):11-4.
13. Azim A, Beck-Esmay J, Chan TM. Editorial processes in free open access medical educational (FOAM) resources. AEM Educ Train. 2018;2(3):204-12.
14. Chan TM, Gottlieb M, Sherbino J, et al. Education Theory Practical. 2017;1.
15. MedEdPORTAL: The Journal of Teaching and Learning Resources. Available at: https://www.mededportal.org/. Accessed June 17, 2019.
16. JetEM: Journal of Education and Teaching in Emergency Medicine. Available at: https:jetem.org/. Accessed June 17, 2019.
17. Cadogan M, Thoma B, Chan TM, et al. Free Open Access Meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002–2013). Emerg Med J. 2014;31(e1):e76-7.
18. Colmers IN, Paterson QS, Lin M, et al. The quality checklists for health professions blogs and podcasts. The Winnower. 2020.
19. Thoma B, Joshi N, Trueger NS, et al. Five strategies to effectively use online resources in emergency medicine. Ann Emerg Med. 2014;64(4):392-5.
20. Lo A, Shappell E, Rosenberg H, et al. Four strategies to find, evaluate, and engage with online resources in emergency medicine. CJEM. 2018:293-9.
21. Waggoner J, Carline JD, Durning SJ. Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Acad Med. 2016;91(5):663-8.
22. Google Accounts. Available at: http://docs.google.com.
23. Woods RA, Artz JD, Carrière B, et al. CAEP 2016 Academic Symposium on Education Scholarship: Training Our Future Clinician Educators in Emergency Medicine. CJEM. 2017;19(S1):S1-8.
24. Kessler CS, Leone KA. The current state of core competency assessment in emergency medicine and a future research agenda: recommendations of the Working Group on Assessment of Observable Learner Performance. Acad Emerg Med. 2012;19(12):1354-9.
25. Glassick CE. Reconsidering scholarship. J Public Health Management Practice. 2000;6:4-9.
26. Sherbino J, Arora VM, Melle EV, et al. Criteria for social media-based scholarship in health professions education. Postgrad Med J. 2015;91(1080):551-5.
27. Trueger NS, Thoma B, Hsu CH, et al. The altmetric score: a new measure for article-level dissemination and impact. Ann Emerg Med. 2015;66(5):549-53.
28. Thoma B, Sanders JL, Lin M, et al. The Social Media Index: measuring the impact of emergency medicine and critical care websites. West J Emerg Med. 2015;16(2):242-9.
29. Monette D, Joshi N. ALiEM Awards 2017: Congratulations to our winners!. Available at: https://www.aliem.com/2017/12/aliem-awards-2017-winners/. Accessed June 17, 2019.
30. Iverson C, Christiansen S, Flanagin A, et al. AMA Manual of Style. 2007.
31. Mason J. How to cite podcasts, videos, and blogs in publication. 2018. Available at: https://www.aliem.com/2018/11/cite-podcasts-videos-blogs-publication/. Accessed June 17, 2019.
32. Available at: https://www.ncbi.nlm.nih.gov/pubmed/. Accessed June 17, 2019.
33. Kobner S. Introducing in-line peer review: advancing the state of academic blogging. 2017. Available at: https://www.aliem.com/2017/01/in-line-expert-peer-review-academic-blogging/. Accesses June 17, 2019.
34. Cabrera D, Vartabedian BS, Spinner RJ, et al. More than likes and tweets: creating social media portfolios for academic promotion and tenure. J Grad Med Educ. 2017;9(4):421-5.
35. Kuhn GJ. Faculty Development: The educator’s portfolio: its preparation, uses, and value in academic medicine. Acad Emerg Med. 2004;11(3):307-11.
36. Seldin P, Miller JE, Seldin CA. The Teaching Portfolio: a Practical Guide to Improved Performance and Promotion/Tenure Decisions. 2010.
37. Lamki N, Marchand M. The medical educator teaching portfolio: its compilation and potential utility. Sultan Qaboos Univ Med J. 2006;6(1):7-12.
38. Thoma B, Sanders JL, Lin M, et al. The Social Media Index: measuring the impact of emergency medicine and critical care websites. West J Emerg Med. 2015;16(2):242-9.
39. Thoma B, Chan TM, Kapur P, et al. The Social Media Index as an Indicator of quality for emergency medicine blogs: a METRIQ study. Ann Emerg Med. 2018;72(6):696-702.
40. Chan TM, Grock A, Paddock M, et al. Examining reliability and validity of an online score (ALiEM AIR) for rating free open access medical education resources. Ann Emerg Med. 2016;68(6):729-35.
41. Colmers-Gray IN, Krishnan K, Chan TM, et al. The revised METRIQ score: a quality evaluation tool for online educational resources. AEM Education and Training. 2019;3(4).
42. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.
43. Gruzd A, Staves K, Wilk A. Tenure and promotion in the age of online social media. Proceedings of the American Society for Information Science and Technology. 2012;48(1):1-9.