Author	Affiliation
Abbas Husain, MD	Staten Island University Hospital – Northwell Health, Department of Emergency Medicine, Staten Island, New York
Zachary Repanshek, MD	Lewis Katz School of Medicine at Temple University, Department of Emergency Medicine, Philadelphia, Pennsylvania
Manpreet Singh, MD	University of California, Los Angeles Medical School, Department of Emergency Medicine, Los Angeles, California
Felix Ankel, MD	University of Minnesota Medical School, Department of Emergency Medicine, Minneapolis, Minnesota
Jennifer Beck-Esmay,	Mount Sinai St. Luke’s-West, Department of Emergency Medicine, New York, New York
Daniel Cabrera, MD	Mayo Clinic College of Medicine, Department of Emergency Medicine, Rochester, Minnesota
Teresa M. Chan, MD, MHPE, FRCPC	McMaster University, Department of Medicine, Hamilton, Ontario
Robert Cooney, MD, MSMedEd	Geisinger Medical Center, Department of Emergency Medicine, Danville, Pennsylvania
Michael Gisondi, MD	Stanford University School of Medicine, Department of Emergency Medicine, Stanford, California
Michael Gottlieb, MD	Rush University Medical Center, Department of Emergency Medicine, Chicago, Illinois
Jay Khadpe, MD	University of Florida College of Medicine, Department of Emergency Medicine, Jacksonville, Florida
Jennifer Repanshek, MD	Lewis Katz School of Medicine at Temple University, Department of Emergency Medicine, Philadelphia, Pennsylvania
Jessica Mason, MD	University of San Francisco-Fresno, Department of Emergency Medicine, Fresno, California
Dimitrios Papanagnou, MD	Sidney Kimmel Medical College at Thomas Jefferson University, Department of Emergency Medicine, Philadelphia, Pennsylvania
Jeff Riddell, MD	University of Southern California Keck School of Medicine, Department of Emergency Medicine, Los Angeles, California
N. Seth Trueger, MD	Stanford University School of Medicine, Department of Emergency Medicine, Stanford, California; Lewis Katz School of Medicine at Temple University, Department of Emergency Medicine, Philadelphia, Pennsylvania
Fareen Zaver, MD	Rush University Medical Center, Department of Emergency Medicine, Chicago, Illinois; University of California, Los Angeles Medical School, Department of Emergency Medicine, Los Angeles, California
Emily Brumfield, MD	Oschner Clinic Foundation, Department of Emergency Medicine, New Orleans, Louisiana

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

The promotion and tenure (P&T) process requires the appraisal of individual scholarly impact in comparison to scholars across institutions and disciplines. Comparative metrics such as the journal impact factor and the h-index are used to quantify and compare the quality of an individual’s scholarship and, therefore, his or her academic merit.¹ As knowledge dissemination methods evolve in the digital era, we must adapt traditional P&T processes to include emerging forms of digital scholarship.² In this paper, we aim to first situate our readers within the literature on the topic of academic scholarship, after which we will describe the process by which we derived and refined our consensus guideline. Finally, we will outline the recommendations for the use of digital scholarship for academic promotion made by this particular guideline group.

The Evolution of Scholarship

Scholarship is persistently dynamic. Analog technologies progressed from tablet and stone to pen and paper; modern digital scholarship is evolving with blogs, podcasts, and digital journals. Still, the standards for evaluation are consistent and focus predominantly on impact and quality of the scholarship.³ In 1990, Ernest Boyer of the Carnegie Foundation originally redefined scholarship for the professoriate as belonging to one of four types.⁴ A decade later, Charles Glassick followed up this work by describing criteria for evaluating scholarship.^5,6 To further develop nuances around the scholarship of teaching and learning, Lee Shulman and Patricia Hutchings further clarified specific criteria for this subtype of scholarship (to differentiate it from high-quality, scholarly, and evidence-based teaching).⁷ These foundational concepts are summarized in Table 1 below.

Traditionally, peer-review processes of academic journals served as a safeguard to ensure overall quality, with evaluators deferring to experts and peers within a scholar’s domain to provide an appraisal for quality and an estimate of impact. Similarly, bibliometrics of journals (eg, journal impact factor)⁸ and number of citations are surrogates for scholarly reach and proof of impact.³ Despite well-described limitations, these metrics are quantifiable and defined processes that are easily compared. Thus, they are highly relied upon by P&T committees to compare scholars from disparate disciplines. When scholarship using new media is produced, it is reasonable to scrutinize the methodology, content, impact, and quality of these new forms of scholarship, such as digital scholarship. Our use of the term “digital scholarship” in this paper reflects original content that is disseminated digitally, whether that content is research, teaching materials, enduring resources, commentaries, or other scholarly work.

It is unsurprising that as the world becomes more digital, so do scholarly contributions.⁹ Online-only journals, pre-print archives,¹⁰ and post-production, peer-review journals (eg, Cureus) are rapidly changing the landscape of peer-reviewed publication.^11,12 Similarly, with the advent of peer-reviewed blogs,¹³ self-published peer-reviewed books,¹⁴ and educational resource repositories, we see an increased breadth of expression from those engaging in the scholarship of teaching. These varied forms increasingly mirror the rigor required by Glassick’s criteria and Shulman’s paradigms.^15,16

Quantity vs Quality

Judging these new forms of scholarship is different. In many ways, with advanced web analytics, it is easier to quantify the reach and attention (eg, pageviews, podcast downloads, IP addresses that have accessed the content, and time on page) of these digital assets. (See Table 2 for common analytics available for new digital scholarship.) For example, the PubMed-indexed repository MedEdPortal provides download analytics of the published resources that aid in describing entries as fitting within the scholarship of teaching.

However, since many disciplines both within and outside of medicine have not yet fully embraced digital scholarship as enthusiastically as emergency medicine (EM) and critical care,¹⁷ it is no surprise that P&T committees do not yet have specific or universal standards for presentation or evaluation of digital scholarship.¹⁸ Those without digital scholarship experience may grapple with understanding the nuances of determining impact and quality in this new era, and their lack of understanding may even result in general skepticism of novel products. Thus, fields that have already established robust methods for determining the quality of digital scholarship can lead the way. Since digital scholarship has matured in EM,^19,20 it is appropriate for our field to call for the identification of best practices for evaluating digital scholarship and for consensus in the inclusion of such items in promotions decisions.

Specific guidelines for P&T are lacking despite robust digital contributions proliferating among academicians. In this work, we provide a guiding framework for the presentation and evaluation of digital scholarship for the applicant for promotion, referees for the candidate, and members of P&T committees.

METHODS

We conducted a blended, expert consensus procedure using a nominal group process to create a consensus document.²¹ Invited participants met at the Council of Emergency Medicine Residency Directors (CORD) Academic Assembly on April 1, 2019 (Seattle, WA), to discuss recommendations for evaluation and promotion of digital scholarship with the intent to develop specific, evidence-supported recommendations to P&T committees and applicants. We began with a live, brainstorming event. The meeting notes were compiled by a leadership team and formatted into a collaborative working document. All authors continued formulating this document via a collaborative online authorship using Google Docs (Google LLC, Mountain View, CA).²²

Participants

The participants were selected by the leadership of the CORD Social Media and Digital Scholarship Committee (EB, ZR, AH). Participants were selected based on criteria of known interest or scholarship in the area, national and international level contributions to EM digital scholarship, and availability to attend the conference in person or by phone. Supplemental Digital Appendix A lists original invitation list and individual selection rationale. The complete list of attendees of the in-conference proceedings is listed in the acknowledgments.

Procedures

As a large group, the consensus conference participants democratically developed the discussion and brainstorming procedures. Based on suggestions from the floor about previous consensus procedures at other similar conferences,^23,24 our group decided to engage in small- group brainstorming discussions aligned with the expertise and interests of the participants, which was then discussed as a large group and vetted by the rest of the participants. Consensus was defined as universal agreement of the participants.

Ideation and refinement

The participants self-identified their areas of expertise or interest, and then separated into three groups based on these content areas using an iterative process to formulate specific recommendations. The three discussion groups were tasked with formulating recommendations for the following:

 

The P&T applicant for promotion of one’s digital scholarship;

 

P&T committee members for evaluation of quality of digital scholarship;

 

P&T committee members for evaluation of the impact of digital scholarship.

 

Small groups presented preliminary recommendations to the entire group and made further revisions via iterative discussion. Participants transcribed an outline of the discussion and final recommendations and agreed upon them in a democratic fashion. Participants self-selected areas of the manuscript to prepare based on expertise, interest and group approval. All members developed the manuscript from the outline via collaborative authorship. All participants contributed to the manuscript, and CORD Social Media and Digital Scholarship committee members (AH, MS, ZR, EB) served as final editors of the manuscript.

RESULTS

Recommendations for Presenting Digital Scholarship for Promotion and Tenure

Demonstrate Scholarship Criteria

When presenting digital scholarship to a P&T committee, begin by ensuring and demonstrating that it meets the criteria of scholarship as defined by Glassick and expanded upon by Sherbino and colleagues with regard to social media.^25,26 The adapted criteria are as follows: 1) create original content; 2) advance the field of health professions education by building on theory, research or best practice; 3) be archived and disseminated, and 4) provide the health professions education community with the ability to comment on and provide feedback in a transparent fashion that informs wider discussion. In addition, consider providing evidence of archival and dissemination, such as Google Scholar indexing or inclusion of a digital object identifier (DOI).

Provide External Evidence of Impact

Ensure that your digital scholarship is reflected consistently throughout your promotions dossier. Dissemination metrics are important to include as measures of impact. For example, some blog editors will provide information about how many times a post has been accessed and the locations of its readers, if requested for P&T purposes. Such metrics of dissemination and impact should be presented in the dossier as evidence of your professional reputation as a scholar in your field.

Additional metrics include pageviews, downloads, and geographic reach. Other programs assessing the reach of scholarship, such as altmetrics, may also be valuable.²⁷ The Social Media Index is a relatively newer technique to assess the impact of websites and could be used as a surrogate for impact, much the same as a journal’s impact factor.²⁸ See Table 2.

Other measures of impact could include letters of support and awards. If permitted by your institution, consider obtaining letters of support with regard to your digital scholarship. You may also consider inviting both peer letters and letters from non-collaborators discussing the dissemination metrics and impact of specific pieces of scholarship, or simply your overall impact. There are also a number of digital scholarship-based awards, which may be of value for demonstrating scholarly impact.²⁹

Include Digital Peer-review Roles

Include editor or peer-reviewer roles for digital scholarly content in your curriculum vitae (CV) in a similar manner as you would for traditional print literature. It is important to highlight these supporting components of digital scholarship and they should be factored into the P&T decisions.²⁵

Citing Digital Scholarship

Cite scholarly work on your CV using a consistent format, whether that work was published in a hard-copy journal or as digital content. Reorganize the categories of scholarly publications on your CV to include a section for “Digital Scholarship,” which is the appropriate subheading for items such as blog posts, podcasts, and videos. See Table 3 below for example subheadings for the scholarly bibliography of your CV. Include only those items that reflect true scholarship and relate to the health professions or sciences. Do not list citations for personal website posts or other digital content that is unrelated to your academic position.

Consistently format your scholarship across all subheadings on your CV following the American Medical Association (AMA) Manual of Style, 10th Ed.³⁰ The AMA Manual describes the methods for citing scholarship in most of the categories listed. Examples of each citation type are provided above, and selected citations are adapted in Table 4. Digital scholarship is best formatted using the AMA Manual instructions for “Internet Documents.”³³ Academic Life in Emergency Medicine (ALiEM) also offers guidelines for citing digital scholarship, with examples.³¹

Digital scholarship is often criticized for lack of peer review, which leads to confusion about the quality and integrity of articles published in exclusively online journals. Peer review is a requirement for all journals to be indexed and available on PubMed, including online journals. Research articles published in online-only journals that have a PubMed unique identifier (PMID)³² should not be listed under “Digital Scholarship,” but rather alongside similar scholarly work published in peer-reviewed print journals. Regardless of the mode of publication, all peer-reviewed research should be listed under the same CV subheading in the “highest” possible category.

Blog posts that are cited under a “Digital Scholarship” CV subheading can be peer reviewed as well. For example, some blogs offer a peer-review process for authors and identify which posts have undergone peer review.³³ Therefore, use qualifiers to identify any digital scholarship citation on your CV that was peer-reviewed or invited. These qualifiers may add additional credibility to your scholarship when a P&T committee reviews your CV.

DISCUSSION

Crafting a Digital Scholarship Mission Statement

A digital scholarship mission statement can provide a framework for your P&T committee to understand and interpret your digital scholarship.³⁴ Akin to the educational philosophy statement of a teaching portfolio, the digital scholarship mission statement provides a lens through which the committee can interpret the congruence and value of your scholarship.^35,36 This narrative should articulate the beliefs that drive your digital work in ways that give perspective to your activities and provide consistency with the academic and social media strategies of each institution. Table 5 below lists specific considerations to include. Please see Supplemental Digital Appendix B for a sample narrative.

Use Traditional Frameworks: Harnessing the Teaching Portfolio

We recommend using traditional frameworks to describe digital scholarly activity and support for academic promotion. One such example of this is the teaching portfolio. As not all institutions require a separate educational portfolio, we recommend that you present your digital scholarship alongside traditional scholarship according to your institutional requirements. Refer to your respective institutional guidelines for requirements and formatting of teaching portfolios. Regardless, to facilitate appraisal by P&T committees you should create a dossier that includes a digital mission statement, demonstrates alignment with overall career development goals, and describes the scholarly significance of your digital work.²⁵

Digital scholarship should not replace materials that are typically included in a teaching portfolio, such as course evaluations or other traditional measures of teaching effectiveness. Teaching portfolios should summarize teaching effort and quality that meet the criteria of Boyer’s scholarship of teaching.^4,37 Within the teaching portfolio, you may reflect and provide exemplars of digital works and curricula that you have created or curated for learners, but you will not actually list item-by-item the digital scholarship you produce; this should take place in the CV. An entry in a portfolio would holistically describe the pedagogical principles behind a digital educational program or innovation (eg, if you are the creator of a popular podcast, you would explain how you developed the podcast, how you engaged stakeholders to develop the podcast, and, if possible, share data to convey its impact at large through analytics). In contrast, entries of digital scholarship on a CV would be entered individually. Table 6 provides some common examples of digital scholarship, and how they might align best with previous descriptions of traditional academic scholarship (as per Boyer, Glassick, Hutchings and Shulman).

Appraising Impact

There are no hard and fast rules for determining impact. Cabrera and his colleagues have previously suggested scale-based assessments of social media-based impact in their 2017 paper.³⁴ They provide ample guidance to promotions committees for comparing size and scale of various media within a specific subtype (eg, international blog vs a local blog). We highly recommend that readers review this article for further guidance.

Another tool is the Social Media Index, which seeks to create an “impact factor”-like metric based on social media followership. This tool would be best used to judge the impact of an entire digital media collection, such as an entire website or podcast. This tool is available online ( https://www.aliem.com/social-media-index/) and has been revised and validated against quality metrics within emergency medicine Free Open Access Medicine resources.³⁸

Appraising Quality

Due to lower barriers of entry allowing digital scholarship to be more easily produced, general skepticism due to less serious, nonmedical online content, as well as pseudoscientific and/or predatory online content, groups have sought to scaffold and support end-users and educators in seeking high-quality online resources.^39,40 The online medical education community has worked to quell skepticism by establishing methods to appraise the quality of digital scholarship.³ See below for a list of critical appraisal tools for rating online secondary resources. For those who have been asked to review files as external referees, these tools may be very useful in guiding us toward high-quality educational content from an educator’s CV or portfolio.

Some scholars in this space have proposed that we move beyond bibliometrics and surrogates for quality (eg, impact factor, citations, altmetrics), and that P&T committees consider applying direct quality assessments to items of interest (eg, applying the revised METRIQ⁴¹ or ALiEM Approved Instructional Resources (AIR series) scores³⁹ to a few choice works of digital scholarship from a faculty member’s CV, or applying the PRISMA⁴² reporting guidelines to a few systematic reviews). Equitably applying both descriptive bibliometrics (eg, citation rate, h-index, etc.) and quality audits to all works of scholarship (digital or otherwise) would go a long way to augment P&T processes. Table 2 contains suggested critical appraisal tools to facilitate secondary resource evaluation.

LIMITATIONS

The live conference was limited to invited participants who could join in person or by phone. Those with scheduling conflicts were therefore excluded from the live session, perhaps limiting valuable insights and contributions. However, those that could not attend the live conference were still heavily involved in the organization and creation of the recommendations post-conference via a collaborative writing process. Additionally, all authors participated robustly in the asynchronous editing of this manuscript, reducing the potential that important viewpoints were excluded. Conference participants were selected by the committee members, and important contributors may have been overlooked. To reduce this possibility, invited members were requested to suggest additional invitees. Finally, as digital scholarship participants and creators, there may be bias toward legitimizing our own work over less-familiar scholarship.

We attempted to ground our recommendations using best available evidence in order to reduce this bias. However, there is certainly a paucity of literature on how social media is viewed upon (or accepted) as a form of scholarship by the academy. Thus, further explorations of the acceptability or evaluation of digital by P&T committees may be a useful program of research going forward. A paper has recently been published about perceptions in the librarian sciences world that is quite interesting, and worthy of replication within academic medicine.⁴³

CONCLUSION

As traditional scholarship continues to evolve within the digital realm, academic medicine must also adapt how that scholarship is evaluated. P&T committees in academia are at the epicenter for supporting the changing paradigm in scholarship. Unlike traditional academic products, where reach and impact were difficult to quantify, web-based metrics allow us to track unique users and their locations. The authors suggest that committees critically appraise digital scholarship using the methods outlined in this paper. Applicants for appointment and promotion should highlight and prepare their digital scholarship in a way that specifically addresses quality, impact, breadth, and relevance. It is our goal to provide specific, timely guidance for both stakeholders to recognize the value of digital scholarship in advancing our field.

Footnotes

Section Editor: Section Editor: Jeffrey Druck, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Abbas Husain, MD, Staten Island University Hospital, Department of Emergency Medicine, 475 Seaview Avenue, Staten Island, NY 10305. Email: abbashu@gmail.com. 7 / 2020; 21:883 – 891

Submission history: Revision received January 6, 2020; Submitted April 4, 2020; Accepted April 9, 2020

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Azer SA, Holen A, Wilson I, et al. Impact factor of medical education journals and recently developed indices: Can any of them support academic promotion criteria?. J Postgrad Med. 2016;61(1):32-9.

2. Schimanski LA, Alperin JP. The evaluation of scholarship in academic promotion and tenure processes: past, present, and future. F1000Res. 2018;7:1605.

3. Chan TM, Kuehl D. On lampposts, sneetches, and stars: a call to go beyond bibliometrics for determining academic value. Acad Emerg Med. 2019;26(6):688-94.

4. Boyer EL. Scholarship Reconsidered: Priorities of the Professoriate. 1990.

5. Glassick CE, Huber MT, Maeroff GI. Scholarship Assessed: Evaluation of the Professoriate. 1997.

6. Glassick CE. Boyer’s expanded definitions of scholarship, the standards for assessing scholarship, and the elusiveness of the scholarship of teaching. Acad Med. 2000;75(9):877-80.

7. Hutchings P, Shulman LS. The scholarship of teaching: new elaborations, new developments. Change: The Magazine of Higher Learning. 1999;31(5):10-5.

8. Cabrera D, Roy D, Chisolm MS. Social media scholarship and alternative metrics for academic promotion and tenure. J Am Coll Radiol. 2018;15(1Pt B):135-41.

9. Thomas B, Chan T, Benitez J, et al. Educational scholarship in the digital age: a scoping review and analysis of scholarly products. The Winnower. 2014.

10. bioRxiv Homepage. Available at: https://www.biorxiv/org/. Accessed June 21, 2019.

11. An introduction to publishing with Cureus. Available at: https://www.cureus.com/author_guide. Accessed June 17, 2019.

12. Adler JR, Chen TM, Blain JB, et al. #OpenAccess: free online, open-access crowdsource-reviewed publishing is the future; traditional peer-reviewed journals are on the way out. CJEM. 2019;21(1):11-4.

13. Azim A, Beck-Esmay J, Chan TM. Editorial processes in free open access medical educational (FOAM) resources. AEM Educ Train. 2018;2(3):204-12.

14. Chan TM, Gottlieb M, Sherbino J, et al. Education Theory Practical. 2017;1.

15. MedEdPORTAL: The Journal of Teaching and Learning Resources. Available at: https://www.mededportal.org/. Accessed June 17, 2019.

16. JetEM: Journal of Education and Teaching in Emergency Medicine. Available at: https:jetem.org/. Accessed June 17, 2019.

17. Cadogan M, Thoma B, Chan TM, et al. Free Open Access Meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002–2013). Emerg Med J. 2014;31(e1):e76-7.

18. Colmers IN, Paterson QS, Lin M, et al. The quality checklists for health professions blogs and podcasts. The Winnower. 2020.

19. Thoma B, Joshi N, Trueger NS, et al. Five strategies to effectively use online resources in emergency medicine. Ann Emerg Med. 2014;64(4):392-5.

20. Lo A, Shappell E, Rosenberg H, et al. Four strategies to find, evaluate, and engage with online resources in emergency medicine. CJEM. 2018:293-9.

21. Waggoner J, Carline JD, Durning SJ. Is there a consensus on consensus methodology? Descriptions and recommendations for future consensus research. Acad Med. 2016;91(5):663-8.

22. Google Accounts. Available at: http://docs.google.com.

23. Woods RA, Artz JD, Carrière B, et al. CAEP 2016 Academic Symposium on Education Scholarship: Training Our Future Clinician Educators in Emergency Medicine. CJEM. 2017;19(S1):S1-8.

24. Kessler CS, Leone KA. The current state of core competency assessment in emergency medicine and a future research agenda: recommendations of the Working Group on Assessment of Observable Learner Performance. Acad Emerg Med. 2012;19(12):1354-9.

25. Glassick CE. Reconsidering scholarship. J Public Health Management Practice. 2000;6:4-9.

26. Sherbino J, Arora VM, Melle EV, et al. Criteria for social media-based scholarship in health professions education. Postgrad Med J. 2015;91(1080):551-5.

27. Trueger NS, Thoma B, Hsu CH, et al. The altmetric score: a new measure for article-level dissemination and impact. Ann Emerg Med. 2015;66(5):549-53.

28. Thoma B, Sanders JL, Lin M, et al. The Social Media Index: measuring the impact of emergency medicine and critical care websites. West J Emerg Med. 2015;16(2):242-9.

29. Monette D, Joshi N. ALiEM Awards 2017: Congratulations to our winners!. Available at: https://www.aliem.com/2017/12/aliem-awards-2017-winners/. Accessed June 17, 2019.

30. Iverson C, Christiansen S, Flanagin A, et al. AMA Manual of Style. 2007.

31. Mason J. How to cite podcasts, videos, and blogs in publication. 2018. Available at: https://www.aliem.com/2018/11/cite-podcasts-videos-blogs-publication/. Accessed June 17, 2019.

32. Available at: https://www.ncbi.nlm.nih.gov/pubmed/. Accessed June 17, 2019.

33. Kobner S. Introducing in-line peer review: advancing the state of academic blogging. 2017. Available at: https://www.aliem.com/2017/01/in-line-expert-peer-review-academic-blogging/. Accesses June 17, 2019.

34. Cabrera D, Vartabedian BS, Spinner RJ, et al. More than likes and tweets: creating social media portfolios for academic promotion and tenure. J Grad Med Educ. 2017;9(4):421-5.

35. Kuhn GJ. Faculty Development: The educator’s portfolio: its preparation, uses, and value in academic medicine. Acad Emerg Med. 2004;11(3):307-11.

36. Seldin P, Miller JE, Seldin CA. The Teaching Portfolio: a Practical Guide to Improved Performance and Promotion/Tenure Decisions. 2010.

37. Lamki N, Marchand M. The medical educator teaching portfolio: its compilation and potential utility. Sultan Qaboos Univ Med J. 2006;6(1):7-12.

38. Thoma B, Sanders JL, Lin M, et al. The Social Media Index: measuring the impact of emergency medicine and critical care websites. West J Emerg Med. 2015;16(2):242-9.

39. Thoma B, Chan TM, Kapur P, et al. The Social Media Index as an Indicator of quality for emergency medicine blogs: a METRIQ study. Ann Emerg Med. 2018;72(6):696-702.

40. Chan TM, Grock A, Paddock M, et al. Examining reliability and validity of an online score (ALiEM AIR) for rating free open access medical education resources. Ann Emerg Med. 2016;68(6):729-35.

41. Colmers-Gray IN, Krishnan K, Chan TM, et al. The revised METRIQ score: a quality evaluation tool for online educational resources. AEM Education and Training. 2019;3(4).

42. Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

43. Gruzd A, Staves K, Wilk A. Tenure and promotion in the age of online social media. Proceedings of the American Society for Information Science and Technology. 2012;48(1):1-9.