Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Alternative metrics (altmetrics) emerged in the early 2010s in response to the limitations of traditional citation-based metrics.
¹
Altmetrics use a broader set of indicators such as page views, downloads, social media mentions, news media coverage, and expert recommendations to provide a more comprehensive understanding of an article’s influence.
²
Platforms like Altmetric.com and Plum Analytics provide tools for evaluating the reach and impact of scholarly articles, helping to track their online dissemination in real time. An increasing amount of evidence indicates that maintaining an active online presence can directly influence a researcher’s credentials as evaluated by conventional measures.
³
By considering various aspects beyond citations alone, altmetrics provides researchers and institutions with a more holistic assessment of their work’s societal impact. While the purpose of these metrics is to measure social impact, early social media visibility after publication can also increase and predict citations.
⁴
Assessing these activities could provide faster evaluations of an article’s impact and predict citations, serving as an early identifier for emerging areas of research growth.
⁵

Within the medical field specifically, these metrics offer insights into both scholarly recognition and public reception of research findings. Moreover, the relationship between these metrics underscores the changing landscape of scholarly communication, as researchers, clinicians, and the public alike engage with and contribute to the dissemination of research findings through online platforms. With the growth of digital communication and social media, the speed and the scale of information-sharing have accelerated, making altmetrics an invaluable tool for assessing real-time impact of articles.
⁶
Emerging trends can be key to a more efficiently functioning field of medicine.
⁷

Emergency medicine (EM) thrives on the timely dissemination of research and information that directly impacts patient care.
⁸
The first altmetrics analysis in EM conducted by Barbic et al found that the most-cited articles on social media in EM from 2011 were often published in non-EM biomedical journals.
⁹
Although this may suggest that authors in the field of EM select high-impact journals to increase the effectiveness of their publications, altmetric scores and journal impact factors are not correlated.
¹⁰
The social impact of an article may be better assessed by focusing on individual altmetric score sources rather than the overall score.
¹¹
This was supported by a recent study that found a direct correlation between X (formerly Twitter) mentions and article citations among EM research.
¹²
Our aim in this study was to investigate the correlations between the altmetric scores, their sources, and citations.

METHODS

In the first stage of the study, we used the Altmetric Explorer tool from Altmetric.com (Altmetric LLP, London, UK) to assess the Altmetric Attention Scores (AAS) of scholarly articles published in EM journals indexed in the Science Citation Index Expanded (SCIE). The AAS is a metric that evaluates the attention a research output receives using a weighted system that assigns distinct values to various sources, such as news outlets, blogs, and social media platforms. Sources with greater impact, such as news articles, are attributed higher weights compared to social media mentions. The score is calculated by a sophisticated algorithm that factors in not only the number of mentions but also variables such as duplicate posts and the credibility of the news sources.
¹³

All Altmetric Explorer assessments were conducted and downloaded as a CSV file on September 2, 2023. The initial analysis focused on examining the distribution of altmetric data by country and journal, as well as evaluating the sources of AAS scores over time. The 2022 Journal Citation Indicator (JCI) scores for the journals were obtained from the Web of Science (WOS) Master Journal List. These JCI scores represent the average citation impact of articles published between 2019–2021.

In the second stage we identified the top 200 articles with the highest AAS published in SCIE EM journals between January 1, 2013–January 1, 2023. We assessed the AAS of these articles along with the sources of their mentions. Citation counts for these articles were evaluated using both WOS and Google Scholar. To calculate the annual citation number, we divided the total citation counts obtained from the search engines by the number of years since the articles’ publication. We obtained full-text access for the articles to determine the article’s subject (eg, emergency department management, trauma, toxicology, resuscitation, critical care, COVID-19) and type (randomized controlled trial, observational study, case reports, reviews (systematic review and meta-analysis). Additionally, we determined the country of the first-named author of each article.

We assessed the correlation between the AAS and citation counts of the articles. Furthermore, two EM specialists from the research team independently reviewed the AAS and all screening processes of the study. In cases where the two evaluators did not reach a consensus, a third EM specialist conducted the evaluation. The study was approved by the Local Ethics Committee of İzmir Provincial Health Directorate Dr. Behcet Uz Pediatric Diseases and Surgery Training and Research Hospital.

Statistical Analysis

We performed statistical analysis using SPSS 29.0 for Windows (SPSS Statistics, IBM Corp, Armonk, NY). Categorical variables were evaluated using the Kolmogorov-Smirnov test. Among the variables, those that fit the normal distribution were presented as the mean ± standard deviation, and those that did not fit the normal distribution were presented as median (interval) or median (interquartile range [IQR]). We used Mann-Whitney U and Kruskal-Wallis tests to compare numerical variables. Spearman correlation analysis was used to assess the relationship between AAS, their sources (mentions), and WOS and Google Scholar citation numbers. Spearman correlation analysis was used to analyze distributions of AAS, the source of the scores (mentions), and WOS and Google Scholar citation numbers. We interpreted the correlations as weak, moderate, strong, and very strong based on the resulting coefficients.
¹⁴
Statistical significance was recognized when P < 0.05.

RESULTS

Of the 40,840 research outputs evaluated, 510,047 were shared across multiple platforms. Altmetric Attention Scores were present for 36,719 articles (89.9%), while 10.1% had no score. The online engagement for this content included 459,391 tweets from 114,708 unique tweeters in 206 countries, 14,355 Facebook posts on 2,141 unique pages in 60 countries, 19,571 news stories by 1,988 unique outlets in 86 countries, and 2,791 policy documents from 79 unique sources in 18 countries. The top 10 journals had no mentions on Pinterest, Syllabi, or LinkedIn. Only two journals received mentions on the Chinese microblogging platform Weibo, each mentioned once. Figure illustrates the frequency of mentions corresponding to the publication years.

The Altmetrics–X demographics data revealed the top five countries contributing to mentions for the analyzed content. The country was not specified in 193,148 posts (42.0%) and 54,517 profiles (47.5%). The largest number of posts were from the United States (Table 1).

Overall, 90% of the 36,780 articles had an AAS ≥ 1. The median (IQR) values for these articles were as follows: AAS 3 (1–9), X mentions 3 (1–9); Mendeley readership 24 (9–50); Dimensions citations 5 (1–15); and blog mentions 0 (0–24). We found a strong correlation between AAS and X mentions (r_s = 0.712, 95% CI 0.707–0.717, P < .001) and a weak correlation between AAS and the number of Mendeley readers (r_s = 0.338 to 356, P < .001).

We found two journals that lacked AAS. The median total mentions for EM journals were 14,154 (ranging from 57–77,460). Mentions on X had a median of 12,284 (ranging from 56–68,078), Facebook mentions had a median of 234 (ranging from 0–2,180), blog mentions had a median of 126.5 (ranging from 0–1,866), and news mentions had a median of 288 (ranging from 0–5,040).

The EM journals with the highest total number of mentions were Annals of Emergency Medicine, Resuscitation, and the American Journal of Emergency Medicine. Conversely, the AAS per article was highest for Academic Emergency Medicine, Annals of Emergency Medicine, and the Emergency Medicine Journal, respectively (Table 2). We found a moderate correlation (r_s = 0.518, P = 0.07, 95% CI 0.152–759) between the 2022 JCI and the total number of mentions.

In the review of the top 200 articles with the highest AAS, the median score of the articles was 382.5 (IQR 301.3–510.8). The AAS with sources and number of citations of the top 50 articles are provided in Table 3. The median WOS citations for the articles was 16 (IQR 5–39), with an annual citation count of 4.7 (IQR 1.8–8.9). The median Google Scholar citations was 29 (IQR 11–65), with an annual citation count of 7.5 (IQR 3.2–15.1). Among these articles, 38% were observational studies, while case reports and reviews/meta-analyses constituted 13%. The most common research topics were ED management and COVID-19 (Table 4).

The AAS of reviews and guidelines were lower, while observational studies and case presentations were higher (P = 0.02). The AAS was higher for topics related to toxicology, COVID-19, and critical care (P = 0.02). While there was an increase in AAS and WOS citations in COVID-19-related papers, these variations were not significant (P = 0.09 and P = 0.08, respectively).

There was no significant correlation between AAS and WOS citation numbers (r_s = −0.041, P = 0.56, 95% CI −0.175–0.087, r_s
² = 0.0017) or Google Scholar citations (r_s = −0.038, P = 0.59, 95 % CI −0.174–0.101, r_s
² = 0.0014). However, there was a very strong positive correlation between WOS and Google Scholar citation numbers (r_s = 0.973, P < .001, 95% CI 0.955–0.984, r_s
² = 0.947). Despite the weak correlation identified between WOS citations and X mentions, and the moderate correlation observed for blog mentions (r_s = 0.330, P < .001, 95 % CI 0.174 to 0.458, r_s
² = 0.109, and r_s = 0.452, P < .001, 95% CI 0.320 to 0.566, r_s
² = 0.204, respectively), there was a very strong positive correlation observed in the number of Mendeley readers. (r_s = 0.873, P < .001, 95% CI 0.822–0.911, r_s
² = 0.762). No correlation was observed between news mentions (r_s = −0.107, P = 0.10, 95% CI −0.246–0.046, r_s
² = 0.0012), and video mentions (r_s = 0.037, P = 0.60, 95% CI −0.078 to 0.145, r_s
² = 0.0013).

DISCUSSION

This study examined altmetrics of EM journal articles from the 2013–2023. Ten percent of EM journal articles were never mentioned on social media. Compared to Barbic et al’s investigations from 2011, AAS for the most cited publications have increased significantly in the subsequent decade.
⁹
Social media followers for EM journal articles have increased significantly in recent years. Interestingly, three-quarters of the 200 most-cited articles were published after this study, with 45% published after the COVID-19 pandemic. Social media followers for EM journal articles have increased significantly in recent years.

Kolahi et al identified a weak but positive correlation between AAS and citations in their meta-analysis; the authors emphasized the importance of continuing to examine the temporal dynamics of this relationship.
¹⁵
In our study, although no correlation was found between AAS and traditional citation counts, we observed a weak correlation between AAS and X mentions, and a moderate correlation between AAS and blog mentions. Notably, there was a very strong correlation observed between AAS and the Mendeley readership numbers.

The AAS is calculated based on the source and frequency of sharing. In this calculation, news, blog mentions, Wiki pages, policy documents, and patents have the most weight, while X has less weight. Mendeley readership and citations are not considered.
¹⁶
It should be noted that the primary purpose of altmetrics is to measure social interest in a given topic, rather than to predict the potential citation count of an article. Traditional citations remain the gold standard for academic recognition. However, the relationship between social media and citations supports the positive impact of researchers and scientific journals using social media to enhance the visibility and influence of their articles. Incorporating the impact of social media into the gold standard of citation counts could be a way to acknowledge this evolving landscape.

The results of a study examining the impact of promoting Cochrane systematic reviews in the field of pediatric EM using X and blog posts revealed a significant increase of 10 times in the AAS of the reviews.
¹⁷
The distribution pattern of articles on social media might vary based on the nature of the sharing. Infographics are visual representations of data meant to enhance engagement and streamline the key elements of a given study.
¹⁸
Some data suggests that presenting research findings visually on social media may lead to a 5–7 times higher number of interactions compared to studies without visual content.
<sup>19

,

20</sup>
Although using visual presentations to share results can reach a larger audience, this effect may only apply to specific areas of expertise.

The extent to which sharing influences the number of downloads and citations of a paper remains uncertain.
²¹
However, altmetrics today play a crucial role as markers for assessing the spread of content via social media to reach the intended audience. Temporal patterns in article altmetrics exhibit variation across different data sources. A study investigating altmetrics temporal trends reported that X engagement started and ended quickly, while Mendeley readership increased steadily over the next few years.
²²
An excellent way to maintain interest in published articles is to use altmetrics data sources in combination with methods that engage the target audience and regularly update the content.

During the COVID-19 pandemic, there was a notable increase in the dissemination of information on social media platforms, with healthcare professionals using these platforms more frequently. Our analysis shows that nearly half of the top 200 publications in EM journals were published after the onset of the pandemic. Additionally, when evaluating the comprehensive altmetrics of articles in EM journals, we found there was a clear rise in mentions during 2020–2021. The evaluation undertaken in this study encompasses references made until the start of 2023. While future studies will determine whether this upward trend will persist, it is foreseeable that the surge in researchers using social media to monitor scientific information will continue as a result of the COVID-19 pandemic. This rise can be linked to the surge in sharing activities associated with COVID-19. Nevertheless, our investigation found no discernible distinction between AAS and WOS citations when comparing papers linked to COVID-19 and those unrelated to it.

LIMITATIONS

This study has several limitations, particularly its emphasis on quantitative data analysis and the use of a single data source. The current Altmetric database mainly emphasizes the number of mentions, and our analysis exclusively compared these metrics. In approximately 50% of the mentions on X we were unable to determine the country associated with the account. In addition to the increased interaction of the attributes of social media shares, it is more important to evaluate the relationship of the information to the target audience, reference, and download.
²⁰
In the early part of the study, we analyzed all articles published in EM journals, but we made comparisons with traditional references for only 200 articles. In contrast to previous studies, we evaluated articles with the highest AAS instead of the altmetrics of the most cited articles in traditional reference indices. Another limitation of this study is the inability to fully differentiate the impact of increased social media usage during the COVID-19 pandemic on AAS. The surge in online content and interactions during the pandemic may have artificially inflated AAS values, particularly for articles published during this period, potentially affecting the relationship between AAS and traditional citation counts.

CONCLUSION

There has been a notable rise in Altmetrics Attention Scores in recent years, driven by increased use of social media for following scientific research, particularly during the COVID-19 pandemic. Articles focusing on toxicology, COVID-19, and resuscitation/critical care tend to receive the highest AAS. While no correlation was found between total AAS and citation counts from WOS and Google Scholar, there is a strong positive correlation between WOS citations and the number of Mendeley readers. Additionally, weak and moderate correlations were observed for mentions on X and blogs, respectively. Further research is needed to explore the relationship between altmetrics and traditional citation metrics, as well as the impact of social media on academic research visibility in EM.

Footnotes

Section Editor: Soheil Saadat, MD, MPH, PhD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Murat Çetin, MD, Dr. Behçet Uz Children’s Education and Research Hospital, Department of Emergency Medicine, Konak, 35210, Izmir, Türkiye. Email: muratcetinn@gmail.com
03 / 2025; 26:353 – 363

Submission history: Revision received May 15, 2024; Submitted December 6, 2024; Accepted December 13, 2024

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

Share on Facebook

Tweet

Share via email