Author	Affiliation
Matthew I. Hwang, BS	University of Illinois College of Medicine at Peoria, Peoria, Illinois
William F. Bond, MD, MS	University of Illinois College of Medicine at Peoria, OSF HealthCare, Jump Simulation and Department of Emergency Medicine, Peoria, Illinois
Emilie S. Powell, MD, MBA, MS	Northwestern University Feinberg School of Medicine, Northwestern Memorial Hospital, Department of Emergency Medicine, Chicago, Illinois

Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Sepsis is defined as life-threatening organ dysfunction due to a dysregulated inflammatory response to infection.¹ It is implicated in an estimated 1.7 million hospitalizations each year and is among the most costly conditions for hospitals.^2,3 Delays in diagnosis of sepsis can lead to delay in treatment,^4,5 which can lead to increased morbidity and mortality.⁶ Quality measures now track time to these treatments as process markers of successful care.⁷ While studies have questioned some of the interventions, such as protocol-driven fluid resuscitation,⁸ there is general agreement that early antibiotic administration reduces mortality from sepsis.^6,9-11

Risk for delays in diagnosis led to the development of automatic electronic sepsis alerts built into electronic health record (EHR) systems.^10,12,13 Some of these systems were created for use in the inpatient ward,^14,15 intensive care unit (ICU),^16,17 and emergency department (ED),^18,19 and some stretch across settings within a healthcare system.^20,21 One study demonstrated that over 75% of sepsis hospitalizations presented in the ED, warranting a focused study of this population.²²

The challenge of demonstrating the marginal impact of these systems is that they act alongside existing sepsis care processes in a very ill population whose incremental change in mortality may be difficult to detect. In addition, thanks to education campaigns for staff,¹⁰ the drive toward improvement in quality measures,²³ and increasing board certification of emergency providers,²⁴ ED personnel have become better trained and are likely better at detecting sepsis. Thus, in the highly visually and electronically monitored ED setting, the benefit of these systems over clinician gestalt may diminish over time.

The possibility still exists that automated sepsis alerts may be an important method to detect more subtle cases or earlier presentations and may have greater value in less monitored settings. The value of these alert systems is measured based on their detection accuracy, with a goal of high sensitivity and, more importantly, their impact on process or outcome measures. However, alert systems carry a risk of alarm fatigue and distraction.^25,26 Sepsis alerts add to already increasing alarms with the EHR, including those for physiology monitors, pharmacy checking, and infectious disease isolation. The positive impact of these automated sepsis alerts and their alarm methods on sepsis care, specific to the ED, remains an open question, and drove the desire for this systematic review.

Alert systems vary in their criteria. Early systems were often rule-based using the Centers for Medicare and Medicaid Services (CMS) Sepsis-1 definition of sepsis: two of four systemic inflammatory response syndrome (SIRS) criteria with a suspected or identified infection source. SIRS is defined as at least two of the following four findings: temperature >38° Celsius (C) (100.4° Fahrenheit [F]) or <36°C (96.8°F); heart rate >90 beats/minute; respiratory rate >20 breaths/minute; or white blood count >12,000 per microliter (μL) or <4000/μL or 10% band forms.¹ CMS with sepsis-2 set elevated temperature at >38.3°C (100.9°F).²⁷ More advanced systems are using algorithms, which expand on the limited criteria of rule-based systems. Such criteria may include past medical history and lab values or vitals with near-real time updating.

Evaluation of the success of these systems is complicated by difficulty establishing consensus²⁸ and evolving definitions for the sepsis spectrum, including the 2016 update to sepsis-3.¹ Thus, the diagnostic criteria are both evolving and in most cases based on discharge diagnosis, rather than information available in the ED. The ability to accurately diagnose and treat a specific disease may be measured by studying discharge diagnosis, but it may not account for clinician decisions made with limited information, as is often encountered in ED settings. Discharge diagnosis as a standard does not account for a clinician’s ability to risk stratify and exclude life- threatening conditions, which is valuable for stabilizing patients and completing the diagnostic workup. Although using chief complaint for quality evaluation or diagnostic criteria has been proposed, it has yet to be standardized.^29,30

 

Population Health Research Capsule

What do we already know about this issue?

The use of automated clinical alerts is increasing, and complex algorithmic models are now being implemented.

What was the research question?

How do sepsis alert systems in the emergency department perform based on accuracy and quality measures?

What was the major finding of the study?

Process measures moderately improved. One low-quality study showed mortality benefit, while no high-quality studies did.

How does this improve population health?

Further research of alert system elements is needed. Our goal is to guide the development of sepsis alerts to improve outcome measures.

 

Due to evolving systems and definitions, we systematically reviewed studies assessing the effectiveness of these alerts. Our objectives were to determine whether automated electronic sepsis alerts in the ED are accurate and whether they have an impact on quality measures and/or mortality.

METHODS

This review followed guidelines presented by the Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies (PRISMA-DTA) and PRISMA-P.^31,32 This review was registered with PROSPERO (Prospective Register of Systematic Reviews).

Search Strategy

Databases for the search included PubMed MEDLINE, Embase, the Cochrane Library, and the Cumulative Index of Nursing and Allied Health Literature (CINAHL), from the earliest available dates to August 2, 2018. We defined the search according to four fields: emergency department; sepsis; electronic health record; and alerts/alarms. Details of the search strategy are described in Appendix A.

Eligibility Criteria

Randomized trials, performance improvement trials (including before and after studies), and cohort studies were included in the screening. Eligible studies included published articles with the following: 1) adult patients in the ED, diagnosed with sepsis, severe sepsis, or septic shock (hereafter referred to as sepsis); 2) an electronic system that alerts a healthcare provider of sepsis in real or near-real time; and 3) measures of diagnostic accuracy or impact on quality of care measures. Exclusion criteria included the following: 1) primary data based on non-ED settings, such as prehospital, ICU, or the general wards; 2) articles studying medical conditions that can present with sepsis, such as specific infections (eg, influenza), pregnancy-related issues, and bacteremia, without assessing sepsis independently; 3) alert systems that screen only at triage, as opposed to reaching an alert trigger threshold at any point in the ED visit; and 4) non-English language articles lacking translation. We ensured chosen articles came from peer-reviewed sources based on the presence of a peer-review process description on the journal homepage.

Study Records

We collected citations in a reference manager software Zotero (Corporation of Digital Scholarship, George Mason University, Fairfax, VA). Article screening was completed through the online software Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia). Two independent reviewers (authors WB and MH), selected for articles based on the inclusion and exclusion criteria in the title and abstract screenings. At the next stage, two independent reviewers (authors WB and EP) selected articles in the full-text screening. Conflicts were resolved through regular meetings or conference calls. Data was collected by WB and MH, and then extracted with Covidence to be stored as a secure Microsoft Excel file (Microsoft Corporation, Redmond, WA).

Data Items

Qualitative data items for extraction included clinical setting, study design, age group, type of alert system, definition/threshold for the alert, method of alert notification, treatment recommendation, and reference standard. The implemented alert system was considered the index test. We classified the alert systems as rule based or algorithm based. Among the eligible studies, the rule-based alerts used SIRS criteria. The algorithmic alerts had unique measures such as vitals, Glasgow Coma Scale, and creatinine. Variations for either system are described in Table 1. Quantitative data items included sample size, population size, accuracy, and outcome measures.

Outcomes and Prioritization and Diagnostic Accuracy Measures

We extracted data from articles on sepsis alerts for both diagnostic accuracy and impact on quality measures. Diagnostic accuracy assesses the ability of the alert to accurately detect sepsis. Measurements included positive and negative predictive values, sensitivity, and specificity. Quality measures of interest were process and outcome measures. Examples of process markers included compliance or time to antibiotic administration, fluid resuscitation, and lactate measurement. Outcome measures included mortality and length of ICU stay, although various additional markers were captured by different authors. When reported by the authors, we used confidence intervals for the given estimates.

Data Synthesis

A qualitative analysis of each study was used. The variation of sepsis definition for the alerts, the set points, methods of alerting, response processes, etc prevented an aggregated quantitative analysis.

Bias and Applicability

Covidence included a bias rating system based on the Cochrane standard of quality assessment. We added criteria from the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) to effectively assess diagnostic accuracy of the articles, per the recommendation of PRISMA-DTA, Leeflang, and Cochrane.^31,33,34 We rated quality measure articles following the guidance of GRADE (Grades of Recommendation Assessment, Development and Evaluation).³⁵ Each article was rated for bias regarding blinding of participants and personnel to the alert, blinding of outcome assessors, incomplete outcome data, selective outcome reporting, the index test, gold standard, and flow and timing. Once each component was finalized, a consensus overall quality rating was decided based on the risk of biases. The overall quality was scaled relative to the cohort study design. No articles had strong experimental designs (ie, randomized controlled trials); therefore, quality was ranked based on comparison within this cohort of articles. Details are recorded in Appendix B.

RESULTS

Study Selection and Characteristics

We imported 731 articles into Covidence. After duplicate removal, 693 were screened by title and abstract. Twenty articles underwent full-text assessment, and 10 were selected for the study (Figure).

Eight of these studies assessed diagnostic accuracy and six assessed quality measures. All studies were prospective or retrospective cohorts and were conducted in urban, tertiary and/or academic medical centers (Table 1). Publishing years ranged from 2009–2018. Two studies had algorithm-based alert systems, while eight had rule sets. All systems used different criteria based on SIRS to define sepsis. There was significant variability in the criteria used for activation of the sepsis alert, the threshold definitions that activated the alert, the presence or absence of triggering links to care order sets, and the degree and type of interventions triggered by the alert. Likewise, there were variations in the diagnostic criteria standards against which the alerts were weighed, with most studies using chart review confirmation, while some used clinician confirmation. Only Nguyen et al had a control group of 300 randomly selected patients during a study period when the alert did not fire.⁴⁴ All of the other articles were either prospective or retrospective cohort designs without control groups.

Diagnostic Accuracy

Diagnostic accuracy was recorded in Table 2 below. Specificity ranged from 78–99%, and positive predictive value (PPV) from 5.8% to 54%. Negative predictive value (NPV) was consistently high at 99–100%. Excluding Meurer et al,⁴¹ sensitivity ranged from 64–100%. Meurer et al had a sensitivity of 33.3% for the electronic alert alone, and 10.7% for the electronic alert and attending confirmation. With attending confirmation, specificity increased from 78.0% to 97.6%. The study had a low activation threshold of ≥2 SIRS criteria, the smallest sample size of 84, and an age range of 70 years or older. Patients were only included if they presented between 3 am and 9 pm on weekdays. This study also only included patients admitted from the ED, instead of all ED patients, risking selection bias. The notification system sent a page to the study coordinator before confirming with a physician.

In contrast, other studies directly notified a member of the clinical team, excluding Nguyen et al, which did not describe the notification method.⁴⁴ Five rule-based studies were of high quality.^{18,36,37,43,44} Two studies had systems with high accuracy. Alsolamy et al¹⁸ had a sensitivity of 93.2%, specificity 98.4%, and PPV 21.0%. Bansal et al³⁷ had a sensitivity of 100%, specificity 96.2%, and PPV 29.3%. Highest PPV was achieved by Nelson et al⁴³ at 54.0% and Nguyen et al⁴⁴ at 44.7%.

Austrian et al³⁶ shared the number of total alerts fired, for any of three criteria sets including SIRS, nurse alert, and physician alert that included progressively more ill criteria. They report sensitivities of 73.0%, 23.8%, and 23.0%, respectively, and PPV of 13.0%, 22.4%, and 26.6% as expected for the more progressively stringent criteria. They did not share the denominator of all ED presenting patients for the retrospective period under study but report the total number of hospitalized sepsis patients based on discharge diagnosis. Septic patients may have been sent home, but if we assume they captured all true positives and false negatives through final diagnosis of sepsis, this allows for calculation of sensitivity and PPV and does not allow the calculation of specificity or NPV. With 2144 patients with a final diagnosis of severe sepsis or septic shock, and 97,216 alerts (any of the three levels included), they had the largest retrospective sample size.

Two studies assessing algorithm-based alerts were deemed low quality. Brown et al³⁹ measured a sensitivity of 76.4%, specificity of 95.3%, and a low PPV of 5.8%. Martin Rico et al⁴⁰ measured a sensitivity of 85%, specificity of 89%, and a PPV of 19%. Prevalence of sepsis compared to total ED patients was 0.3–2% in five studies.^{18,37,39,40,43} Meurer et al had a prevalence of 14.4%, but this was among patients ≥70 years old and it was the sole study with only SIRS criteria (a low threshold) for its sepsis definition.

Quality Measures

Quality measures are described in Table 3. Two studies evaluating quality measures were high quality: Austrian et al³⁶ and Nelson et al.⁴³ In Austrian et al, process markers of time to first lactate and vasopressor use significantly improved. Statistically insignificant findings included blood cultures drawn before antibiotic administration and whether a lactate was drawn. Antibiotic timing was not reported. For Nelson et al, process markers of blood culture collection and chest radiograph before admission improved. Statistically insignificant findings included lactate collection and antibiotics given in the ED. Outcome measures of ICU transfer, ICU length of stay (LOS), and total LOS significantly improved for Austrian et al. Mortality did not improve significantly for either study.

Four studies (Bansal, Berger, Martin Rico, Narayanan)^37,38,40,42 were judged to be of low quality. Berger et al had significant improvement in lactate testing. Narayanan et al improved antibiotics within 60 minutes, time to antibiotics, and LOS. Bansal et al³⁷ had nearly 100% sensitivity and specificity with a modest PPV of 29.3%, and had no significant improvements in survival rate. Of note, we established the article to be high quality in regard to diagnostic accuracy, while outcome measures were low quality. In contrast, Austrian et al³⁶ had improvements in LOS and Martin Rico et al⁴⁰ had improvements in mortality, while both exhibited moderate accuracy.

None of the rule-based studies showed statistically significant improvements in mortality.^36-38,42,43 The only outcome reported by an algorithm-based study (Martin Rico et al)⁴⁰ was mortality, which showed significant improvement, although the study was judged to be of low quality. The alert system Narayanan et al⁴² studied did not recommend treatment as other systems did. For this rule-based system “antibiotics in 60 minutes” meant time to antibiotics, and LOS significantly improved.

DISCUSSION

Overall, most of the study designs used to assess the impact of sepsis alerts were weak, and the review authors had difficulty isolating the impact of the automated sepsis alert itself from broader interventions such as response teams or order set bundles. Thus, our review conclusions must be couched within the strength of the overall low-quality evidence.

The limited evidence available suggests that sepsis alerts in the ED setting can be tuned to a high sensitivity for the detection of sepsis. Evidence from both low- and high-quality studies showed some improved process-of-care markers, including time to antibiotics, with the use of automated sepsis alerts.^{36,38,40,42,43} Lactate testing was studied by four groups with two producing significant results. Other than lactate measurement, no single measure consistently improved across studies. A lack of consistency of measured items and measurement methods creates a challenge in forming a conclusion. For example, one study examined whether blood cultures were collected, as opposed to blood cultures collected before antibiotic administration.

No high-quality studies showed a difference in mortality, and only one high-quality study showed impacts on ICU LOS and vasopressor use.³⁶ Our findings are in keeping with a review by Makam in 2015 that covered alerts both inside and outside of the ED environment.⁴⁵ Our review added recently published articles, including those that now use an algorithmic as opposed to simple rules-based approaches, and was focused on patients presenting to the ED. The strongest study designs we reviewed for inclusion were prospective cohort studies, but we would call attention to a well-executed performance improvement study conducted by Gatewood et al.⁴⁶ They included a computerized alert with a multipronged intervention and showed a substantial improvement in sepsis bundle of care compliance. However, they did not show differences in mortality in part due to the inclusion of lower risk patients on the sepsis spectrum.

Sepsis alerts represent a difficult area to study with traditional randomized methods. One challenge is that in the course of operational improvement, sepsis alert criteria and/or alert thresholds may be subtly changed in the background. This may be done by information technology, analytics, or EHR personnel to address PPV or safety concerns, usually with a clinician’s input, but often without alerting all ED staff to the change. Moving to a more rigorous study design requires holding the alert constant and ethical approval for a non-alert or clinician gestalt arm. Thus, success will likely be found in future studies that use time series, or perhaps cluster randomized rollout methods across healthcare systems. Likewise, future areas for study could include comparisons of the method of alert, and the presence or absence of treatment recommendations.

None of the studies addressed potential harms. Harm may include the alarm issues impacting staff, missing alternative diagnoses due to early anchoring on sepsis, and the follow-on effects of early, aggressive fluid intervention, which has been questioned more broadly in the sepsis literature.⁸ Significant further work is needed on the alarm consequences of the sensitivity set points, and if possible, such work should incorporate influences from other non-sepsis alarms in alarm fatigue.

Although low quality, one algorithmic system showed significant mortality improvement, potentially validating its further development.⁴⁰ Systems such as this are being developed to improve accuracy and PPV, and may include risk factors such as comorbid conditions and past medical history. These systems can effectively insert multiple variables into an equation using current and past patient data as regression coefficients, running the calculation repeatedly over the course of a patient stay as more predictor variables become available. The data creating the coefficients of such a regression-based equation would influence the predictor’s value. For example, a sepsis predictor tool based on the elderly would likely not be predictive for children. The newest models of sepsis alerts include machine learning. Complex algorithmic models may use well over 50 variables, and a machine-learning program may be integrated into them. Machine learning uses computer programming to identify patterns and significant predictors beyond the reasonable capabilities of humans. With continual analysis, it can fine-tune coefficients and thresholds of the algorithm. Initial studies show promise,^47-49 and additional research is required to assess its impact on clinical outcomes.

LIMITATIONS

Our limitations include a risk of publication bias because we did not search the gray literature or clinical trials for studies in progress. There are likely many hospital systems that have implemented sepsis alerts, collected data, and did not report it. Our consensus group was small in number, but we followed a rigorous process using review rubrics guided by well-accepted grading criteria.

CONCLUSION

Automated sepsis alerts in the ED may be set to a high sensitivity. Process measures show moderate benefit; however, no single measure has consistently improved, and high-quality studies have yet to demonstrate, a mortality benefit. Specific components of these systems, alarm fatigue, and sensitivity set points should be examined further. Sepsis alerts demonstrate utility and future research is indicated to build a more ideal alert system.

Footnotes

Section Editor: Gabriel Wardi, MD

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Matthew Hwang, BS, University of Illinois College of Medicine at Peoria, Jump Simulation, 1306 N Berkeley Ave., Peoria, IL 61603. Email: mihwang2@uic.edu. 9 / 2020; 21:1201 – 1210

Submission history: Revision received November 29, 2020; Submitted May 5, 2020; Accepted May 17, 2020

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Singer M, Deutschman CS, Seymour CW, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315(8):801-10.

2. Torio CM, Andrews RM. National Inpatient hospital costs: the most expensive conditions by payer, 2011. 2013. Available at: https://www.hcup-us.ahrq.gov/reports/statbriefs/sb160.pdf. Accessed February 2, 2020.

3. Rhee C, Dantes R, Epstein L, et al. Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009–2014. JAMA. 2017;318(13):1241-9.

4. Candel FJ, Sá MB, Belda S, et al. Current aspects in sepsis approach. Turning things around. Rev Esp Quimioter. 2018;31(4):298-315.

5. Kassyap CK, Abraham SV, Krishnan SV, et al. Factors affecting early treatment goals of sepsis patients presenting to the emergency department. Indian J Crit Care Med. 2018;22(11):797-800.

6. Andersson M, Östholm-Balkhed Å, Fredrikson M, et al. Delay of appropriate antibiotic treatment is associated with high mortality in patients with community-onset sepsis in a Swedish setting. Eur J Clin Microbiol Infect Dis. 2019;38(7):1223-34.

7. Severe sepsis and septic shock: management bundle. 2014. Available at: https://cmit.cms.gov/CMIT_public/ViewMeasure?MeasureId=1017. Accessed April 13, 2020.

8. Peake SL, et al. Goal-directed resuscitation for patients with early septic shock. N Engl J Med. 2014;371(16):1496-1506.

9. Ballester L, Martínez R, Méndez J, et al. Differences in hypotensive vs. non-hypotensive sepsis management in the emergency department: door-to-antibiotic time impact on sepsis survival. Med Sci. 2018;6(4):91.

10. Rhodes A, Evans LE, Alhazzani W, et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016. Intensive Care Med. 2017;43(3):304-77.

11. Johnston ANB, Park J, Doi SA, et al. Effect of immediate administration of antibiotics in patients with sepsis in tertiary care: a systematic review and meta-analysis. Clin Ther. 2017;39(1):190-202.e6.

12. Tafelski S, Nachtigall I, Deja M, et al. Computer-assisted decision support for changing practice in severe sepsis and septic shock. J Int Med Res. 2010;38(5):1605-16.

13. Herasevich V, Pieper MS, Pulido J, et al. Enrollment Into a time sensitive clinical study in the critical care setting: results from computerized septic shock sniffer implementation. J Am Med Inform Assoc. 2011;18(5):639-44.

14. McRee L, Thanavaro JL, Moore K, et al. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546-9.

15. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real-time computerized sepsis alert in non-intensive care unit patients. Crit Care Med. 2011;39(3):469-73.

16. Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit. Crit Care Med. 2012;40(7):2096-101.

17. Brandt BN, Gartner AB, Moncure M, et al. Identifying severe sepsis via electronic surveillance. Am J Med Qual. 2015;30(6):559-65.

18. Alsolamy S, Al Salamah M, Al Thagafi M, et al. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak. 2014;14:105.

19. Balamuth F, Alpern ER, Abaddessa MK, et al. Improving recognition of pediatric severe sepsis in the emergency department: contributions of a vital sign based electronic alert and bedside clinician identification. Ann Emerg Med. 2017;70(6):759-68.e2.

20. Guirgis FW, Jones L, Esma R, et al. Managing sepsis: electronic recognition, rapid response teams, and standardized care save lives. J Crit Care. 2017;40:296-302.

21. Westphal GA, Pereira AB, Fachin SM, et al. An electronic warning system helps reduce the time to diagnosis of sepsis. Rev Bras Ter Intensiva. 2018;30(4):414-22.

22. Angel C, Leisman DE, Schneider SM, et al. Sepsis presenting in emergency department versus inpatient settings: divergences in prevalence, patient characteristics, initial resuscitation, outcomes, and costs. Ann Emerg Med. 2018;72(4 Supplement):S149.

23. Ramsdell TH, Smith AN, Kerkhove E. Compliance with updated sepsis bundles to meet new sepsis core measure in a tertiary care hospital. Hosp Pharm. 2017;52(3):177-86.

24. Reiter M, Wen LS, Allen BW. The emergency medicine workforce: profile and projections. J Emerg Med. 2016;50(4):690-3.

25. Guardia-LaBar LM, Scruth EA, Edworthy J, et al. Alarm fatigue: the human-system interface. Clin Nurse Spec. 2014;28(3):135-7.

26. Curry JP, Jungquist CR. A critical assessment of monitoring practices, patient deterioration, and alarm fatigue on inpatient wards: a review. Patient Saf Surg. 2014;8(1):29.

27. Sartelli M, Kluger Y, Ansaloni L, et al. Raising concerns about the sepsis-3 definitions. World J Emerg Surg. 2018;13:6.

28. Kalantari A, Mallemat H, Weingart SD. Sepsis definitions: the search for gold and what CMS got wrong. West J Emerg Med. 2017;18(5):951-6.

29. Griffey RT, Pines JM, Farley HL, et al. Chief complaint-based performance measures: a new focus for acute care quality measurement. Ann Emerg Med. 2015;65(4):387-95.

30. Advancing chief complaint-based quality measurement. 2019. Available at: http://www.qualityforum.org/Publications/2019/06/Advancing_Chief_Complaint-Based_Quality_Measurement_Final_Report.aspx. Accessed April 13, 2020.

31. McInnes MDF, Moher D, Thombs BD, et al. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: the PRISMA-DTA statement. JAMA. 2018;319(4):388-96.

32. Moher D, Shamseer L, Clarke M, et al. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 Statement. Syst Rev. 2015;4(1):1.

33. Leeflang MMG, Deeks JJ, Gatsonis C, et al. Systematic reviews of diagnostic test accuracy. Ann Intern Med. 2008;149(12):889-97.

34. Deeks JJ, Wisniewski S, Davenport C. Guide to the contents of a Cochrane diagnostic test accuracy protocol. 2013. Available at: https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/public/uploads/Ch04_Sep2013.pdf. Accessed March 2, 2020.

35. Dijkers M. Introducing GRADE: a systematic approach to rating evidence in systematic reviews and to guideline development. KT Update. 2013;1(5):9.

36. Austrian JS, Jamin CT, Doty GR, et al. Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. J Am Med Inform Assoc. 2018;25(5):523-9.

37. Bansal V, Festić E, Mangi MA, et al. Early machine-human interface around sepsis severity identification: from diagnosis to improved management?. Acta Med Acad. 2018;42(1):27-38.

38. Berger T, Birnbaum A, Bijur P, et al. A computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394-407.

39. Brown SM, Jones J, Kuttler KG, et al. Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department. BMC Emerg Med. 2016;16(1):31.

40. Martin Rico P, Valdivia Perez A, Lacalle Martinez JM. Electronic alerting and decision support for early sepsis detection and management: impact on clinical outcomes. Eur J Clin Pharmacol. 2017;19(1):33-40.

41. Meurer WJ, Smith BL, Losman ED, et al. Real-time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):40-5.

42. Narayanan N, Gross AK, Pintens M, et al. Effect of an electronic medical record alert for severe sepsis among ED patients. Am J Emerg Med. 2016;34(2):185-8.

43. Nelson JL, Smith BL, Jared JD, et al. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500-4.

44. Nguyen SQ, Mwakalindile E, Booth JS, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.

45. Makam AN, Nguyen OK, Auerbach AD. Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review. J Hosp Med. 2015;10(6):396-402.

46. Gatewood MO, Wemple M, Greco S, et al. A quality improvement project to improve early sepsis care in the emergency department. BMJ Qual Saf. 2015;24(12):787-95.

47. Delahanty RJ, Alvarez J, Flynn LM, et al. Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis. Ann Emerg Med. 2019;73(4):334-44.

48. Shimabukuro DW, Barton CW, Feldman MD, et al. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir Res. 2017;4(1):e000234.

49. Nemati S, Holder A, Razmi F, et al. An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit Care Med. 2018;46(4):547-53.