|Matthew I. Hwang, BS||University of Illinois College of Medicine at Peoria, Peoria, Illinois|
|William F. Bond, MD, MS||University of Illinois College of Medicine at Peoria, OSF HealthCare, Jump Simulation and Department of Emergency Medicine, Peoria, Illinois|
|Emilie S. Powell, MD, MBA, MS||Northwestern University Feinberg School of Medicine, Northwestern Memorial Hospital, Department of Emergency Medicine, Chicago, Illinois|
For early detection of sepsis, automated systems within the electronic health record have evolved to alert emergency department (ED) personnel to the possibility of sepsis, and in some cases link them to suggested care pathways. We conducted a systematic review of automated sepsis-alert detection systems in the ED.
We searched multiple health literature databases from the earliest available dates to August 2018. Articles were screened based on abstract, again via manuscript, and further narrowed with set inclusion criteria: 1) adult patients in the ED diagnosed with sepsis, severe sepsis, or septic shock; 2) an electronic system that alerts a healthcare provider of sepsis in real or near-real time; and 3) measures of diagnostic accuracy or quality of sepsis alerts. The final, detailed review was guided by QUADAS-2 and GRADE criteria. We tracked all articles using an online tool (Covidence), and the review was registered with PROSPERO registry of reviews. A two-author consensus was reached at the article choice stage and final review stage. Due to the variation in alert criteria and methods of sepsis diagnosis confirmation, the data were not combined for meta-analysis.
We screened 693 articles by title and abstract and 20 by full text; we then selected 10 for the study. The articles were published between 2009–2018. Two studies had algorithm-based alert systems, while eight had rule-based alert systems. All systems used different criteria based on systemic inflammatory response syndrome (SIRS) to define sepsis. Sensitivities ranged from 10–100%, specificities from 78–99%, and positive predictive value from 5.8–54%. Negative predictive value was consistently high at 99–100%. Studies showed some evidence for improved process-of-care markers, including improved time to antibiotics. Length of stay improved in two studies. One low quality study showed improved mortality.
The limited evidence available suggests that sepsis alerts in the ED setting can be set to high sensitivity. No high-quality studies showed a difference in mortality, but evidence exists for improvements in process of care. Significant further work is needed to understand the consequences of alert fatigue and sensitivity set points.
Sepsis is defined as life-threatening organ dysfunction due to a dysregulated inflammatory response to infection.1 It is implicated in an estimated 1.7 million hospitalizations each year and is among the most costly conditions for hospitals.2,3 Delays in diagnosis of sepsis can lead to delay in treatment,4,5 which can lead to increased morbidity and mortality.6 Quality measures now track time to these treatments as process markers of successful care.7 While studies have questioned some of the interventions, such as protocol-driven fluid resuscitation,8 there is general agreement that early antibiotic administration reduces mortality from sepsis.6,9-11
Risk for delays in diagnosis led to the development of automatic electronic sepsis alerts built into electronic health record (EHR) systems.10,12,13 Some of these systems were created for use in the inpatient ward,14,15 intensive care unit (ICU),16,17 and emergency department (ED),18,19 and some stretch across settings within a healthcare system.20,21 One study demonstrated that over 75% of sepsis hospitalizations presented in the ED, warranting a focused study of this population.22
The challenge of demonstrating the marginal impact of these systems is that they act alongside existing sepsis care processes in a very ill population whose incremental change in mortality may be difficult to detect. In addition, thanks to education campaigns for staff,10 the drive toward improvement in quality measures,23 and increasing board certification of emergency providers,24 ED personnel have become better trained and are likely better at detecting sepsis. Thus, in the highly visually and electronically monitored ED setting, the benefit of these systems over clinician gestalt may diminish over time.
The possibility still exists that automated sepsis alerts may be an important method to detect more subtle cases or earlier presentations and may have greater value in less monitored settings. The value of these alert systems is measured based on their detection accuracy, with a goal of high sensitivity and, more importantly, their impact on process or outcome measures. However, alert systems carry a risk of alarm fatigue and distraction.25,26 Sepsis alerts add to already increasing alarms with the EHR, including those for physiology monitors, pharmacy checking, and infectious disease isolation. The positive impact of these automated sepsis alerts and their alarm methods on sepsis care, specific to the ED, remains an open question, and drove the desire for this systematic review.
Alert systems vary in their criteria. Early systems were often rule-based using the Centers for Medicare and Medicaid Services (CMS) Sepsis-1 definition of sepsis: two of four systemic inflammatory response syndrome (SIRS) criteria with a suspected or identified infection source. SIRS is defined as at least two of the following four findings: temperature >38° Celsius (C) (100.4° Fahrenheit [F]) or <36°C (96.8°F); heart rate >90 beats/minute; respiratory rate >20 breaths/minute; or white blood count >12,000 per microliter (μL) or <4000/μL or 10% band forms.1 CMS with sepsis-2 set elevated temperature at >38.3°C (100.9°F).27 More advanced systems are using algorithms, which expand on the limited criteria of rule-based systems. Such criteria may include past medical history and lab values or vitals with near-real time updating.
Evaluation of the success of these systems is complicated by difficulty establishing consensus28 and evolving definitions for the sepsis spectrum, including the 2016 update to sepsis-3.1 Thus, the diagnostic criteria are both evolving and in most cases based on discharge diagnosis, rather than information available in the ED. The ability to accurately diagnose and treat a specific disease may be measured by studying discharge diagnosis, but it may not account for clinician decisions made with limited information, as is often encountered in ED settings. Discharge diagnosis as a standard does not account for a clinician’s ability to risk stratify and exclude life- threatening conditions, which is valuable for stabilizing patients and completing the diagnostic workup. Although using chief complaint for quality evaluation or diagnostic criteria has been proposed, it has yet to be standardized.29,30
Population Health Research Capsule
What do we already know about this issue?
The use of automated clinical alerts is increasing, and complex algorithmic models are now being implemented.
What was the research question?
How do sepsis alert systems in the emergency department perform based on accuracy and quality measures?
What was the major finding of the study?
Process measures moderately improved. One low-quality study showed mortality benefit, while no high-quality studies did.
How does this improve population health?
Further research of alert system elements is needed. Our goal is to guide the development of sepsis alerts to improve outcome measures.
Due to evolving systems and definitions, we systematically reviewed studies assessing the effectiveness of these alerts. Our objectives were to determine whether automated electronic sepsis alerts in the ED are accurate and whether they have an impact on quality measures and/or mortality.
This review followed guidelines presented by the Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies (PRISMA-DTA) and PRISMA-P.31,32 This review was registered with PROSPERO (Prospective Register of Systematic Reviews).
Databases for the search included PubMed MEDLINE, Embase, the Cochrane Library, and the Cumulative Index of Nursing and Allied Health Literature (CINAHL), from the earliest available dates to August 2, 2018. We defined the search according to four fields: emergency department; sepsis; electronic health record; and alerts/alarms. Details of the search strategy are described in Appendix A.
Randomized trials, performance improvement trials (including before and after studies), and cohort studies were included in the screening. Eligible studies included published articles with the following: 1) adult patients in the ED, diagnosed with sepsis, severe sepsis, or septic shock (hereafter referred to as sepsis); 2) an electronic system that alerts a healthcare provider of sepsis in real or near-real time; and 3) measures of diagnostic accuracy or impact on quality of care measures. Exclusion criteria included the following: 1) primary data based on non-ED settings, such as prehospital, ICU, or the general wards; 2) articles studying medical conditions that can present with sepsis, such as specific infections (eg, influenza), pregnancy-related issues, and bacteremia, without assessing sepsis independently; 3) alert systems that screen only at triage, as opposed to reaching an alert trigger threshold at any point in the ED visit; and 4) non-English language articles lacking translation. We ensured chosen articles came from peer-reviewed sources based on the presence of a peer-review process description on the journal homepage.
We collected citations in a reference manager software Zotero (Corporation of Digital Scholarship, George Mason University, Fairfax, VA). Article screening was completed through the online software Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia). Two independent reviewers (authors WB and MH), selected for articles based on the inclusion and exclusion criteria in the title and abstract screenings. At the next stage, two independent reviewers (authors WB and EP) selected articles in the full-text screening. Conflicts were resolved through regular meetings or conference calls. Data was collected by WB and MH, and then extracted with Covidence to be stored as a secure Microsoft Excel file (Microsoft Corporation, Redmond, WA).
Qualitative data items for extraction included clinical setting, study design, age group, type of alert system, definition/threshold for the alert, method of alert notification, treatment recommendation, and reference standard. The implemented alert system was considered the index test. We classified the alert systems as rule based or algorithm based. Among the eligible studies, the rule-based alerts used SIRS criteria. The algorithmic alerts had unique measures such as vitals, Glasgow Coma Scale, and creatinine. Variations for either system are described in Table 1. Quantitative data items included sample size, population size, accuracy, and outcome measures.
Table 1Characteristics of included studies.
|Source (the article)||Study design||Demographic||Type of alert system (the index test)||Definition/Threshold for the alert||Method of alert notification||Treatment recommended||Reference standard (compared to the index test)|
|Alsolamy 201418||Prospective cohort||>14 years old||Rule based||≥2 SIRS criteria and organ dysfunction, or 2 organ dysfunction criteria^||Notification to nurse who pages the physician||No||Clinical evaluation by an EM or ICU physician following 2012 surviving sepsis campaign guidelines|
|Austrian 201836||Retrospective cohort||≥18 years old||Rule based||1st alert is SIRS based. 2nd and 3rd alerts are sepsis based, which is the SIRS alert plus systolic blood pressure <90mmHg or lactate ≥4 mg/dL||Electronic notifications to the following, Alert 1: nurseAlert 2: nurseAlert 3: provider||Yes (to all 3 alerts)||ICD-9 coding for severe sepsis or septic shock on admission only|
|Bansal 201837||Prospective cohort||Adult patients (though not clearly specified)||Rule based||1st alert is SIRS based. 2nd alert is a sepsis alert, which is the SIRS alert plus WBC ≥12K or ≤4K Blood cultures ordered OR Lactate >4 mg/dL alone||Team leader paged||Yes, a sepsis response team in the post alert group||2 physician reviewers using standardized sepsis criteria, approved by Mayo Clinic enterprise subspecialty councils for EM and critical care|
|Berger 201038||Prospective cohort||≥19 years old||Rule based||>2 SIRS criteria plus infection source||Electronic notification to clinician||Yes, lactate recommended||≥2 SIRS criteria and clinical suspicion, retrospectively|
|Brown 201639||Prospective cohort||≥14 years old||Algorithm based||75 parameters including demographics, encounter details, lab tests, SIRS criteria, and other clinical measurements||Page and email to charge nurse||Not specified||Admitted from ED to ICU and either 1) ICD-9 discharge diagnosis relating to sepsis or infection or 2) identification by a QI coordinator in the ICU.|
|Martin Rico 201740||Prospective cohort||≥14 years old||Algorithm based||Series of parameters including lab tests, SIRS criteria, vitals, and Glasgow coma scale score||Electronic notification to clinician||Yes, with an e-order set||Chart review with “clinical experts” with ICD-9 CM discharge diagnosis of sepsis|
|Meurer 200941||Prospective cohort||≥70 years old||Rule based||≥2 SIRS criteria||Page to study coordinator who confirms a source of infection from the physician||No||Chart reviewers (3) confirmed or excluded the diagnosis|
|Narayanan 201642||Retrospective cohort||≥18 years old||Rule based||1st alert is SIRS based. 2nd alert is a sepsis alert, which is the SIRS alert plus end organ dysfunction or fluid nonresponsive hypotension||Electronic notification to clinician||No||Chart review with ICD-9 code diagnosis of severe sepsis and septic shock|
|Nelson 201143||Prospective cohort||≥18 years old||Rule based||≥2 SIRS criteria and 2 systolic blood pressure measurements less than 90mmHg||All caregivers notified with a page||Yes||Chart review with the same SIRS and hypotension criteria|
|Nguyen 201444||Retrospective cohort||All ED patients*||Rule based||≥2 SIRS criteria, and systolic blood pressure ≤90mmHg or lactic acid ≥2.0mg/dL.||Not specified||Not specified||300 patients for which the alert did not fire were randomly selected|
^Systolic blood pressure <90 to 86 mmHg with intravenous fluids or <86 mm Hg regardless of fluids, blood oxygen saturation <90% to 85% with supplemental oxygen or <85% without oxygen, or lactate >2 mmol/L.
SIRS, systemic inflammatory response syndrome; ICU, intensive care unit; ICD-9, International Classification of Diseases, 9th ed; mmHG, millimeters of mercury; mg/dL, milligram per deciliter; mmol/L, millimole per liter; WBC, white blood count; K, thousand; EM, emergency medicine; ED, emergency department; QI, quality improvement.
*”While children have different ranges for SIRS criteria, <1% of emergency department (ED) patients were <18 years old…”
SIRS, systemic inflammatory response syndrome; ICD-9, International Classification of Diseases, 9th ed; mmHg, millimeters of mercury; mg/dL milligram per deciliter; ED, emergency department.
Outcomes and Prioritization and Diagnostic Accuracy Measures
We extracted data from articles on sepsis alerts for both diagnostic accuracy and impact on quality measures. Diagnostic accuracy assesses the ability of the alert to accurately detect sepsis. Measurements included positive and negative predictive values, sensitivity, and specificity. Quality measures of interest were process and outcome measures. Examples of process markers included compliance or time to antibiotic administration, fluid resuscitation, and lactate measurement. Outcome measures included mortality and length of ICU stay, although various additional markers were captured by different authors. When reported by the authors, we used confidence intervals for the given estimates.
A qualitative analysis of each study was used. The variation of sepsis definition for the alerts, the set points, methods of alerting, response processes, etc prevented an aggregated quantitative analysis.
Bias and Applicability
Covidence included a bias rating system based on the Cochrane standard of quality assessment. We added criteria from the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) to effectively assess diagnostic accuracy of the articles, per the recommendation of PRISMA-DTA, Leeflang, and Cochrane.31,33,34 We rated quality measure articles following the guidance of GRADE (Grades of Recommendation Assessment, Development and Evaluation).35 Each article was rated for bias regarding blinding of participants and personnel to the alert, blinding of outcome assessors, incomplete outcome data, selective outcome reporting, the index test, gold standard, and flow and timing. Once each component was finalized, a consensus overall quality rating was decided based on the risk of biases. The overall quality was scaled relative to the cohort study design. No articles had strong experimental designs (ie, randomized controlled trials); therefore, quality was ranked based on comparison within this cohort of articles. Details are recorded in Appendix B.
Study Selection and Characteristics
We imported 731 articles into Covidence. After duplicate removal, 693 were screened by title and abstract. Twenty articles underwent full-text assessment, and 10 were selected for the study (Figure).
Eight of these studies assessed diagnostic accuracy and six assessed quality measures. All studies were prospective or retrospective cohorts and were conducted in urban, tertiary and/or academic medical centers (Table 1). Publishing years ranged from 2009–2018. Two studies had algorithm-based alert systems, while eight had rule sets. All systems used different criteria based on SIRS to define sepsis. There was significant variability in the criteria used for activation of the sepsis alert, the threshold definitions that activated the alert, the presence or absence of triggering links to care order sets, and the degree and type of interventions triggered by the alert. Likewise, there were variations in the diagnostic criteria standards against which the alerts were weighed, with most studies using chart review confirmation, while some used clinician confirmation. Only Nguyen et al had a control group of 300 randomly selected patients during a study period when the alert did not fire.44 All of the other articles were either prospective or retrospective cohort designs without control groups.
Diagnostic accuracy was recorded in Table 2 below. Specificity ranged from 78–99%, and positive predictive value (PPV) from 5.8% to 54%. Negative predictive value (NPV) was consistently high at 99–100%. Excluding Meurer et al,41 sensitivity ranged from 64–100%. Meurer et al had a sensitivity of 33.3% for the electronic alert alone, and 10.7% for the electronic alert and attending confirmation. With attending confirmation, specificity increased from 78.0% to 97.6%. The study had a low activation threshold of ≥2 SIRS criteria, the smallest sample size of 84, and an age range of 70 years or older. Patients were only included if they presented between 3 am and 9 pm on weekdays. This study also only included patients admitted from the ED, instead of all ED patients, risking selection bias. The notification system sent a page to the study coordinator before confirming with a physician.
Table 2Diagnostic accuracy.
|Source||Sample size (n)*||Population size (N)^||Sensitivity (95% CI)||Specificity (95% CI)||Positive predictive value (95% CI)||Negative predictive value (95% CI)||Overall Quality|
|Alsolamy 201418||205||49,838||93.18 (88.78–96.00)||98.44 (98.33–98.55)||20.98 (18.50–23.70)||99.97 (99.95–99.98)||High|
|Austrian 201836||1306||Not specified||73||14.6||High|
|Bansal 201837||419||27106||100 (99.12–100)||96.21 (95.97–96.43)||29.3||100||High|
|Martin Rico 201740||178||37,323||85 (67.2–99.5)||89 (88.8–89.7)||19||99||Low|
|Meurer 200941||Alert alone: 26Alert and attending confirmation: 9||583||Alert alone: 33.3 (23.3–43.4)Alert and attending confirmation: 10.7 (4.1–17.3)||Alert alone: 78.0 (71.7–84.4)Alert and attending confirmation: 97.6 (95.2–99.9)||Low|
|Nelson 201143||Sens. and Spec.: 1375PPV and NPV: 1386||33460||64||99||54||99||High|
|Nguyen 201444||795||Not specified||44.7 (41.2–48.2)||High|
*Alerts for sepsis meeting the diagnostic criterion standard of the individual article.
^Patients presenting to the emergency department (ED).
CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value; Sens. and Spec., sensitivity and specificity.
In contrast, other studies directly notified a member of the clinical team, excluding Nguyen et al, which did not describe the notification method.44 Five rule-based studies were of high quality.18,36,37,43,44 Two studies had systems with high accuracy. Alsolamy et al18 had a sensitivity of 93.2%, specificity 98.4%, and PPV 21.0%. Bansal et al37 had a sensitivity of 100%, specificity 96.2%, and PPV 29.3%. Highest PPV was achieved by Nelson et al43 at 54.0% and Nguyen et al44 at 44.7%.
Austrian et al36 shared the number of total alerts fired, for any of three criteria sets including SIRS, nurse alert, and physician alert that included progressively more ill criteria. They report sensitivities of 73.0%, 23.8%, and 23.0%, respectively, and PPV of 13.0%, 22.4%, and 26.6% as expected for the more progressively stringent criteria. They did not share the denominator of all ED presenting patients for the retrospective period under study but report the total number of hospitalized sepsis patients based on discharge diagnosis. Septic patients may have been sent home, but if we assume they captured all true positives and false negatives through final diagnosis of sepsis, this allows for calculation of sensitivity and PPV and does not allow the calculation of specificity or NPV. With 2144 patients with a final diagnosis of severe sepsis or septic shock, and 97,216 alerts (any of the three levels included), they had the largest retrospective sample size.
Two studies assessing algorithm-based alerts were deemed low quality. Brown et al39 measured a sensitivity of 76.4%, specificity of 95.3%, and a low PPV of 5.8%. Martin Rico et al40 measured a sensitivity of 85%, specificity of 89%, and a PPV of 19%. Prevalence of sepsis compared to total ED patients was 0.3–2% in five studies.18,37,39,40,43 Meurer et al had a prevalence of 14.4%, but this was among patients ≥70 years old and it was the sole study with only SIRS criteria (a low threshold) for its sepsis definition.
Quality measures are described in Table 3. Two studies evaluating quality measures were high quality: Austrian et al36 and Nelson et al.43 In Austrian et al, process markers of time to first lactate and vasopressor use significantly improved. Statistically insignificant findings included blood cultures drawn before antibiotic administration and whether a lactate was drawn. Antibiotic timing was not reported. For Nelson et al, process markers of blood culture collection and chest radiograph before admission improved. Statistically insignificant findings included lactate collection and antibiotics given in the ED. Outcome measures of ICU transfer, ICU length of stay (LOS), and total LOS significantly improved for Austrian et al. Mortality did not improve significantly for either study.
Table 3Quality Measures.
|Source||Sample size (n)*||Population size (N)^||Significant results (process/outcome marker: prior vs. after)||Insignificant results||Overall quality|
|Austrian 201836||Before sepsis alert: 838After Sepsis alert: 1306||2144#||ICU transfer: 36.9% vs. 25.8%, p<0.001ICU length of stay (days): 1.8 (3.7) vs. 1.2 (3.1), p<0.001Length of stay (days): 10.1 (SD 10.1) vs. 8.6 (SD 7.9), p<0.001Time to first lactate (days): 0.19 (0.94) vs. 0.16 (0.58), p<0.001Vasopressor used: 28.8% vs. 22.7%, p<0.01||Blood cultures drawn prior to antibiotics: 79.0% vs 79.2%, p=0.92In-hospital mortality: 8.5% vs. 7%, p=0.22Lactate drawn (excluding ≥24hrs after ED arrival): 90.7% vs. 91.3%, p=0.65||High|
|Bansal 201837||Whole cohort: n=419Severe sepsis and septic shock: n=252||27106||In-hospital survival rate with SSRT activation in full cohort: 0.69 (95% CI, 0.31 to 1.56)In-hospital survival rate with SSRT activation among severe sepsis/septic shock patients: 0.53 (95% CI, 0.26 to 1.11)||Low|
|Berger 201038||Before sepsis alert: Lactate-151, Hyperlactatemia-33, Mortality-908.After sepsis alert: Lactate-366, Hyperlactatemia-54, Mortality-890||Before alert: 2903After alert: 2893&||Lactate testing: 5.2% vs. 12.7% (95% CI, 6.0 to 9.0%) absolute increase p<0.001||Change in frequency of hyperlactatemia if lactate was tested: 21.9% vs. 14.8% (95% CI, −0.4 to 14.6)Mortality: 5.7% vs. 5.2% (95% CI, −1.6 to 2.6%, p=0.64)||Low|
|Martin Rico 201740||1190||37,323||Mortality: 36.3% vs. 26.1%Adjusted risk reduction for mortality: 36% (0.43–0.97)Incidence Rate Ratio: 0.64, p=0.036||Low|
|Narayanan 201642||Prior to sepsis alert: n=111After sepsis alert: n=103||not specified||Antibiotics in 60 minutes: 48.6% vs. 76.7%, p<.001Length of stay odds ratio: [0.66 (0.53–0.82)]Mean time to antibiotics (minutes): 61.5 (33–171) vs. 29 (2–59), p<.001||Mortality odds ratio: 0.64 (0.26–1.57)||Low|
|Nelson 201143||184||33460||Blood culture collected odds ratio: [2.9 (1.1–7.7)]Chest radiograph before admission odds ratio: [3.2 (1.1–9.9)]||Antibiotic given in ED odds ratio: [2.8 (0.9–8.6)]Lactate collected odds ratio: [1.7 (0.9–3.2)]||High|
*Alerts for sepsis meeting the diagnostic criterion standard of the individual article.
^Patients presenting to the emergency department.
#All hospitalizations with severe sepsis or septic shock
&All patients with sepsis. Total ED presentations not specified.
ICU, intensive care unit; vs, versus; SD, standard deviation; ED, emergency department; CI, confidence interval; SSRT, sepsis and shock response team.
Four studies (Bansal, Berger, Martin Rico, Narayanan)37,38,40,42 were judged to be of low quality. Berger et al had significant improvement in lactate testing. Narayanan et al improved antibiotics within 60 minutes, time to antibiotics, and LOS. Bansal et al37 had nearly 100% sensitivity and specificity with a modest PPV of 29.3%, and had no significant improvements in survival rate. Of note, we established the article to be high quality in regard to diagnostic accuracy, while outcome measures were low quality. In contrast, Austrian et al36 had improvements in LOS and Martin Rico et al40 had improvements in mortality, while both exhibited moderate accuracy.
None of the rule-based studies showed statistically significant improvements in mortality.36-38,42,43 The only outcome reported by an algorithm-based study (Martin Rico et al)40 was mortality, which showed significant improvement, although the study was judged to be of low quality. The alert system Narayanan et al42 studied did not recommend treatment as other systems did. For this rule-based system “antibiotics in 60 minutes” meant time to antibiotics, and LOS significantly improved.
Overall, most of the study designs used to assess the impact of sepsis alerts were weak, and the review authors had difficulty isolating the impact of the automated sepsis alert itself from broader interventions such as response teams or order set bundles. Thus, our review conclusions must be couched within the strength of the overall low-quality evidence.
The limited evidence available suggests that sepsis alerts in the ED setting can be tuned to a high sensitivity for the detection of sepsis. Evidence from both low- and high-quality studies showed some improved process-of-care markers, including time to antibiotics, with the use of automated sepsis alerts.36,38,40,42,43 Lactate testing was studied by four groups with two producing significant results. Other than lactate measurement, no single measure consistently improved across studies. A lack of consistency of measured items and measurement methods creates a challenge in forming a conclusion. For example, one study examined whether blood cultures were collected, as opposed to blood cultures collected before antibiotic administration.
No high-quality studies showed a difference in mortality, and only one high-quality study showed impacts on ICU LOS and vasopressor use.36 Our findings are in keeping with a review by Makam in 2015 that covered alerts both inside and outside of the ED environment.45 Our review added recently published articles, including those that now use an algorithmic as opposed to simple rules-based approaches, and was focused on patients presenting to the ED. The strongest study designs we reviewed for inclusion were prospective cohort studies, but we would call attention to a well-executed performance improvement study conducted by Gatewood et al.46 They included a computerized alert with a multipronged intervention and showed a substantial improvement in sepsis bundle of care compliance. However, they did not show differences in mortality in part due to the inclusion of lower risk patients on the sepsis spectrum.
Sepsis alerts represent a difficult area to study with traditional randomized methods. One challenge is that in the course of operational improvement, sepsis alert criteria and/or alert thresholds may be subtly changed in the background. This may be done by information technology, analytics, or EHR personnel to address PPV or safety concerns, usually with a clinician’s input, but often without alerting all ED staff to the change. Moving to a more rigorous study design requires holding the alert constant and ethical approval for a non-alert or clinician gestalt arm. Thus, success will likely be found in future studies that use time series, or perhaps cluster randomized rollout methods across healthcare systems. Likewise, future areas for study could include comparisons of the method of alert, and the presence or absence of treatment recommendations.
None of the studies addressed potential harms. Harm may include the alarm issues impacting staff, missing alternative diagnoses due to early anchoring on sepsis, and the follow-on effects of early, aggressive fluid intervention, which has been questioned more broadly in the sepsis literature.8 Significant further work is needed on the alarm consequences of the sensitivity set points, and if possible, such work should incorporate influences from other non-sepsis alarms in alarm fatigue.
Although low quality, one algorithmic system showed significant mortality improvement, potentially validating its further development.40 Systems such as this are being developed to improve accuracy and PPV, and may include risk factors such as comorbid conditions and past medical history. These systems can effectively insert multiple variables into an equation using current and past patient data as regression coefficients, running the calculation repeatedly over the course of a patient stay as more predictor variables become available. The data creating the coefficients of such a regression-based equation would influence the predictor’s value. For example, a sepsis predictor tool based on the elderly would likely not be predictive for children. The newest models of sepsis alerts include machine learning. Complex algorithmic models may use well over 50 variables, and a machine-learning program may be integrated into them. Machine learning uses computer programming to identify patterns and significant predictors beyond the reasonable capabilities of humans. With continual analysis, it can fine-tune coefficients and thresholds of the algorithm. Initial studies show promise,47-49 and additional research is required to assess its impact on clinical outcomes.
Our limitations include a risk of publication bias because we did not search the gray literature or clinical trials for studies in progress. There are likely many hospital systems that have implemented sepsis alerts, collected data, and did not report it. Our consensus group was small in number, but we followed a rigorous process using review rubrics guided by well-accepted grading criteria.
Automated sepsis alerts in the ED may be set to a high sensitivity. Process measures show moderate benefit; however, no single measure has consistently improved, and high-quality studies have yet to demonstrate, a mortality benefit. Specific components of these systems, alarm fatigue, and sensitivity set points should be examined further. Sepsis alerts demonstrate utility and future research is indicated to build a more ideal alert system.
Section Editor: Gabriel Wardi, MD
Full text available through open access at http://escholarship.org/uc/uciem_westjem
Address for Correspondence: Matthew Hwang, BS, University of Illinois College of Medicine at Peoria, Jump Simulation, 1306 N Berkeley Ave., Peoria, IL 61603. Email: firstname.lastname@example.org. 9 / 2020; 21:1201 – 1210
Submission history: Revision received November 29, 2020; Submitted May 5, 2020; Accepted May 17, 2020
Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.
1. Singer M, Deutschman CS, Seymour CW, et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). JAMA. 2016;315(8):801-10.
2. Torio CM, Andrews RM. National Inpatient hospital costs: the most expensive conditions by payer, 2011. 2013. Available at: https://www.hcup-us.ahrq.gov/reports/statbriefs/sb160.pdf. Accessed February 2, 2020.
3. Rhee C, Dantes R, Epstein L, et al. Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009–2014. JAMA. 2017;318(13):1241-9.
4. Candel FJ, Sá MB, Belda S, et al. Current aspects in sepsis approach. Turning things around. Rev Esp Quimioter. 2018;31(4):298-315.
5. Kassyap CK, Abraham SV, Krishnan SV, et al. Factors affecting early treatment goals of sepsis patients presenting to the emergency department. Indian J Crit Care Med. 2018;22(11):797-800.
6. Andersson M, Östholm-Balkhed Å, Fredrikson M, et al. Delay of appropriate antibiotic treatment is associated with high mortality in patients with community-onset sepsis in a Swedish setting. Eur J Clin Microbiol Infect Dis. 2019;38(7):1223-34.
7. Severe sepsis and septic shock: management bundle. 2014. Available at: https://cmit.cms.gov/CMIT_public/ViewMeasure?MeasureId=1017. Accessed April 13, 2020.
8. Peake SL, et al. Goal-directed resuscitation for patients with early septic shock. N Engl J Med. 2014;371(16):1496-1506.
9. Ballester L, Martínez R, Méndez J, et al. Differences in hypotensive vs. non-hypotensive sepsis management in the emergency department: door-to-antibiotic time impact on sepsis survival. Med Sci. 2018;6(4):91.
10. Rhodes A, Evans LE, Alhazzani W, et al. Surviving Sepsis Campaign: International Guidelines for Management of Sepsis and Septic Shock: 2016. Intensive Care Med. 2017;43(3):304-77.
11. Johnston ANB, Park J, Doi SA, et al. Effect of immediate administration of antibiotics in patients with sepsis in tertiary care: a systematic review and meta-analysis. Clin Ther. 2017;39(1):190-202.e6.
12. Tafelski S, Nachtigall I, Deja M, et al. Computer-assisted decision support for changing practice in severe sepsis and septic shock. J Int Med Res. 2010;38(5):1605-16.
13. Herasevich V, Pieper MS, Pulido J, et al. Enrollment Into a time sensitive clinical study in the critical care setting: results from computerized septic shock sniffer implementation. J Am Med Inform Assoc. 2011;18(5):639-44.
14. McRee L, Thanavaro JL, Moore K, et al. The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung. 2014;43(6):546-9.
15. Sawyer AM, Deal EN, Labelle AJ, et al. Implementation of a real-time computerized sepsis alert in non-intensive care unit patients. Crit Care Med. 2011;39(3):469-73.
16. Hooper MH, Weavind L, Wheeler AP, et al. Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit. Crit Care Med. 2012;40(7):2096-101.
17. Brandt BN, Gartner AB, Moncure M, et al. Identifying severe sepsis via electronic surveillance. Am J Med Qual. 2015;30(6):559-65.
18. Alsolamy S, Al Salamah M, Al Thagafi M, et al. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak. 2014;14:105.
19. Balamuth F, Alpern ER, Abaddessa MK, et al. Improving recognition of pediatric severe sepsis in the emergency department: contributions of a vital sign based electronic alert and bedside clinician identification. Ann Emerg Med. 2017;70(6):759-68.e2.
20. Guirgis FW, Jones L, Esma R, et al. Managing sepsis: electronic recognition, rapid response teams, and standardized care save lives. J Crit Care. 2017;40:296-302.
21. Westphal GA, Pereira AB, Fachin SM, et al. An electronic warning system helps reduce the time to diagnosis of sepsis. Rev Bras Ter Intensiva. 2018;30(4):414-22.
22. Angel C, Leisman DE, Schneider SM, et al. Sepsis presenting in emergency department versus inpatient settings: divergences in prevalence, patient characteristics, initial resuscitation, outcomes, and costs. Ann Emerg Med. 2018;72(4 Supplement):S149.
23. Ramsdell TH, Smith AN, Kerkhove E. Compliance with updated sepsis bundles to meet new sepsis core measure in a tertiary care hospital. Hosp Pharm. 2017;52(3):177-86.
24. Reiter M, Wen LS, Allen BW. The emergency medicine workforce: profile and projections. J Emerg Med. 2016;50(4):690-3.
25. Guardia-LaBar LM, Scruth EA, Edworthy J, et al. Alarm fatigue: the human-system interface. Clin Nurse Spec. 2014;28(3):135-7.
26. Curry JP, Jungquist CR. A critical assessment of monitoring practices, patient deterioration, and alarm fatigue on inpatient wards: a review. Patient Saf Surg. 2014;8(1):29.
27. Sartelli M, Kluger Y, Ansaloni L, et al. Raising concerns about the sepsis-3 definitions. World J Emerg Surg. 2018;13:6.
28. Kalantari A, Mallemat H, Weingart SD. Sepsis definitions: the search for gold and what CMS got wrong. West J Emerg Med. 2017;18(5):951-6.
29. Griffey RT, Pines JM, Farley HL, et al. Chief complaint-based performance measures: a new focus for acute care quality measurement. Ann Emerg Med. 2015;65(4):387-95.
30. Advancing chief complaint-based quality measurement. 2019. Available at: http://www.qualityforum.org/Publications/2019/06/Advancing_Chief_Complaint-Based_Quality_Measurement_Final_Report.aspx. Accessed April 13, 2020.
31. McInnes MDF, Moher D, Thombs BD, et al. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: the PRISMA-DTA statement. JAMA. 2018;319(4):388-96.
32. Moher D, Shamseer L, Clarke M, et al. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015 Statement. Syst Rev. 2015;4(1):1.
33. Leeflang MMG, Deeks JJ, Gatsonis C, et al. Systematic reviews of diagnostic test accuracy. Ann Intern Med. 2008;149(12):889-97.
34. Deeks JJ, Wisniewski S, Davenport C. Guide to the contents of a Cochrane diagnostic test accuracy protocol. 2013. Available at: https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/public/uploads/Ch04_Sep2013.pdf. Accessed March 2, 2020.
35. Dijkers M. Introducing GRADE: a systematic approach to rating evidence in systematic reviews and to guideline development. KT Update. 2013;1(5):9.
36. Austrian JS, Jamin CT, Doty GR, et al. Impact of an emergency department electronic sepsis surveillance system on patient mortality and length of stay. J Am Med Inform Assoc. 2018;25(5):523-9.
37. Bansal V, Festić E, Mangi MA, et al. Early machine-human interface around sepsis severity identification: from diagnosis to improved management?. Acta Med Acad. 2018;42(1):27-38.
38. Berger T, Birnbaum A, Bijur P, et al. A computerized alert screening for severe sepsis in emergency department patients increases lactate testing but does not improve inpatient mortality. Appl Clin Inform. 2010;1(4):394-407.
39. Brown SM, Jones J, Kuttler KG, et al. Prospective evaluation of an automated method to identify patients with severe sepsis or septic shock in the emergency department. BMC Emerg Med. 2016;16(1):31.
40. Martin Rico P, Valdivia Perez A, Lacalle Martinez JM. Electronic alerting and decision support for early sepsis detection and management: impact on clinical outcomes. Eur J Clin Pharmacol. 2017;19(1):33-40.
41. Meurer WJ, Smith BL, Losman ED, et al. Real-time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009;57(1):40-5.
42. Narayanan N, Gross AK, Pintens M, et al. Effect of an electronic medical record alert for severe sepsis among ED patients. Am J Emerg Med. 2016;34(2):185-8.
43. Nelson JL, Smith BL, Jared JD, et al. Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011;57(5):500-4.
44. Nguyen SQ, Mwakalindile E, Booth JS, et al. Automated electronic medical record sepsis detection in the emergency department. PeerJ. 2014;2:e343.
45. Makam AN, Nguyen OK, Auerbach AD. Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review. J Hosp Med. 2015;10(6):396-402.
46. Gatewood MO, Wemple M, Greco S, et al. A quality improvement project to improve early sepsis care in the emergency department. BMJ Qual Saf. 2015;24(12):787-95.
47. Delahanty RJ, Alvarez J, Flynn LM, et al. Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis. Ann Emerg Med. 2019;73(4):334-44.
48. Shimabukuro DW, Barton CW, Feldman MD, et al. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir Res. 2017;4(1):e000234.
49. Nemati S, Holder A, Razmi F, et al. An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit Care Med. 2018;46(4):547-53.