Introduction
Methods
Results
Discussion
Limitations
Conclusion
Supplementary information

INTRODUCTION

Emergency department (ED) crowding adversely affects at least two of the Institute of Medicine’s (IOM) domains of high-quality care: safety and timeliness.^1,2 While the causes of ED crowding are multi-factorial, “output” (flow of admitted patients into inpatient settings) is cited as a leading cause.^3,4 The IOM identifies the phenomenon of holding patients in the ED after the decision to admit (known as boarding) as a public health crisis and has urged hospitals and accrediting bodies to improve inpatient resources and flow to reduce boarding of patients in the ED.⁴ Advocates recommend a number of countermeasures to improve the flow of boarded patients, and a common theme among them is the importance of recognizing that the flow of ED patients is a systemic, hospital-wide issue, rather than a problem localized to the ED.^4-6 Sporadic adoption of recommended ED-boarding countermeasures to date has led some authors to suggest that “enhanced regulation” may be required if current strategies fail to reduce boarding.^7,8

It appears the Commonwealth of Massachusetts’s Department of Public Health (DPH) is the first state regulatory body to mandate specific, hospital action plans to reduce ED boarding via its “Code Help” concept.⁹ A number of regulatory and state agencies support efforts to reduce ED boarding by permitting inpatient floor boarding or mandating reporting of ED flow data, but none except for the Massachusetts DPH appear to have mandated specific, hospital action plans with pre-defined triggers.^9,10

In December 2002, as part of a multi-pronged attempt to eliminate ambulance diversion, the DPH sent a letter to Massachusetts hospitals that included a mandate to develop individual hospital “Code Help” policies with “provisions to redeploy hospital staff and resources with a goal of moving all admitted patients out of the ED within 30 minutes [of activation].”¹¹ When Massachusetts became the first state to ban ambulance diversion,¹² the DPH stipulated that Code Help policies must include escalation to “appropriate emergency management/disaster plans and protocols” should the initial actions not adequately decompress the ED within two hours. Hospitals were required to submit their Code Help plans for review in early 2010, and the DPH’s subsequent assessment revealed that many of the plans were inadequate and lacked the specificity required by regulation.¹¹ In 2015, the DPH re-emphasized the importance of Code Help and insinuated that plans would be reviewed as part of their routine, hospital-survey processes.¹¹

Despite the DPH’s consistent emphasis on compliance with the Code Help countermeasure, boarding continues to be a critical issue across the Commonwealth,¹¹ raising questions as to the effectiveness of the Code Help initiative. As a part of process improvement efforts for our institution, one author (MAR) solicited informal feedback from Massachusetts ED directors regarding the application and effectiveness of Code Help at other institutions. Most respondents reported no qualitative improvement in ED flow after creating Code Help policies at their hospitals. Some added that Code Help was inconsistently applied and suggested that this contributed to its lack of effectiveness Martin A. Reznek, (unpublished personal communication).

Population Health Research Capsule

What do we already know about this issue?

Emergency department (ED) overcrowding adversely affects quality and patient safety, but countermeasures are limited. Massachusetts mandated hospital action plans (“Code Help”), the impact of which is unknown.

What was the research question?

Does Code Help mitigate adverse effects of overcrowding by reducing boarding time and ED length of when the policy is followed?

What was the major finding of the study?

Code Help implementation is associated with shorter ED boarding time and length of stay when the policy is consistently followed.

How does this improve population health?

If the effects of this single-center experience are replicated more broadly, mandates on hospitals may have potential to decrease patients’ exposure to the negative effects of overcrowding.

Anecdotal experience at our institution was similar, and we found adherence to an effective Code Help procedure to be historically difficult and inconsistent. However, as the policy gained broader acceptance from hospital leadership, we saw an opportunity to evaluate whether Code Help is effective when completely implemented and the DPH guidelines followed. We hypothesized that compliance with the Code Help policy would be associated with reduction in ED boarding time and admitted-patient total ED length of stay (LOS), compared to patients seen when the Code Help policy was not followed.

METHODS

Study Design and Setting

This was a cohort study conducted at a 364-bed urban, academic, tertiary referral center with trauma, stroke, and cardiac programs serving approximately 27,000 adult inpatients annually, with 65,000 annual adult ED visits and a monthly ED adult admission rate of 26–30%. There is a co-located pediatric center, but it is operationally distinct and includes a separate pediatric ED.

The hospital developed and implemented its Code Help policy in accordance with the DPH mandate and guidelines. The policy includes standardized triggers, activation processes, next steps in the event of failure, and testing and evaluation as outlined by the DPH.¹¹ In general, three levels of activation were observed: “normal operations;” “Code Help;” and escalation to the hospital emergency operations plan (also known as “disaster plan activation”). The hospital disaster plan required in-person or conference-call response of all hospital managers, use of the Hospital Incident Command System, including a defined incident commander, conducting regularly scheduled briefings, and a continuously operational Emergency Operations Center where resources and decision-makers for the hospital system are located. Table 1 includes relevant text from the Code Help policy.

Measurements and Selection of Participants

We created a prospective Code Help event registry on October 1, 2014, enabling ascertainment of Code Help event timestamps following that date. We retrospectively queried the electronic health record (EHR) for consecutive individual patient visits of all adult ED patients from October 2014 through January 2016. For all admitted patients, we extracted EHR timestamps tracking four patient flow events: ED arrival, ED triage completion, admission decision, and ED departure time (physically moved to an inpatient unit). The electronic inpatient bed request placed by the ED provider following a verbal acceptance of the patient by the admitting team served as a proxy for admission decision.¹³ We included patients admitted to either a medical/surgical acute care hospital bed (including telemetry) or critical care bed. Patients admitted to psychiatry, labor and delivery, or directly to a procedural area (operating room or cardiac catheterization lab) were excluded, as the EHR admission timestamp data were known to be unreliable for these patients due to unique admission processes related to those units. We defined boarding patients as those who remained in the ED after the decision to admit and defined boarding time as the interval between the admission-decision time and the departure time from the ED.¹⁰ We defined total ED LOS as the interval between ED arrival time and physical departure time from the ED.¹⁰

Code Help Exposure Status

We matched each patient visit against our prospectively collected registry of Code Help events, which contained start and stop times for each Code Help event as well as hospital disaster-plan activation time, if applicable. For each patient, we determined the hospital’s concurrent Code Help “status” (normal operations, Code Help, or disaster plan) at the time of each of four patient flow events: ED arrival time, ED triage time, admission decision time, and ED departure time.

We then determined whether the concurrent hospital status complied with the Code Help policy or violated the policy at the time of each patient flow event. Any of three possible scenarios constituted a policy violation: (1) the ED operational environment met Code Help activation criteria, but Code Help was not activated; (2) Code Help criteria had been met for greater than two hours without escalation to the hospital disaster plan; or (3) Code Help was re-activated within 24 hours without escalating directly to the hospital disaster plan. We determined the latter two violations by using the Code Help registry timestamps to calculate the elapsed time in the Code Help status and the elapsed time since the last Code Help/disaster event, respectively. Determination of the first violation type (the ED operational environment met criteria for Code help activation but remained in normal operations) required a standardized measure of the state of operations and flow in the ED.

Recognizing inherent limitations of all current quantitative measures of ED crowding, we selected ED occupancy ratio (EDOR), the number of patients currently in the ED divided by the number of licensed ED treatment spaces, as a surrogate for ED resource demand due to its prior use in the literature and relative ease of calculation as an instantaneous measure.^14-22 An EDOR greater than 100%, by definition, would fulfill the Code Help activation criterion of “capacity of the ED exceeds licensed bed capacity” (Table 1).¹¹ However, our ED routinely operated with staffed, unlicensed “hallway” beds, and the number of these beds varied in response to patient demands and resource availability. As such, an EDOR of 100% would accurately reflect the ED licensed bed capacity but would underestimate our functional ED capacity.

We had no way to determine the exact number of unlicensed, staffed treatment spaces at any given time, so we sought to identify a surrogate EDOR threshold to more accurately reflect our functional ED capacity limit. Our initial analysis suggested that EDOR of 200% corresponded to the 99th percentile for all hours during the study period. Further, we verified that for each hour where EDOR exceeded 200%, there was at least one boarded patient in the ED (minimum 5, median 21, interquartile range [IQR] 9), which fulfilled the second trigger criterion for Code Help (Table 1). We categorized any time during which EDOR exceeded 200%, but neither Code Help nor the disaster plan were active, as being a probable violation of the policy. We validated this approach against an alternative logistic regression model (see Appendix).

Statistical analysis

To assess the effects of compliance and non-compliance with the Code Help policy, we performed univariate comparisons of boarding time and overall ED LOS for patients cared for during periods of Code Help policy compliance and during periods of Code Help policy violation, both with reference to patients cared for during normal operations, using Steel’s method, the nonparametric version of Dunnett’s test, which controls the error rate for multiple comparisons vs. the control group.²³ We chose to compare each scenario to a common reference standard (normal operations) because doing so improved the overall error rate compared to pairwise comparisons and allowed us to evaluate the efficacy of Code Help in maintaining patient flow as close to normal operations as possible, despite the crowding and adverse circumstances that triggered Code Help activation. Performing only a direct comparison between policy compliance and policy violation would have ignored the valuable data from the large number of patients seen during normal operations, who could serve as a common control group, and would have dramatically reduced our statistical power to identify a between-group difference because of the reduction in population size.

Using the same technique, we performed a secondary analysis of the same metrics during any Code Help event or disaster activation (regardless of policy compliance), with reference to patients cared for during normal operations. We felt this secondary analysis was important to evaluate the effects of Code Help/disaster itself, even if misapplied or inconsistently followed. We also performed a number of sensitivity analyses to validate our analytic choices (see Appendix). Analyses were conducted using JMP Pro 12 (SAS Institute Inc., Cary, NC). The study was approved by our institutional review board.

RESULTS

Characteristics of Admissions and Code Help Events

Of 89,587 adult patients who presented to the ED during the study period 26,065 (29.1%) were admitted, 24,017 (92.1%) of whom were admitted to either an acute care or critical care bed and included in further analysis. Of the admitted patients, the median age was 64 (IQR 26), and 48% were female. Boarding time ranged from zero to 67 hours 30 minutes (median 4 hours 31 minutes) and was less than two hours for 14.2% of admitted patients. Total ED LOS for admitted patients ranged from 11 minutes to 85 hours 25 minutes (median 9 hours). ED occupancy ratio at the time of decision to admit ranged from 34% to 243% (mean 128%, standard deviation 33) and was stable over the time period of the study.

There were 89 Code Help events recorded in the registry during the study period (every 5.4 days on average), and 23 (26%) progressed to disaster plan activation. The probability of progressing to disaster plan increased over time, while the monthly frequency of Code Help events decreased (Figure). Time from Code Help activation until disaster activation ranged from 57 minutes to 3 hours 25 minutes (median 2 hours 39 minutes), and there were 64 instances of not escalating to the hospital disaster plan, despite meeting the two-hour criteria.

Policy Violations

Among all admitted patients, 2,219 (9.2%) had a decision to admit during Code Help, and 492 (2.0%) had a decision to admit during disaster. Among these 2,711 patients, 1,383 (51%) were admitted during a policy violation (1,227 while Code Help had been active greater than two hours, but the disaster plan had not yet been activated, and 156 during a re-activation of Code Help within 24 hours without activating the disaster plan). We identified an additional 94 patients admitted during a presumptive policy violation, where EDOR exceeded 200% but neither Code Help nor the hospital disaster plan were active.

Code Help Effectiveness

Each Code Help event was associated with a mean 17% reduction (95% CI [12%–22%]) in the number of patients boarding at the end of Code Help, compared to the time of activation. However, much of this reduction was accomplished after the first 30 minutes of Code Help, despite the stated policy goal of removing all boarding patients from the ED within 30 minutes. In the first 30 minutes, there was a mean 0% reduction in boarding patients (95% CI [3.4% increase to 0.2% decrease]).

Main Results

When not accounting for policy compliance, median boarding time and total ED LOS were longest during disaster activation and shortest during normal operations (Table 2). However, when accounting for Code Help policy compliance vs. violations of the policy, patients admitted during periods of any type of Code Help policy violation had significantly longer boarding times and total ED LOS, compared to patients admitted under normal operations (Hodges-Lehmann estimate of 25 minutes [95% CI {13–37 minutes} of additional boarding time and 45 minutes [95% CI {26 minutes to 1 hour 5 minutes}] of additional ED LOS). Among patients admitted during periods of Code Help policy compliance, in contrast, we found no significant difference in either metric, compared to normal operations. Table 3 reports the distributions of each metric for each subgroup.

Sensitivity Analyses

Our results were insensitive to the choice of patient flow-event timestamp linkages. Of the four events, we selected decision to admit for the primary analysis because we presumed that Code Help countermeasures were likely to have the greatest potential impact on a patient’s flow into the inpatient setting if active at the time of the decision to admit. Lagged effects of Code Help were maintained at 30 and 60 minutes after the end of a Code Help or disaster event, but effects did not persist at 90 minutes or six hours. Our results were substantially unchanged when considering only the second two policy-violation types, discarding the EDOR threshold.

DISCUSSION

Our results suggest that when the Code Help concept is implemented in a manner that complies with DPH requirements and the policy is followed, both ED boarding time and total ED LOS for admitted patients appear to be reduced to durations typical of normal operations, despite increased ED demand. Violations of the Code Help policy appear associated with the loss of those benefits. We observed a 14-minute relative increase in median boarding time and a 28-minute increase in ED LOS among patients admitted during periods of policy violation, compared to those admitted during periods of policy-compliant Code Help or disaster, despite equally adverse ED operational conditions. This difference cumulatively represents approximately 689 patient-hours (28.7 patient-days) of ED capacity during the study period, which would otherwise have been available to care for additional ED patients had the policy been followed.

Our results suggest that compliance with the Code Help policy is pivotal in achieving improved ED flow, rather than simply having the policy in place but not following its guidelines. The DPH’s Code Help concept, when implemented correctly and consistently, may reduce ED boarding and crowding and represents an important countermeasure to supplement the relatively limited armamentarium of current strategies.^5,7 It is worth noting, however, that even when the policy is followed Code Help does not appear to achieve its stated objective of removing all boarding patients from the ED within 30 minutes of activation.

While the overall results of this study are encouraging, it is not clear what specific factors within the Code Help policy implemented at our institution led to flow improvements. We believe that its effectiveness lies in the fact that the policy sets clear expectations, has a defined escalation process, requires hospital-wide leadership involvement, and establishes real-time accountability. It mandates action by leaders outside of the ED, who can problem-solve on a system level and engage in real-time, team-based solutions, and it provides a standardized structure for how to do so. At its core, Code Help provides for a hospital-wide response to a hospital-wide patient flow problem, even if the primary manifestation of that problem appears only in the ED.

One shortcoming of the Code help concept is that it focuses on reactive, rather than proactive, responses to crowding. While lessons learned during each Code Help activation may result in incremental process improvements during normal operations, Code Help actions do little to directly smooth flow or increase throughput when the plan is not active. Another disadvantage of the Code Help concept is the potential frequency with which the hospital disaster plan must be activated in the event Code Help is ineffective after two hours. When Code Help is routinely activated, the demand for frequent briefings and conference calls may compete with hospital leaders’ other duties. This may raise awareness regarding crowding in the short term, but other long-term priorities may be inadvertently adversely affected.

To our knowledge, the DPH Code Help regulation is the first of its kind in the U.S. that mandates specific hospital actions to alleviate ED boarding.⁹ The Centers for Medicare and Medicaid Services now requires reporting of ED flow measures, and the Joint Commission requires hospitals to have committees that oversee hospital flow, but neither mandates specific, ED-boarding countermeasures.²⁴ The DPH Code Help initiative presents a unique opportunity to evaluate whether “enhanced regulation” may reduce ED boarding, as suggested in prior literature.⁷ While it remains to be seen if the DPH Code Help regulation will be successful across the Commonwealth over time, the results of this study suggest that it may be effective if hospital policy meets the DPH requirements and is followed consistently.

LIMITATIONS

Given the potential confounders and time-dependent nature of this dataset, we considered a number of analytic approaches and found that each approach, including our final analysis plan, had substantial limitations. ED LOS and boarding time are time-to-event data. Although they do not exhibit censoring (i.e., we have available LOS data for each patient, no matter how long they waited for admission), our observed boarding times do have some similarities to survival data, in that the probability of a given patient remaining in the ED at any point in time is conditional on the patient’s presence in the ED during all preceding times since their arrival. Thus, we considered using Cox proportional hazards regression, but our dataset did not seem to fit the assumption that the “hazards” (that is, the probability of a given patient ending their ED LOS in the next minute) are strictly proportional between groups. We also considered a time-series approach, which still did not completely address our limitations. Our simpler approach of comparing group medians and distributions was less powerful, but we felt more assured that our data satisfied the prerequisites of the more conservative Steel test.

Comparisons of group medians, however, are inherently disadvantageous, in part because our method of stratifying patients to policy-compliant and policy-violation groups insinuates that there is a clear delineation between these cohorts. In fact, there is very little discernable difference between a patient admitted 119 minutes into a Code Help event (technically policy-complaint) and a patient admitted at 121 minutes without disaster plan escalation (a policy violation). Similarly, while the operational environment when the EDOR is 195% is quite similar to that when the EDOR is 205% when Code Help is not active, our approach would trigger a policy violation only for the latter. It is implausible that Code Help would have a differential effect at EDOR 205%, compared to 195%, or 119 minutes, compared to 121 minutes, but our analytical approach assumes that it may.

A related limitation is that detecting failure to activate Code Help when criteria were initially met required a surrogate marker because of the retrospective nature of the investigation. Because ED census exceeding licensed ED bed capacity was a criterion for Code Help activation, EDOR was a natural choice as a marker. Our data demonstrated that there were always admitted patients boarding in the ED when EDOR exceed 102%. However, it was also the case that our usual operations included evaluating and treating patients in staffed but unlicensed hallway spaces, so 100% occupancy is likely an overestimate of functional crowding in our ED. Our threshold of 200% (99^th percentile of EDOR) was intended to be conservative and more specific than sensitive. By design, the risk of falsely categorizing a patient as having been admitted during a policy violation was low, but we likely failed to identify some true violations that may have occurred at times when EDOR was between 102% and 200%. This type of violation accounted for only 11% of the patients admitted during any policy violation and probably underestimates actual violations. In our post-hoc sensitivity analysis to consider a comparison of only absolute policy violations (by considering the 94 patients admitted with EDOR >200% to be in the normal operations group), our findings were substantially unchanged.

We also had no mechanism to measure overall hospital demand-capacity mismatch outside of the ED nor insight into the specific decision-making that resulted in Code Help policy violations. Consequently, it is possible that violations were associated with hidden, external factors, such as leaders sensing complete hospital resource saturation and not following the policy due to feelings of futility. Further, a prospective power analysis was not possible given the study design. It is conceivable that any of these factors may have resulted in our failure to detect a difference in outcomes between the policy-compliant and normal operations groups, where one existed in reality (a Type II error). Nevertheless, the magnitude of difference between the policy-compliant and policy-violation groups and the fact that metrics were worse under Code Help when ignoring policy compliance (Table 2) suggest that policy compliance likely has a real differential effect.

Finally, our study reports the experience of a single center, which naturally limits the generalizability of our findings. It is likely that the specific interventions that occur during Code Help/disaster at our institution may not be as effective at other sites because they were designed to fit our local work environment and processes. However, a key strength of the DPH Code Help concept may be that, while it does call for adherence to specific guiding principles, it does not mandate specific tactics. We believe the general principles set forth in the regulations are generalizable to all hospitals, even if they require different implementation tactics. In fact, the failure to customize these specifics to each institution’s unique workflow may be partially responsible for the initially slow adoption of the Code Help concept more generally. Based on the DPH’s own assessment, Code Help has not been effective Commonwealth-wide,¹¹ but its analysis suggests this may be due to the fact that many individual hospital policies do not meet DPH requirements. Currently, we are unaware of any penalties levied by the DPH against hospitals that do not comply with the Code Help requirements. Our study may lend credence to the idea that regulators should value actual policy compliance, as opposed to hospitals simply having created a Code Help policy.

CONCLUSION

In our single-center experience, implementation of the DPH Code Help regulation is associated with shorter ED boarding time and ED length of stay when the policy is consistently followed. However, our analytic approach has important limitations that necessitate cautious interpretation of our findings. It remains to be determined whether the regulation will result in improved outcomes more broadly across Massachusetts.

Supplementary Information

Footnotes

Section Editor: Kenneth S. Whitlow, DO

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Sean Michael, MD, MBA, University of Colorado School of Medicine, Department of Emergency Medicine, 12401 East 17th Avenue, Mail Stop B-215, Aurora, CO 80045. Email: sean.michael@ucdenver.edu. 5 / 2018; 19:501 – 509

Submission history: Revision received October 3, 2017; Accepted January 1, 2018

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Crossing the Quality Chasm: A New Health System for the 21st Century. 2001.

2. Bernstein SL, Aronsky D, Duseja R, et al. The effect of emergency department crowding on clinically oriented outcomes. Acad Emerg Med. 2009;16(1):1-10.

3. Crowded Conditions Vary among Hospitals and Communities. 2003.

4. Committee on the Future of Emergency Care in the United States Health System. Hospital-based emergency care: at the breaking point. 2007.

5. Moskop JC, Sklar DP, Geiderman JM, et al. Emergency department crowding, part 2–barriers to reform and strategies to overcome them. Ann Emerg Med. 2009;53(5):612-7.

6. Moskop JC, Sklar DP, Geiderman JM, et al. Emergency department crowding, part 1–concept, causes, and moral consequences. Ann Emerg Med. 2009;53(5):605-11.

7. Rabin E, Kocher K, McClelland M, et al. Solutions to emergency department ‘boarding’ and crowding are underused and may need to be legislated. Health Aff (Millwood). 2012;31(8):1757-66.

8. Warner LS, Pines JM, Chambers JG, et al. The most crowded US hospital emergency departments did not adopt effective interventions to improve flow, 2007–10. Health Aff (Millwood). 2015;34(12):2151-9.

9. State approaches to boarding and crowding. 2016. Available at: https://www.acep.org/Advocacy/State-Approaches-to-Boarding-and-Crowding/. Accessed April 22, 2016.

10. Welch SJ, Asplin BR, Stone-Griffith S, et al. Emergency department operational metrics, measures and definitions: results of the Second Performance Measures and Benchmarking Summit. Ann Emerg Med. 2011;58(1):33-40.

11. Code Help Plans for Hospitals. Available at: http://www.mass.gov/eohhs/gov/departments/dph/programs/hcq/healthcare-quality/health-care-facilities/hospitals/code-help-plans.html. Accessed March 6, 2016.

12. Burke L. Ending ambulance diversion in massachusetts. Virtual Mentor. 2010;12(6):483-6.

13. Forum NQF. National Voluntary Consensus Standards for Emergency Care: A Consensus Report. 2009.

14. McCarthy ML, Aronsky D, Jones ID, et al. The emergency department occupancy rate: a simple measure of emergency department crowding?. Ann Emerg Med. 2008;51(1):15-24.

15. Pitts SR, Pines JM, Handrigan MT, et al. National trends in emergency department occupancy, 2001 to 2008: effect of inpatient admissions versus emergency department practice intensity. Ann Emerg Med. 2012;60(6):679-686.e673.

16. Epstein SK, Huckins DS, Liu SW, et al. Emergency department crowding and risk of preventable medical errors. Intern Emerg Med. 2012;7(2):173-80.

17. Tekwani KL, Kerem Y, Mistry CD, et al. Emergency department crowding is associated with reduced satisfaction scores in patients discharged from the emergency department. West J Emerg Med. 2013;14(1):11-5.

18. Halverson S, Malani PN, Newton DW, et al. Impact of hourly emergency department patient volume on blood culture contamination and diagnostic yield. J Clin Microbiol. 2013;51(6):1721-6.

19. Shin TG, Jo IJ, Choi DJ, et al. The adverse effect of emergency department crowding on compliance with the resuscitation bundle in the management of severe sepsis and septic shock. Crit Care. 2013;17(5):R224.

20. Derose SF, Gabayan GZ, Chiu VY, et al. Emergency department crowding predicts admission length-of-stay but not mortality in a large health system. Med Care. 2014;52(7):602-11.

21. Jo S, Jin YH, Lee JB, et al. Emergency department occupancy ratio is associated with increased early mortality. J Emerg Med. 2014;46(2):241-9.

22. Verelst S, Wouters P, Gillet JB, et al. Emergency department crowding in relation to in-hospital adverse medical events: a large prospective observational cohort study. J Emerg Med. 2015;49(6):949-61.

23. Hsu J. Multiple Comparisons: Theory and Methods. 1996.

24. Wiler JL, Welch S, Pines J, et al. Emergency department performance measures updates: proceedings of the 2014 emergency department benchmarking alliance consensus summit. Acad Emerg Med. 2015;22(5):542-53.