Introduction
Methods
Results
Discussion
Limitations
Conclusion

INTRODUCTION

Coronavirus disease 2019 (COVID-19) is an infectious disease caused by the severe acute respiratory syndrome coronavirus 2 that has infected millions of individuals worldwide.¹ Its emergence has been regarded as a worldwide public health emergency that has prompted the transformation of healthcare systems, including those of the emergency department (ED).² Appropriate and effective treatment of acute hypoxemic respiratory failure (AHRF) due to COVID-19 early in the ED is essential to improve patients’ outcomes.

High-flow nasal cannula (HFNC), an oxygen-delivering technique whereby heated and humidified air is delivered with positive pressure generated, has been proven to be an effective initial respiratory support measure for patients with AHRF of many etiologies in both inpatient and ED settings.^3-6 It has also been used successfully as a non-invasive airway management strategy for AHRF in COVID-19 patients.^7,8 Nevertheless, HFNC therapy should always be administered with caution, as its failure may result in delayed intubation and increased mortality.^9,10 Therefore, predicting HFNC success or failure and determining the optimal timing of treatment escalation to invasive mechanical ventilation are critical to avoid delayed intubation and possibly prevent mortality.

The ROX (respiratory rate oxygenation) index, a ratio of oxygen saturation (SpO₂)/fraction of inspired oxygen (FiO₂) to respiratory rate, has been demonstrated to be a reliable predictor of HFNC success for AHRF patients in the intensive care unit (ICU) and inpatient settings.^11,12 Many studies have externally validated the ROX index as a predictor of HFNC outcomes for COVID-19 patients; however, reported results were inconsistent, possibly due to different clinical settings and cut-points employed, as well as heterogeneous population.^13-16 The modified ROX index is another index incorporating heart rate (HR) to the original ROX index, which has been shown to be a good predictor of HFNC outcomes for HFNC application post-extubation.¹⁷ However, no studies have validated its utility in COVID-19 patients.

The SpO₂/FiO₂ (SF) ratio can also be employed as a predictive marker for HFNC outcomes.¹⁸ The SF ratio may even have superior prognostic utility to the ROX or the modified ROX indices for COVID-19 pneumonia, a specific condition in which patients usually do not present with an abnormal respiratory pattern despite severe hypoxia.¹⁹ Therefore, respiratory rate, a component of both types of ROX indices, may not be a good predictor of HFNC outcomes in COVID-19 patients. Although a previous study reported a superior prognosticating ability of the SF ratio over the ROX index in COVID-19 patients, it was a single-center study conducted in an inpatient setting.²⁰

No studies have validated and compared the SF ratio with the ROX index or its modified version in the ED setting, where HFNC is usually initiated earlier in the disease course. Consequently, we conducted this study to evaluate and compare the prognostic utility of the SF ratio, the ROX index, and the modified ROX index in predicting HFNC success in patients with AHRF secondary to COVID-19 in the ED.

Population Health Research Capsule

What do we already know about this issue?

The respiratory rate oxygenation (ROX) index is a validated predictor of high-flow nasal cannula success only in non-COVID-19 patients.

What was the research question?

Our goal was to compare the prognostic utility of the SpO₂/FiO₂ ratio, ROX, and modified ROX index in COVID-19 patients.

What was the major finding of the study?

The two-hour SpO₂/FiO₂ ratio has the best discriminative ability (AUROC 0.651, 95% CI 0.558–0.744).

How does this improve population health?

For emergency COVID-19 patients, the SpO₂/FiO₂ ratio should be used for prognostication instead of the ROX or modified ROX indices.

METHODS

Study Design and Setting

This multicenter retrospective observational study was conducted between January 1–December 31, 2021 at five EDs in Thailand. A variety of EDs from various regions of the country, including those of university hospitals and secondary- and tertiary-level hospitals, participated in the study. The five study centers were Siriraj Hospital (the nation’s largest tertiary university hospital); Banphaeo Hospital (a large general hospital); Ratchaburi Hospital (a provincial teaching hospital); Buddhachinaraj hospital (a tertiary regional advanced-level hospital); and Prachuap Khiri Khan hospital (a general standard-level hospital). The Central Research Ethics Committee of Thailand approved the study (certificate number CREC044/2022). Due to its retrospective nature, informed consent was waived. The study was reported according to the STROBE guidelines.²¹

Participants

Adult patients over 18 years of age diagnosed with AHRF due to COVID-19 who received HFNC in the ED were included. We excluded COVID-19 patients who did not receive HFNC therapy initiated in the ED. Also excluded were those with a do-not-intubate order who received HFNC for palliative purposes.

Study Process and Data Collection

Using International Classification of Diseases, 10^th Revision, codes, we retrospectively reviewed consecutive patients visiting the participating EDs with the diagnosis of COVID-19 infection made before or within the index ED visit. Their electronic health records were reviewed to determine whether they had received HFNC in the ED. We used this data if all the inclusion and exclusion criteria were satisfied. With retrospective chart review performed by trained data abstractors at each study center,²² we recorded the patients’ baseline characteristics, physiologic parameters, relevant blood examination results, HFNC settings, co-treatments, and important clinical outcomes.

Parameters required for calculating the SF ratio, the ROX index, and the modified ROX index were recorded before HFNC application (hour 0) and at 2 hours after HFNC initiation. All these parameters were measured while the patients were still in the ED awaiting disposition. The decisions to initiate HFNC to the patients, adjust HFNC settings, and escalate the treatment toward a more invasive respiratory support measure were determined by the attending physicians at each study center. Another study coordinator double-checked the recorded data in the electronic case-report forms to ensure the reliability and accuracy of the study data.

Study Parameters and Outcomes

At 0 and 2 hours after HFNC initiation, we calculated three parameters and assessed them for their utility in predicting HFNC outcomes: the SF ratio; the ROX index; and the modified ROX index. The SF ratio was calculated from the ratio of SpO₂ to FiO₂.¹⁸ The ROX index was calculated from the ratio of SpO₂/FiO₂ to respiratory rate,¹¹ and the modified ROX index was defined as the ratio of the ROX index over heart rate multiplied by 100.¹⁷ The primary outcome was HFNC success, defined as no requirement of mechanical ventilation following HFNC treatment at HFNC termination. The secondary outcome was overall treatment failure, defined as a requirement of escalation to mechanical ventilation following HFNC termination or mortality at hospital discharge.^23,24

Statistical Analyses

We employed descriptive statistics to describe patients’ characteristics. Categorical data is reported as frequency and percentage. Continuous variables are reported as mean and standard deviation (SD) or median and interquartile range for normally distributed and non-normally distributed data, respectively, evaluated based on histograms and Q-Q plots. We compared these variables between the success and failure groups by using the chi-squared or Fisher exact test for categorical data and an independent t-test or the Mann-Whitney U test for continuous data.

As an external validation study, the predictive performance of the SF ratio, the ROX index, and the modified ROX index for the study outcomes were assessed primarily with their discrimination and calibration capacities, coupled with other additional analyses.²⁵ We chose the parameter with an overall superior ability over the others among all analyses performed as the best parameter in predicting the study outcomes.^25-27 We reported the discrimination of each parameter with the area under the receiver operator characteristics curve (AUROC) and its 95% confidence interval (CI). We also made comparisons between the AUROCs of the study parameters for each study outcome.²⁸ Calibration was reported with calibration plots and the Hosmer-Lemeshow test.^29,30

Moreover, we evaluated overall model performance using the Nagelkerke R-squared. A parameter that could yield a higher R-squared value should perform better than others.^31,32 We also evaluated the clinical usefulness of the parameters at the optimal cut-off values according to the Youden index by reporting their sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, negative predictive value, positive predictive value, and diagnostic odds ratio. We also performed univariate and multivariate logistic regression analyses to identify independent predictors of adverse clinical outcomes: HFNC failure and overall treatment failure.

Age, gender, body mass index, day of symptoms, Charlson Comorbidity Index, Sequential Organ Failure Assessment (SOFA) score, D-dimer, C-reactive protein, and steroid were determined a priori based as existing evidence as potential associating variables with HFNC outcomes to be adjusted for in the multivariate models. We included variables with univariate P-value <0.2 in the multivariate regression model for each outcome. Nonetheless, each multivariate model evaluated only one potential predictor value among the three parameters at one time point to avoid multicollinearity.

We performed all statistical analyses using SPSS version 18.0 (IBM Corporation, Armonk, NY), R version 3.6.1 (R Foundation for Statistical Computing, Vienna, Austria) with the rms, Hmisc, foreign, pROC, sciplot, and dca packages, and MedCalc for Windows version 19 (MedCalc Software, Ltd, Ostend, Belgium).

RESULTS

Study Population

Between January 1–December 31, 2021, a total of 978 COVID-19 patients visited the participating EDs. Of these, 184 patients were treated with HFNC initiated in the ED and 11 (6%) had do-not-intubate status. Consequently, 173 patients were included in the study. Their characteristics are shown in Table 1. Of all the included patients, 92 were male (53.2%), and their mean age was 64.8±16.2 years. A total of 118 patients (68.2%) were successfully treated with HFNC, while the other 55 (31.8%) were mechanically ventilated at HFNC termination. Meanwhile, 87 patients met the criteria for overall treatment failure, and 72.4% of them had mortality at hospital discharge. The HFNC failure group had significantly higher mean age than the success group (Table 1). Otherwise, baseline demographics and initial physiologic variables were generally comparable between the two groups. Nevertheless, the failure group required higher FiO₂ on HFNC, were on HFNC for a shorter duration, and had more complications and longer hospital length of stay than the success group (Table 1).

Study Parameters

There were no missing values for any parameters evaluated at either the 0- or 2-hour time points. We included all 173 samples in the analyses for both time points because no outcome events occurred prior to the two-hour time point at which the second parameter values were measured. The mean values of the three indices were lower in the HFNC failure group than in the success group; however, only the parameters measured at two hours were significantly different between the groups for both study outcomes (Table 2). Distributions of the parameter values among the study population are shown in Figure 1. For all parameters, a higher proportion of patients with higher parameter values were those with overall treatment success, implying strong associations between the parameter values and overall treatment failure (Figure 1D–1F). However, such trends and associations were not as prominent for any parameters with HFNC success (Figure 1A–1C).

Parameters’ Performance

The ROC curves of all parameters are shown in Figure 1S, and their AUROCs are presented in Table 3. The SF ratio measured at two hours post-HFNC application had the highest discriminating capacity (AUROC 0.651, 95% CI 0.558–0.744), followed by the two-hour ROX index (AUROC 0.612, 95% CI 0.516–0.707), and two-hour modified ROX index (AUROC 0.606, 95% CI 0.512–0.700). However, none of these AUROCs were significantly different from each other (P-value for difference among AUROCs=0.80). Similarly, the two-hour SF ratio could yield higher AUROC for overall treatment failure than the two-hour ROX and modified ROX indices (Table 3), but these AUROCs were also not significantly different (P-value=0.21). The parameters measured at hour 0 all had lower discrimination than those at two hours for both study outcomes (Table 3).

The two-hour SF ratio was the parameter with the best overall performance based on the Nagelkerke R-squared for both study outcomes (Table 3). Calibration based on the Hosmer-Lemeshow tests also showed that the two-hour SF ratio calibrated well with both outcomes, especially with overall treatment failure. Although the calibration plots in Figure 2 imply that there may have been an underestimation of the probability of both outcomes in the lowest and highest quintiles of probability predictions for the SF ratio that was worse than the other two parameters, these plots need to be interpreted with caution due to the small number of sample and event rates at the very low and very high predicted probabilities, possibly resulting in over/underestimation of risks due to random noise.

The two-hour SF ratio at the optimal cut-point of 128.19 for predicting HFNC success yielded the most balanced sensitivity (65.3%) and specificity (61.8%) compared to two-hour ROX≥3.23 and two-hour modified ROX≥4.27 (Table 4). Although it could also detect the lowest proportion of patients (56.6%), the rate of false positives was the lowest (21.4%) compared to the other indices. For overall treatment failure, two-hour SF ratio≥119.38 had the most balanced sensitivity and specificity and could detect the highest proportion of patients with the least false positives (Table 5).

Additionally, two-hour SF ratio was the chosen variable to be included in the multivariate regression models because it was the strongest predictor of the outcomes based on the univariate regression results (Table 5) and because it had the most superior diagnostic ability (based on Table 3 and 4) with the least input variables compared to the ROX and the modified ROX indices. From the multivariate model, two-hour SF ratio≥128.19 was the only variable independently associated with HFNC failure after adjusting for other potential confounders (adjusted odds ratio [aOR] 0.29, 95% CI 0.13–0.65; P=0.003) (Table 5). For overall treatment failure, the two-hour SF ratio≥119.38 was also significantly and strongly associated with the outcome (aOR 0.19, 95% CI 0.08–0.38; P<0.001) (Table 5).

DISCUSSION

To the best of our knowledge, this study was the first to directly compare the prognostic utility of the SF ratio, the ROX index, and the modified ROX index for AHRF patients secondary to COVID-19 in the ED setting. We found that the SF ratio measured at two hours post-HFNC application was the best predictor of HFNC and overall treatment success since it could yield the highest discriminating ability and overall performance index, as well as good calibration, well-balanced diagnostic accuracy indices, and strong association with HFNC and overall treatment failure.

High-flow nasal cannula has been recommended as the respiratory and oxygenation support measure for patients with AHRF due to COVID-19 since it has been shown to provide many physiologic benefits and may reduce adverse outcomes, such as mechanical ventilation rate.^7,8 Therefore, it has been increasingly used in many settings, including the ED, where the disease trajectory may differ from inpatient or ICU settings given that HFNC is initiated earlier in the disease course. Still, patients with failed HFNC treatment may end up requiring mechanical ventilation, and a delay in this process may result in mortality.⁹ To avoid these undesirable outcomes, it is necessary to employ adequate and appropriate patient monitoring using effective and efficient instruments.

The ROX index, the most widely validated prognostic marker of HFNC outcomes, has been shown by many studies to also have acceptable prognostic utility in COVID-19 patients.^{14-16, 27} However, only one single-center study has evaluated its utility in the ED setting.³⁴ Moreover, no multicenter studies have compared the ROX index to the SF ratio, a more convenient and possibly more relevant tool for COVID-19 considering its pathophysiology and general patient characteristics, or the modified ROX index, a relatively more complex measure that also incorporates another vital sign (HR) that may be related to the disease severity and progression.

Consequently, the present study has added to the current body of evidence that the SF ratio, the simplest parameter among the three, could outperform the ROX and the modified ROX indices in predicting HFNC success for COVID-19 patients in the ED setting. Although the discrimination based on AUROC of the SF ratio was not significantly higher than those of the other parameters, it was still superior to the others by a wide range of other statistical analyses, including calibration, overall model performance, diagnostic accuracy indices based on the optimal cut-point, and associations with the outcome based on regression analyses. The superiority of the SF ratio over other more complex parameters could have been because patients with COVID-19 usually present with silent hypoxia, a condition in which other physiologic parameters can appear normal despite very low oxygenation.¹⁹ As a result, the study parameter with only variables relevant to oxygenation was more highly related and predictive of the outcome. The present study yielded concordant results with a study by Kim et al, who evaluated 133 COVID-19 patients receiving HFNC treatment in an inpatient setting and found that the SF ratio at one hour provided superior AUROC to that of the ROX index.²⁰

However, it is important to note that although the SF ratio offered a more balanced sensitivity and specificity than the other two indices for HFNC success, the ROX index yielded a largely higher sensitivity and the proportion of detectable patients. This controversy might have been because the Youden index employed in these analyses may not be the most appropriate method to identify the optimal cut-point for this outcome as it only focused on the highest product of sensitivity combined with specificity without considering their balance. This matter was evidenced by a very high sensitivity and low specificity for the ROX index compared to the other two parameters.

Interestingly, we found that the SF ratio clearly had better predictive ability than the other indices for overall treatment failure based on all statistical analyses performed. Its performance was even better than the SF ratio for HFNC success. This result adds to the current body of evidence that mortality could have been another measure of adverse clinical outcomes of COVID-19 in patients treated with HFNC that predictive scoring systems, especially the SF ratio, could be able to predict accurately. With its unique clinical progression, disease-specific mortality could imply severe disease deterioration despite initial successful HFNC weaning.²⁴

Nevertheless, despite the SF ratio having the highest AUROC, the AUROCs found in our study were generally lower than in other studies in COVID-19 patients.^14-16,20,33 Such a contrast could have been explained partly by different settings and population between this study and previous studies (ED vs non-ED). Also, it could have been because of the characteristics of the population and setting specific to Thailand, a middle-income country, where the quality of care and available healthcare resources are much more limited than in other higher income countries. Our generally higher mortality and HFNC failure rates compared to other studies of higher income countries may reflect our limited-resources situation. Also, the events of in-hospital mortality and mechanical ventilation experienced by our study population might not have arisen directly due to COVID-19 but also partly due to limited healthcare provisions and suboptimal quality of care. These issues might have explained the lower discrimination capacity of the study parameters in the present study. Nevertheless, they underlie the importance of the present study as the much higher validation AUROCs in previous studies from higher income countries would not have been applicable to our scenario.

Nonetheless, from the present study, it was still appropriate to conclude that the SF ratio was superior to the ROX index and the modified ROX index in predicting HFNC success and overall treatment failure for emergency COVID-19 patients not only because of its superior performance over a wide range of statistical analytic methods but also because the SF ratio is easier to calculate at bedside, thereby being more efficient to be used in the ED.

LIMITATIONS

There were some limitations to this study. First, the study was conducted in a middle-income country, which may limit its generalizability even though it involved multiple EDs of hospitals with varying levels of care. Second, the data was collected retrospectively, which may have caused possible errors and corresponding bias associated with the nature of a retrospective study. Third, we only measured the parameters at two hours post-HFNC application and not at any later time points because it was the longest duration that all the patients were still in the EDs; therefore, the clinical utility of the parameters could also involve aiding in ED disposition decision-making. The other reason was that there were many missing variables at later time points because the physiologic parameters were not monitored simultaneously among inpatient units of participating hospitals. Nevertheless, had the parameters been followed for longer than two hours, their prognostic utility and their relative ability in prognostication could have changed. Regardless, the applicability of those findings to the ED setting may be limited.

CONCLUSION

The SF ratio measured two hours after high-flow nasal cannula initiation was better than the ROX index and the modified ROX index at predicting HFNC success in patients with acute hypoxemic respiratory failure secondary to COVID-19 in the ED setting. Compared to the other two ROX indices, the two-hour SF ratio had the greatest prognostic utility, as well as the utmost simplicity and bedside efficiency. Therefore, it may be an appropriate tool to guide appropriate disposition, further management, and potential escalation therapy for COVID-19 patients treated with HFNC therapy in the ED.

Footnotes

Full text available through open access at http://escholarship.org/uc/uciem_westjem

Address for Correspondence: Nattakarn Praphruetkit, MD, Mahidol University, Department of Emergency Medicine, 2 Wanglang Road, Bangkok Noi, Bangkok, Thailand 10700. Email: nattakarn107@yahoo.com 5 / 2023; 24:511 – 521

Submission history: Revision received August 7, 2022; Submitted January 27, 2023; Accepted February 11, 2023

Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.

REFERENCES

1. Zhu N, Zhang D, Wang W, et al. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med. 2020;382(8):727-33.

2. Sahu KK, Mishra AK, Lal A. Trajectory of the COVID-19 pandemic: chasing a moving target. Ann Transl Med. 2020;8(11):694.

3. Roca O, Riera J, Torres F, et al. High-flow oxygen therapy in acute respiratory failure. Respir Care. 2010;55(4):408-13.

4. Makdee O, Monsomboon A, Surabenjawong U, et al. High-flow nasal cannula versus conventional oxygen therapy in emergency department patients with cardiogenic pulmonary edema: a randomized controlled trial. Ann Emerg Med. 2017;70(4):465-72.

5. Ruangsomboon O, Limsuwat C, Praphruetkit N, et al. Nasal high-flow oxygen versus conventional oxygen therapy for acute severe asthma patients: a pilot randomized controlled trial. Acad Emerg Med. 2021;28(5):530-41.

6. Ruangsomboon O, Dorongthom T, Chakorn T, et al. High-flow nasal cannula versus conventional oxygen therapy in relieving dyspnea in emergency palliative patients with do-not-intubate status: a randomized crossover study. Ann Emerg Med. 2020;75(5):615-26.

7. Ospina-Tascón GA, Calderón-Tapia LE, García AF, et al. Effect of high-flow oxygen therapy vs conventional oxygen therapy on invasive mechanical ventilation and clinical recovery in patients with severe COVID-19: a randomized clinical trial. JAMA. 2021;326(21):2161-71.

8. Calligaro GL, Lalla U, Audley G, et al. The utility of high-flow nasal oxygen for severe COVID-19 pneumonia in a resource-constrained setting: A multi-centre prospective observational study. EClinicalMedicine. 2020;28:100570.

9. Kang BJ, Koh Y, Lim CM, et al. Failure of high-flow nasal cannula therapy may delay intubation and increase mortality. Intensive Care Med. 2015;41(4):623-32.

10. Roca O, de Acilu MG, Caralt B, et al. Humidified high flow nasal cannula supportive therapy improves outcomes in lung transplant recipients readmitted to the intensive care unit because of acute respiratory failure. Transplantation. 2015;99(5):1092-8.

11. Roca O, Messika J, Caralt B, et al. Predicting success of high-flow nasal cannula in pneumonia patients with hypoxemic respiratory failure: the utility of the ROX index. J Crit Care. 2016;35:200-5.

12. Roca O, Caralt B, Messika J, et al. An index combining respiratory rate and oxygenation to predict outcome of nasal high-flow therapy. Am J Respir Crit Care Med. 2018;199(11):1368-76.

13. Fink DL, Goldman NR, Cai J, et al. Ratio of oxygen saturation index to guide management of COVID-19 pneumonia. Ann Am Thorac Soc. 2021;18(8):1426-8.

14. Patel M, Chowdhury J, Mills N, et al. Utility of the rox index in predicting intubation for patients with COVID-19-related hypoxemic respiratory failure receiving high-flow nasal therapy: retrospective cohort study. JMIRx Med. 2021;2(3):e29062.

15. Prakash J, Bhattacharya PK, Yadav AK, et al. ROX index as a good predictor of high flow nasal cannula failure in COVID-19 patients with acute hypoxemic respiratory failure: A systematic review and meta-analysis. J Crit Care. 2021;66:102-8.

16. Vega ML, Dongilli R, Olaizola G, et al. COVID-19 pneumonia and ROX index: time to set a new threshold for patients admitted outside the ICU. Authors’ reply. Pulmonology. 2021;27(5):475-6.

17. Goh KJ, Chai HZ, Ong TH, et al. Early prediction of high flow nasal cannula therapy outcomes using a modified ROX index incorporating heart rate. J Intensive Care. 2020;8(1):41.

18. Kim GE, Choi SH, Park M, et al. SpO2/FiO2 as a predictor of high flow nasal cannula outcomes in children with acute hypoxemic respiratory failure. Sci Rep. 2021;11(1):13439.

19. Tobin MJ, Jubran A, Laghi F. Misconceptions of pathophysiology of happy hypoxemia and implications for management of COVID-19. Respir Res. 2020;21(1):249.

20. Kim JH, Baek AR, Lee SI, et al. ROX index and SpO2/FiO2 ratio for predicting high-flow nasal cannula failure in hypoxemic COVID-19 patients: a multicenter retrospective study. PLOS ONE. 2022;17(5):e0268431.

21. von Elm E, Altman DG, Egger M, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ. 2007;335(7624):806-8.

22. Worster A, Bledsoe RD, Cleve P, et al. Reassessing the methods of medical record review studies in emergency medicine research. Ann Emerg Med. 2005;45(4):448-51.

23. Ma XH, An MM, Yin F, et al. Factors associated with failure of high-flow nasal cannula oxygen therapy in patients with severe COVID-19: a retrospective case series. J Int Med Res. 2022;50(5):3000605221103525.

24. Ferrer S, Sancho J, Bocigas I, et al. ROX index as predictor of high flow nasal cannula therapy success in acute respiratory failure due to SARS-CoV-2. Respir Med. 2021;189:106638.

25. Ramspek CL, Jager KJ, Dekker FW, et al. External validation of prognostic models: what, why, how, when and where?. Clin Kidney J. 2021;14(1):49-58.

26. Ruangsomboon O, Boonmee P, Limsuwat C, et al. The utility of the rapid emergency medicine score (REMS) compared with SIRS, qSOFA and NEWS for Predicting in-hospital Mortality among Patients with suspicion of Sepsis in an emergency department. BMC Emerg Med. 2021;21(1):2.

27. Praphruetkit N, Boonchana N, Monsomboon A, et al. ROX index versus HACOR scale in predicting success and failure of high-flow nasal cannula in the emergency department for patients with acute hypoxemic respiratory failure: a prospective observational study. Int J Emerg Med. 2023;16(1):3.

28. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44(3):837-45.

29. Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for some traditional and novel measures. Epidemiology. 2010;21(1):128-38.

30. Pencina MJ, D’Agostino RB, Demler OV. Novel metrics for evaluating improvement in discrimination: net reclassification and integrated discrimination improvement for normal variables and nested models. Stat Med. 2012;31(2):101-13.

31. Hu B, Palta M, Shao J. Properties of R(2) statistics for logistic regression. Stat Med. 2006;25(8):1383-95.

32. Tjur T. Coefficients of determination in logistic regression models—a new proposal: the coefficient of discrimination. The American Statistician. 2009;63(4):366-72.

33. Chandel A, Patolia S, Brown AW, et al. High-flow nasal cannula therapy in COVID-19: using the ROX index to predict success. Respir Care. 2021;66(6):909-19.

34. Valencia CF, Lucero OD, Castro OC, et al. Comparison of ROX and HACOR scales to predict high-flow nasal cannula failure in patients with SARS-CoV-2 pneumonia. Sci Rep. 2021;11(1):22559.