
==== Front
Indian J Crit Care Med
Indian J Crit Care Med
IJCCM
Indian Journal of Critical Care Medicine : Peer-reviewed, Official Publication of Indian Society of Critical Care Medicine
0972-5229
1998-359X
Jaypee Brothers Medical Publishers

38585312
10.5005/jp-journals-10071-24673
Original Article
Comparative Predictive Accuracies of the Simplified Mortality Score for the Intensive Care Unit, Sepsis Severity Score, and Standard Severity Scores for 90-day Mortality in Sepsis Patients
Sathaporn Natthaka 1https://orcid.org/0000-0001-8524-3156

Khwannimit Bodin 2https://orcid.org/0000-0002-5436-8519

1,2 Division of Critical Care Medicine, Department of Internal Medicine, Faculty of Medicine, Prince of Songkla University, Hat Yai, Songkhla, Thailand
Bodin Khwannimit, Division of Critical Care Medicine, Department of Internal Medicine, Faculty of Medicine, Prince of Songkla University, Hat Yai, Songkhla, Thailand, Phone: +66 74-451452, e-mail: kbordinbodin@gmail.com
4 2024
30 3 2024
28 4 343348
11 12 2023
10 1 2024
Copyright © 2024; The Author(s).
2024
https://creativecommons.org/licenses/by-nc/4.0/ © The Author(s). 2024 Open Access. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by-nc/4.0/), which permits unrestricted use, distribution, and non-commercial reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Abstract

Background

The standard severity scores were used for predicting hospital mortality of intensive care unit (ICU) patients. Recently, the new predictive score, Simplified Mortality Score for the ICU (SMS–ICU), was developed for predicting 90-day mortality.

Objective

To validate the ability of the SMS–ICU and compare with sepsis severity score (SSS) and original severity scores for predicting 90-day mortality in sepsis patients.

Method

An analysis of retrospective data was conducted in the ICU of a university teaching hospital. Also, 90-day mortality was used for the primary outcome.

Results

A total of 1,161 patients with sepsis were included. The 90-day mortality was 42.4%. The SMS–ICU presented the area under the receiver operating characteristic curve (AUROC) of 0.71, whereas the SSS had significantly higher AUROC than that of the SMS–ICU (AUROC 0.876, p < 0.001). The acute physiology and chronic health evaluation (APACHE) II and IV, and the simplified acute physiology scores (SAPS) II demonstrated good discrimination, with an AUROC above 0.90. The SMS–ICU provides poor calibration for 90-day mortality prediction, similar to the SSS and other standard severity scores. Furthermore, 90-day mortality was underestimated by the SMS–ICU, which had a standardized mortality ratio (SMR) of 1.36. The overall performance by Brier score demonstrated that the SMS–ICU was inferior to the SSS (0.222 and 0.169, respectively). Also, SAPS II presented the best overall performance with a Brier score of 0.092.

Conclusion

The SMS–ICU indicated lower performance compared to the SSS, standard severity scores. Consequently, modifications are required to enhance the performance of the SMS–ICU.

How to cite this article

Sathaporn N, Khwannimit B. Comparative Predictive Accuracies of the Simplified Mortality Score for the Intensive Care Unit, Sepsis Severity Score, and Standard Severity Scores for 90-day Mortality in Sepsis Patients. Indian J Crit Care Med 2024;28(4):343–348.

Keywords

Intensive care unit
Mortality
Risk prediction
Severity score
==== Body
pmcHighlights

The conventional severity scores were designed to predict in-hospital mortality. However, assessing outcomes through in-hospital mortality might not be suitable as it can be influenced by discharge policies, and critical illness could continue to affect patients’ postdischarge. Thus, it is recommended to use extended fixed-time outcomes to determine the outcomes of critically ill patients.

Introduction

Mortality prediction in patients with critical conditions is challenging, especially in sepsis patients which is the most prevalent issue in the intensive care unit (ICU).1 One in every five deaths worldwide is associated with sepsis, which has a mortality rate of 20% to more than 50%.2,3 Severity scores are tools that provide objective and standardized assistance in the assessment of temporal trends in mortality and the impact of treatment protocols over time. These scores serve as valuable tools for benchmarking, which ultimately helps improve patient care in the ICU setting through the identification of areas of improvement by medical personnel and the application of evidence-based practices to determine the appropriate level of care.4–6

The acute physiology and chronic health evaluation (APACHE) and simplified acute physiology score (SAPS) are the two models that are frequently used to determine the severity of illness in ICU patients.6,7 Nevertheless, both APACHE and SAPS are intricate models, necessitating a multitude of laboratory parameters and consuming significant time.8

The sepsis severity score (SSS) was formulated specifically to explicitly predict mortality in sepsis patients. The surviving sepsis campaign database served as the source of the SSS and demonstrated commendable performance in estimating the in-hospital mortality of sepsis patients. A prior investigation conducted in our ICU revealed that the SSS exhibited good discrimination comparable to that of conventional severity scores.9,10 However, external validation of these severity scores in our previous study demonstrated poor calibration.10

Sepsis and septic shock impact both immediate survival and long-term consequences. After hospitalization for sepsis, these patients may experience chronic health issues and an increased mortality risk.11 Thus, long-term sequelae of sepsis should receive more attention. Despite discharge policies and the potential impact of critical illness on patients after hospital discharge, it may not be appropriate to quantify outcomes based on in-hospital mortality. Therefore, the longer fixed-time outcome is recommended to measure the outcomes in critically ill patients.12

The SMS–ICU is a new, simple clinical prediction model that has been developed, by using data from 4,086 patients in three randomized controlled trials in sepsis patients and two large cohort studies in critically ill patients, to estimate 90-day mortality among adult patients in the ICU.13 The SMS–ICU uses only seven readily available variables within the 24 hours of ICU admission, comprising two numeric parameters (age and lowest systolic blood pressure) and five binary factors (presence of hematologic malignancy/metastatic cancer, surgical admission, administration of vasoactive or inotropic agents, requirement for mechanical ventilator support, and need for renal replacement therapy). External evaluation of the SMS–ICU vs SAPS 3 demonstrated that SMS–ICU discrimination was good, although somewhat lower than that of SAPS 3.14 Thus, the SMS–ICU was developed for long-term mortality prediction in critically ill patients and represents a simple tool with minimal expected missing data.

This study was conducted with two objectives: (A) to assess the SMS–ICU performance and (B) to compare the ability of the SMS–ICU with disease-specific scores such as the SSS, as well as other widely used severity scores in predicting 90-day mortality among sepsis patients.

Methods

A retrospective study was undertaken using a prospective registry database of patients diagnosed with sepsis, alongside the evaluation of severity scoring systems, within the medical ICU of the university hospital. This study was approved by our Human Research Ethics Committee (REC 63-293-14-1). The retrospective design led to the waiver of informed consent.

The study included individuals aged 15 years and above who were diagnosed with sepsis and hospitalized in the ICU from 1 January 2017 to 30 June 2021. Sepsis or septic shock was diagnosed using the Sepsis-3 criteria.15 Readmission episodes to the ICU during a single hospitalization were excluded.

All parameters required for the SMS–ICU (Supplementary Table S1), SSS (Supplementary Table S2), APACHE II, APACHE IV, SAPS II, and sequential organ failure assessment (SOFA) calculations as reported in the original studies were collected. The scores were computed using the most unfavorable physiological data obtained within the initial 24 hours following ICU. The probability of mortality was predicted using previously validated algorithms based on the SMS–ICU, SSS, APACHE II, and SAPS II.9,13,16,17 The predictions for hospital mortality using APACHE IV were acquired via https://intensivecarenetwork.com/Calculators/Files/Apache4.html. The primary outcome was the 90-day mortality. Hospital mortality was used to assess the performance of these scores for secondary outcome.

The scores were assessed based on discrimination, calibration, and overall performance. Discrimination is the scoring system's capacity to differentiate between high and low-risk patients by identifying patients who have died and those who have survived, as measured by the area under the receiver operating characteristic curve (AUROC) with a 95% confidence interval (CI).18,19 Furthermore, AUROCs of above 0.90, 0.80–0.89, and 0.70–0.79 are considered excellent, good, and moderate, respectively.20 Comparison of the AUROCs were conducted using the approach recommended by Delong et al.21 Calibration was performed to assess the model's prognostic accuracy across various risk levels. The standardized mortality ratio (SMR) and the Hosmer–Lemeshow goodness of fit H and C statistics were used to assess the calibrations.18 The disparity between the patient groups’ observed and anticipated mortality rates was evaluated using goodness-of-fit statistics. Good calibration was defined as a goodness-of-fit test p-value above 0.05. The SMR referred to the ratio of the actual number of deaths and the number of deaths that the severity score predicted. An overestimation of mortality is indicated by an SMR less than 1, whereas an underestimation is shown by an SMR greater than 1. Plotting the expected mortality rates stratified by 10% increments in the anticipated mortality vs actual mortality rates allowed calibration curves to be created. Concerning discrimination and calibration, the Brier score offers a comprehensive assessment of the overall performance.4,18 Higher accuracy was reflected by a lower score. The statistical analysis was performed using Stata 15.

Results

A total of 1,161 sepsis patients were enrolled. Septic shock was diagnosed in 641 (55.2%) patients. The 90-day mortality rates were 42.4%, whereas in-hospital mortality rates were 41.5%. Pre-existing diseases affected 452 patients (38.9%), with the three most prevalent being hematologic malignancies (10.8%), immunocompromised conditions (7.8%), and solid tumors with metastases (6.6%). Respiratory tract infections (58.6%) were the most common source, followed by primary bloodstream infections (10.5%), gastrointestinal tract infections (10.4%), and urinary tract infections (10.2%). Microorganisms were identified in 941 (81%) patients. Positive blood cultures were detected in 27.6% of the patients. Most patients (89.2%) required mechanical ventilation. Patient characteristics and severity scores stratified by 90-day mortality are displayed in Table 1.

Table 1 Clinical demographic data*

	All patients (n = 1,161)	Survivors at 90 days (n = 669)	Death within 90 days (n = 492)	p-value	
Age	65 (51–77)	65 (50–77)	65 (54–77)	0.25	
Male [n (%)]	674 (58)	373 (55.75)	301 (61.18)	0.06	
Comorbidities [n (%)]	452 (38.9)	189 (28.25)	263 (53.46)	<0.001	
  Hematologic malignancy	125 (10.8)	37 (5.53)	88 (17.89)	<0.001	
  Immunocompromised	91 (7.8)	51 (7.62)	40 (8.13)	0.75	
  Metastatic cancer	76 (6.6)	24 (3.59)	52 (10.57)	<0.001	
Site of infection					
  Respiratory tract	681 (58.8)	382 (57.1)	299 (60.77)	0.21	
  Primary bloodstream	122 (10.5)	54 (8.07)	68 (13.82)	0.002	
  Gastrointestinal tract	121 (10.4)	74 (11.06)	47 (9.55)	0.41	
  Urinary tract	118 (10.2)	77 (11.51)	41 (8.33)	0.08	
Severity scores					
  SMS–ICU	18 (13–22)	17 (13–20)	22 (17–25)	<0.001	
  SSS	71 (49–90)	54 (39–71)	91 (77–107)	<0.001	
  APACHE II	22 (15–31)	17 (13–21)	33 (27–37)	<0.001	
  APACHE IV	78 (56–120)	60 (47–72)	124 (105.5–142)	<0.001	
  SAPS II	51 (39–69)	41 (33–49)	72 (64–82)	<0.001	
SOFA	10 (6–13)	7 (4–10)	13 (11–15)	<0.001	
Lactate (mmol/L)	4.7 ± 5.3	3.3 ± 4.3	6.7 ± 5.8	<0.001	
ICU LOS (days)	4 (2–8)	4 (2–8)	4 (2–9)	0.78	
Hospital LOS (days)	18 (9–33)	19 (10–36)	16 (6–31)	<0.001	
*Unless otherwise indicated, the data are presented as medians with interquartile ranges. APACHE, acute physiology and chronic health evaluation; ICU, intensive care unit; LOS, length of stay; SAPS, simplified acute physiology score; SOFA, sequential organ failure assessment; SMS–ICU, simplified mortality score for the intensive care unit; SSS, sepsis severity score

The range of scores for sepsis patients was 3–35 for the SMS–ICU and 13–138 for the SSS. Patients with SMS–ICU above 25 or SSS above 90 had 81.3% and 89.1% 90-day mortality rates, respectively. Only 10.6% of the patients had an SMS–ICU score of more than 25, whereas 24.5% of our patients with sepsis had an SSS score greater than 90. The distributions of SMS–ICU and SSS scores coupled with 90-day mortality are shown in Figures 1 and 2, respectively.

Fig. 1 The 90-day mortality stratified by the SMS-ICU

Fig. 2 The 90-day mortality stratified by the SSS

Tables 2 and 3 provide an overview the performance of the SMS–ICU, SSS, and other standard severity scores. In comparison to the SSS, which showed good discrimination (AUROC, 0.876; 95% CI: 0.856–0.896), its AUROC was substantially greater than the SMS–ICU (p < 0.001), the SMS–ICU revealed moderate discrimination in its ability to predict 90-day mortality (AUROC, 0.71; 95% CI: 0.680–0.740). Nonetheless, the standard severity scores had the AUROC significantly higher than the SMS–ICU and SSS for predicting both 90-day and in-hospital mortality (p < 0.001). Furthermore, APACHE IV had the highest AUROC for predicting both outcomes, whereas SMS–ICU had the lowest AUROC (Table 2 and Fig. 3).

Table 2 Predictive performance of the SMS–ICU, SSS, APACHE II, APACHE IV, and SAPS II for 90-day mortality

Severity score	AUROC (95% CI)	SMR (95% CI)	H-Chi2, p-value	C-Chi2, p-value	Brier score	
SMS–ICU	0.710 (0.680–0.740)	1.36 (1.24–1.48)	102.1, <0.0001	79.0, <0.0001	0.222	
SSS	0.876 (0.856–0.896)	1.01 (0.93–1.11)	45.3, <0.0001	8.3, 0.60	0.169	
APACHE II	0.936 (0.922–0.950)	0.97 (0.89–1.06)	43.1, <0.0001	5.6, 0.85	0.099	
APACHE IV	0.957 (0.946–0.968)	0.92 (0.84–1.01)	89.9, <0.0001	10.9, 0.37	0.111	
SAPS II	0.950 (0.938–0.962)	0.96 (0.88–1.05)	74.2, <0.0001	7.3, 0.70	0.092	
APACHE, acute physiology and chronic health evaluation; AUROC, area under the receiver operating characteristic; C, Hosmer–Lemeshow goodness-of-fit C test; CI, confidence interval; H, Hosmer–Lemeshow goodness-of-fit H test; SAPS, simplified acute physiology score; SMS–ICU, simplified mortality score for the intensive care unit; SSS, sepsis severity score

Table 3 Predictive performance of the SMS–ICU, SSS, APACHE II, APACHE IV, and SAPS II for in-hospital mortality

Severity score	AUROC (95% CI)	SMR (95% CI)	H-Chi2, p-value	C-Chi2, p-value	Brier score	
SMS–ICU	0.696 (0.664–0.726)	1.33 (1.22–1.46)	92.5, <0.0001	68.4, <0.0001	0.224	
SSS	0.879 (0.859–0.898)	0.99 (0.91–1.09)	46.2, <0.0001	7.25, 0.70	0.168	
APACHE II	0.941 (0.927–0.954)	0.95 (0.87–1.04)	52.4, <0.0001	5.87, 0.83	0.096	
APACHE IV	0.959 (0.948–0.970)	0.90 (0.82–0.99)	88.4, <0.0001	13.4, 0.20	0.112	
SAPS II	0.952 (0.940–0.963)	0.95 (0.86–1.03)	81.2, <0.0001	7.6, 0.67	0.091	
APACHE, acute physiology and chronic health evaluation; AUROC, area under the receiver operating characteristic; C, Hosmer–Lemeshow goodness-of-fit C test; CI, confidence interval; H, Hosmer–Lemeshow goodness-of-fit H test; SAPS, simplified acute physiology score; SMS–ICU, simplified mortality score for the intensive care unit; SSS, sepsis severity score

Fig. 3 Comparison of the AUROC of the SMS–ICU, SSS, and other severity scores for the 90-day mortality prediction in sepsis patients

For the calibration assessment, the SMS–ICU reported SMRs of 1.36 (95% CI: 1.24–1.48) for the 90-day and 1.33 (95% CI: 1.22–1.46) for in-hospital mortalities, which were underestimated in both mortalities, whereas the APACHE II, SAPS II, and SSS had SMRs between 0.95 and 1.01, which represented appropriate estimations for the 90-day and in-hospital mortalities (Tables 2 and 3). However, the APACHE IV overpredicted the 90-day and in-hospital mortalities with an SMR of 0.92 (95% CI: 0.84–1.01) and 0.90 (95% CI: 0.82–0.99), consecutively. The SMS–ICU demonstrated poor calibration in predicting the 90-day and in-hospital mortalities, similar to that seen with both the disease-specific SSS and commonly used standard severity scores according to the Hosmer–Lemeshow goodness-of-fit H; p < 0.05 (Tables 2 and 3). None of these scores showed good calibration for forecasting the 90-day mortality, as shown in the calibration graph (Fig. 4).

Fig. 4 Comparison of the calibration of the SMS–ICU, SSS, and other standard severity scores for 90-day mortality prediction in sepsis patients

According to the Brier scores, which represent the overall performance, the SMS–ICU had inferior overall performance than the SSS in both 90-day and in-hospital mortality predictions (Tables 2 and 3). The SAPS II presented the best overall performance with Brier score 0.092.

Discussion

For the prediction of 90-day mortality in ICU-admitted sepsis patients, this study demonstrated that the SMS–ICU had moderate discrimination. In comparison to disease-specific severity scores such as SSS and standard severity scores such as SAPS II, APACHE II, and APACHE IV, the SMS–ICU demonstrated lower overall performance and discrimination.

The intricacy of frequently using standard severity scores gradually lessens their clinical usefulness over time.22 Therefore, the development of a simple severity score may be a strategy to improve the use of severity score in ICU patients. Granholm et al. developed the SMS–ICU to predict 90-day mortality.13 This score was obtained from a database of cohort studies and randomized controlled trials that primarily included patients with sepsis. The SMS–ICU contained seven clinical and physiological variables that ranged from 0 to 42. Internal validation showed that the instrument had a high calibration and discrimination of 0.72.13 However, SMS–ICU had lower discrimination than SAPS II, which revealed an AUROC of 0.88 in 1993,17 and SAPS 3, which showed an AUROC of 0.848 in 2005.23

The SMS–ICU was tested for external validation in critically ill patients in Brazil, which was recently published in 2020 and compared with the SAPS 3.15 The findings from the validation cohort demonstrated that the discrimination of the SMS–ICU was good (AUROC, 0.817), although slightly lower than that of the SAPS 3 with an AUROC of 0.845. In terms of calibration, the customized predictions generated by the SMS–ICU showed similarity to those produced by the SAPS 3. However, in the subgroup analysis of Brazilian ICU patients admitted due to infection, the SMS–ICU's discrimination declined to 0.785, which is a moderate discrimination comparable to that of the SMS–ICU in our investigation and the original.

Regarding calibration, the present study showed that the 90-day and in-hospital mortality rates were underestimated by the SMS–ICU. Underestimating the risk could result in inaccurate admission policies and an undervaluation of care quality, performance, and effectiveness, particularly when utilized for benchmarking purposes.24 The reason for the poor calibration may be that the SMS–ICU contained only seven parameters with five dichotomous variables that could not be assessed in detail in our critical sepsis patients, who had high severity scores and higher mortality rates than those in the original study. The 90-day mortality rates of the populations in our study and the original study were 42.4 and 34.3% respectively. Various studies have revealed a correlation between the dose of vasopressors/inotropic agents and mortality;25–28 however, the SMS–ICU only recorded whether vasopressors/inotropes were used. Furthermore, the SMS–ICU only evaluates the usage of respiratory support, and the PaO2/FiO2 ratio is generally one of the measures that correlates with the severity and mortality of respiratory problems.29–31 Our population did not require acute surgical admission, which is one of the domains in the SMS–ICU scoring. Therefore, customizing changes by converting some dichotomous variables to ordinal parameters, such as the stratified dose of vasopressors/inotropic agents, PaO2/FiO2 ratio instead of recording only the use of respiratory support, and considering the collected lactate levels, which revealed significant differences between survivors and deaths on day 90 in our study population, may improve the overall performance of the SMS–ICU. Finally, the accuracy of the performance of these scores in our study of sepsis diagnosis by Sepsis-3 criteria that were developed in 2016 may not be the same result as using alternative sepsis definitions from the original study that developed the SMS–ICU and investigated sepsis/septic shock patients between 2009 and 2016.

Compared to the disease-specific SSS, the SMS–ICU showed moderate discrimination, whereas the SSS showed good discrimination. The SSS model had 34 categorical variables, whereas the SMS–ICU model featured seven domains with five dichotomous variables. The SSS included vital sign parameters for responding to hypotension, tachypnea, and assessing body temperature. In addition, the SSS also contains specific mechanical ventilator parameters, including plateau pressure, as well as crucial parameters related to organ failure, similar to the SOFA score comprising the domain of organ failure, which presented a good discrimination in forecasting hospital death among patients with sepsis.32,33

According to the findings of our study, conventional scoring systems could be employed in populations with sepsis because they provide good discriminating with an AUC above 0.9. Although APACHE IV had the best discrimination, its practical application is difficult due to its reliance on 142 physiological parameters and the need for proprietary computer software to compute the expected mortality. On the contrary, the SAPS II had the best overall performance and was simpler to use because it had fewer variables and the ability to gather these parameters during ICU care, and it had comparable discrimination to APACHE IV. Therefore, the SAPS II may be appropriate for use in ICU sepsis patients.

There were some limitations in our study. First, because the investigation was performed solely in a medical ICU, it is essential to recognize that our findings might not be universally applicable or generalizable to other ICU settings. Therefore, prospective multicenter studies are warranted. Second, this study evaluated sepsis patients solely in the medical ICU; hence, our results could not be representative of surgical ICU sepsis patients for acute surgical admission.

Conclusion

The SMS–ICU indicated lower discrimination and overall performance than the disease-specific SSS and other original severity scores. The calibration of the SMS–ICU was poor and revealed an underestimation when predicting the 90-day mortality in sepsis patients. Consequently, customization of the SMS–ICU is imperative to enhance its performance.

Ethical Approval

This study was approved by the Human Research Ethics Committee of the Faculty of Medicine at Prince of Songkla University, Hat Yai, Songkhla, Thailand (REC 63-293-14-1). The requirement for informed consent was waived because of the retrospective design.

Authors’ Contribution

BK: Conceptualization, data collection, data curation, formal analysis, review and editing manuscript.

NS: formal analysis, manuscript draft.

All authors read and approved the manuscript.

Supplementary Materials

All the supplementary materials are available online on the website of https://www.ijccm.org/journalDetails/IJCCM.

Orcid

Natthaka Sathaporn https://orcid.org/0000-0001-8524-3156

Bodin Khwannimit https://orcid.org/0000-0002-5436-8519

Source of support: Nil

Conflict of interest: None
==== Refs
References

1. De Backer D Biston P Devriendt J Madl C Chochrad D Aldecoa C et al. Comparison of dopamine and norepinephrine in the treatment of shock N Engl J Med 2010 362 9 779 789 10.1056/NEJMoa0907118 20200382
2. Yang WS Kang HD Jung SK Lee YJ Oh SH Kim YJ et al. A mortality analysis of septic shock, vasoplegic shock and cryptic shock classified by the third international consensus definitions (Sepsis-3) Clin Respir J 2020 14 9 857 863 10.1111/crj.13218 32438528
3. Bauer M Gerlach H Vogelmann T Preissing F Stiefel J Adam D Mortality in sepsis and septic shock in Europe, North America and Australia between 2009 and 2019—results from a systematic review and meta-analysis Crit Care 2020 24 1 239 10.1186/s13054-020-02950-2 32430052
4. Le Gall JR The use of severity scores in the intensive care unit Intensive Care Med 2005 31 12 1618 1623 10.1007/s00134-005-2825-8 16244878
5. Vincent JL Moreno R Clinical review: Scoring systems in the critically ill Crit Care 2010 14 2 207 10.1186/cc8204 20392287
6. Rapsang AG Shyam DC Scoring systems in the intensive care unit: A compendium Indian J Crit Care Med 2014 18 4 220 228 10.4103/0972-5229.130573 24872651
7. Breslow MJ Badawi O Severity scoring in the critically ill: Part 1 – Interpretation and accuracy of outcome prediction scoring systems Chest 2012 141 1 245 252 10.1378/chest.11-0330 22215834
8. Parajuli BD Shrestha GS Pradhan B Amatya R Comparison of acute physiology and chronic health evaluation II and acute physiology and chronic health evaluation IV to predict intensive care unit mortality Indian J Crit Care Med 2015 19 2 87 91 10.4103/0972-5229.151016 25722550
9. Osborn TM Phillips G Lemeshow S Townsend S Schorr CA Levy MM et al. Sepsis severity score: An internationally derived scoring system from the surviving sepsis campaign database* Crit Care Med 2014 42 9 1969 1976 10.1097/CCM.0000000000000416 24919160
10. Sathaporn N Khwannimit B Validation the performance of New York sepsis severity score compared with sepsis severity score in predicting hospital mortality among sepsis patients J Crit Care 2019 53 155 161 10.1016/j.jcrc.2019.06.017 31247514
11. Prescott HC Osterholzer JJ Langa KM Angus DC Iwashyna TJ Late mortality after sepsis: Propensity matched cohort study BMJ 2016 353 i2375 10.1136/bmj.i2375 27189000
12. Jammer I Wickboldt N Sander M Smith A Schultz MJ Pelosi P et al. Standards for definitions and use of outcome measures for clinical effectiveness research in perioperative medicine: European Perioperative Clinical Outcome (EPCO) definitions: A statement from the ESA–ESICM joint taskforce on perioperative outcome measures Eur J Anaesthesiol 2015 32 2 88 105 10.1097/EJA.0000000000000118 25058504
13. Granholm A Perner A Krag M Hjortrup PB Haase N Holst LB et al. Development and internal validation of the simplified mortality score for the intensive care unit (SMS–ICU) Acta Anaesthesiol Scand 2018 62 3 336 346 10.1111/aas.13048 29210058
14. Zampieri FG Granholm A Moller MH Scotti AV Alves A Cabral MM et al. Customization and external validation of the simplified mortality score for the intensive care unit (SMS–ICU) in Brazilian critically ill patients J Crit Care 2020 59 94 100 10.1016/j.jcrc.2020.05.016 32585439
15. Singer M Deutschman CS Seymour CW Shankar–Hari M Annane D Bauer M et al. The Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3) JAMA 2016 315 8 801 810 10.1001/jama.2016.0287 26903338
16. Knaus WA Draper EA Wagner DP Zimmerman JE APACHE II: A severity of disease classification system Crit Care Med 1985 13 10 818 829 3928249 3928249
17. Le Gall JR Lemeshow S Saulnier F A new simplified acute physiology score (SAPS II) based on a European/North American multicenter study JAMA 1993 270 24 2957 2963 10.1001/jama.270.24.2957 8254858
18. Keegan MT Gajic O Afessa B Severity of illness scoring systems in the intensive care unit Crit Care Med 2011 39 1 163 169 10.1097/CCM.0b013e3181f96f81 20838329
19. Hanley JA McNeil BJ A method of comparing the areas under receiver operating characteristic curves derived from the same cases Radiology 1983 148 3 839 843 10.1148/radiology.148.3.6878708 6878708
20. Afessa B Gajic O Keegan MT Severity of illness and organ failure assessment in adult intensive care units Crit Care Clin 2007 23 3 639 658 10.1016/j.ccc.2007.05.004 17900487
21. DeLong ER DeLong DM Clarke-Pearson DL Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach Biometrics 1988 44 3 837 845 10.2307/2531595 3203132
22. Harrison DA Brady AR Parry GJ Carpenter JR Rowan K Recalibration of risk prediction models in a large multicenter cohort of admissions to adult, general critical care units in the United Kingdom Crit Care Med 2006 34 5 1378 1388 10.1097/01.CCM.0000216702.94014.75 16557153
23. Moreno RP Metnitz PG Almeida E Jordan B Bauer P Campos RA et al. SAPS 3—From evaluation of the patient to evaluation of the intensive care unit. Part 2: Development of a prognostic model for hospital mortality at ICU admission Intensive Care Med 2005 31 10 1345 1355 10.1007/s00134-005-2763-5 16132892
24. Steyerberg EW Vickers AJ Cook NR Gerds T Gonen M Obuchowski N et al. Assessing the performance of prediction models: A framework for traditional and novel measures Epidemiology 2010 21 1 128 138 10.1097/EDE.0b013e3181c30fb2 20010215
25. Chawla LS Russell JA Bagshaw SM Shaw AD Goldstein SL Fink MP et al. Angiotensin II for the Treatment of high-output shock 3 (ATHOS-3): Protocol for a phase III, double-blind, randomised controlled trial Crit Care Resusc 2017 19 1 43 49 28215131 28215131
26. Khanna A English SW Wang XS Ham K Tumlin J Szerlip H et al. Angiotensin II for the treatment of vasodilatory shock N Engl J Med 2017 377 5 419 430 10.1056/NEJMoa1704154 28528561
27. Benbenishty J Weissman C Sprung CL Brodsky–Israeli M Weiss Y Characteristics of patients receiving vasopressors Heart Lung 2011 40 3 247 252 10.1016/j.hrtlng.2010.04.007 20630594
28. Jenkins CR Gomersall CD Leung P Joynt GM Outcome of patients receiving high dose vasopressor therapy: A retrospective cohort study Anaesth Intensive Care 2009 37 2 286 289 10.1177/0310057X0903700212 19400494
29. Force ADT Ranieri VM Rubenfeld GD Thompson BT Ferguson ND Caldwell E et al. Acute respiratory distress syndrome: The Berlin Definition JAMA 2012 307 23 2526 33 10.1001/jama.2012.5669 22797452
30. Sakr Y Francois B Sole–Violan J Kotfis K Jaschinski U Estella A et al. Temporal changes in the epidemiology, management, and outcome from acute respiratory distress syndrome in European intensive care units: A comparison of two large cohorts Crit Care 2021 25 1 87 10.1186/s13054-020-03455-8 33632247
31. Villar J Blanco J del Campo R Andaluz–Ojeda D Diaz–Dominguez FJ Muriel A et al. Assessment of PaO2/FiO2 for stratification of patients with moderate and severe acute respiratory distress syndrome BMJ Open 2015 5 3 e006812 10.1136/bmjopen-2014-006812
32. Khwannimit B Bhurayanontachai R Vattanavanit V Comparison of the performance of SOFA, qSOFA and SIRS for predicting mortality and organ failure among sepsis patients admitted to the intensive care unit in a middle-income country J Crit Care 2018 44 156 160 10.1016/j.jcrc.2017.10.023 29127841
33. Raith EP Udy AA Bailey M McGloughlin S MacIsaac C Bellomo R et al. Prognostic accuracy of the SOFA score, SIRS criteria, and qSOFA score for in-hospital mortality among adults with suspected infection admitted to the intensive care unit JAMA 2017 317 3 290 300 10.1001/jama.2016.20328 28114553
