
==== Front
JTCVS Open
JTCVS Open
JTCVS Open
2666-2736
Elsevier

S2666-2736(24)00158-X
10.1016/j.xjon.2024.06.001
Adult: Mechanical Circulatory Support
Acute brain injury risk prediction models in venoarterial extracorporeal membrane oxygenation patients with tree-based machine learning: An Extracorporeal Life Support Organization Registry analysis
Kalra Andrew BS ab
Bachina Preetham BS a
Shou Benjamin L. BS a
Hwang Jaeho MD, MPH c
Barshay Meylakh BS d
Kulkarni Shreyas BS d
Sears Isaac BS d
Eickhoff Carsten PhD efg
Bermudez Christian A. MD h
Brodie Daniel MD i
Ventetuolo Corey E. MD, MS j
Kim Bo Soo MD i
Whitman Glenn J.R. MD a
Abbasi Adeel MD, ScM j
Cho Sung-Min DO, MHS csungmi1@jhmi.edu
ak∗
CollaboratorsKim Bo Soo
Hager David
Keller Steven P.
Bush Errol L.
Stephens R. Scott
Khanduja Shivalika
Kang Jin Kook
Chinedozi Ifeanyi David
Darby Zachary
Rando Hannah J.
Brown Trish
Kim Jiah
Wilcox Christopher
Leng Albert
Geeza Andrew
Akbar Armaan F.
Feng Chengyuan Alex
Zhao David
Sussman Marc
Mendez-Tellez Pedro Alejandro
Sun Philip
Capili Karlo
Riojas Ramon
Alejo Diane
Stephen Scott
Flaster Harry
∗
a Division of Cardiac Surgery, Department of Surgery, Johns Hopkins Hospital, Baltimore, Md
b Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, Pa
c Division of Epilepsy, Department of Neurology, Johns Hopkins Hospital, Baltimore, Md
d Warren Alpert Medical School of Brown University, Providence, RI
e Department of Computer Science, Brown University, Providence, RI
f Faculty of Medicine, University of Tübingen, Tübingen, Germany
g Institute for Bioinformatics and Medical Informatics, University of Tübingen, Tübingen, Germany
h Division of Cardiovascular Surgery, Department of Surgery, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pa
i Division of Pulmonary and Critical Care Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, Md
j Division of Pulmonary, Critical Care and Sleep Medicine, Warren Alpert Medical School of Brown University, Providence, RI
k Division of Neurosciences Critical Care, Department of Neurology, Neurosurgery, Anesthesiology and Critical Care Medicine, Johns Hopkins Hospital, Baltimore, Md
∗ Address for reprints: Sung-Min Cho, DO, MHS, Division of Neurosciences Critical Care, The Johns Hopkins Hospital, 600 N Wolfe St, Phipps 455, Baltimore, MD 21287. csungmi1@jhmi.edu
∗ Collaborators: Bo Soo Kim, David Hager, Steven P. Keller, Errol L. Bush, R. Scott Stephens, Shivalika Khanduja, Jin Kook Kang, Ifeanyi David Chinedozi, Zachary Darby, Hannah J. Rando, Trish Brown, Jiah Kim, Christopher Wilcox, Albert Leng, Andrew Geeza, Armaan F. Akbar, Chengyuan Alex Feng, David Zhao, Marc Sussman, Pedro Alejandro Mendez-Tellez, Philip Sun, Karlo Capili, Ramon Riojas, Diane Alejo, Scott Stephen, Harry Flaster.

08 6 2024
8 2024
08 6 2024
20 6488
10 4 2024
23 5 2024
3 6 2024
© 2024 The Author(s)
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Objective

We aimed to determine if machine learning can predict acute brain injury and to identify modifiable risk factors for acute brain injury in patients receiving venoarterial extracorporeal membrane oxygenation.

Methods

We included adults (age ≥18 years) receiving venoarterial extracorporeal membrane oxygenation or extracorporeal cardiopulmonary resuscitation in the Extracorporeal Life Support Organization Registry (2009-2021). Our primary outcome was acute brain injury: central nervous system ischemia, intracranial hemorrhage, brain death, and seizures. We used Random Forest, CatBoost, LightGBM, and XGBoost machine learning algorithms (10-fold leave-1-out cross-validation) to predict and identify features most important for acute brain injury. We extracted 65 total features: demographics, pre-extracorporeal membrane oxygenation/on-extracorporeal membrane oxygenation laboratory values, and pre-extracorporeal membrane oxygenation/on-extracorporeal membrane oxygenation settings.

Results

Of 35,855 patients receiving venoarterial extracorporeal membrane oxygenation (nonextracorporeal cardiopulmonary resuscitation) (median age of 57.8 years, 66% were male), 7.7% (n = 2769) experienced acute brain injury. In venoarterial extracorporeal membrane oxygenation (nonextracorporeal cardiopulmonary resuscitation), the area under the receiver operator characteristic curves to predict acute brain injury, central nervous system ischemia, and intracranial hemorrhage were 0.67, 0.67, and 0.62, respectively. The true-positive, true-negative, false-positive, false-negative, positive, and negative predictive values were 33%, 88%, 12%, 67%, 18%, and 94%, respectively, for acute brain injury. Longer extracorporeal membrane oxygenation duration, higher 24-hour extracorporeal membrane oxygenation pump flow, and higher on-extracorporeal membrane oxygenation partial pressure of oxygen were associated with acute brain injury. Of 10,775 patients receiving extracorporeal cardiopulmonary resuscitation (median age of 57.1 years, 68% were male), 16.5% (n = 1787) experienced acute brain injury. The area under the receiver operator characteristic curves for acute brain injury, central nervous system ischemia, and intracranial hemorrhage were 0.72, 0.73, and 0.69, respectively. Longer extracorporeal membrane oxygenation duration, older age, and higher 24-hour extracorporeal membrane oxygenation pump flow were associated with acute brain injury.

Conclusions

In the largest study predicting neurological complications with machine learning in extracorporeal membrane oxygenation, longer extracorporeal membrane oxygenation duration and higher 24-hour pump flow were associated with acute brain injury in nonextracorporeal cardiopulmonary resuscitation and extracorporeal cardiopulmonary resuscitation venoarterial extracorporeal membrane oxygenation.

Graphical Abstract

Key Words

acute brain injury
Extracorporeal Life Support Organization
extracorporeal membrane oxygenation
machine learning
neurological complications
Abbreviations and Acronyms

ABG arterial blood gas

ABI acute brain injury

AUC-ROC area under the receiver operating characteristic curve

CNS central nervous system

CT computed tomography

ECMO extracorporeal membrane oxygenation

ECPR extracorporeal cardiopulmonary resuscitation

ELSO Extracorporeal Life Support Organization

ICH intracranial hemorrhage

IQR interquartile range

LOOCV leave-1-out-cross-validation

ML machine learning

MRI magnetic resonance imaging

NPV negative predictive value

OR odds ratio

PaO2 partial pressure of oxygen

PPV positive predictive value

SHAP Shapley Additive Explanations

VA venoarterial

VV venovenous
==== Body
pmc Most important factors for predicting ABI in 35,855 patients on VA-ECMO.

Central Message

ML predicted ABI in patients on VA-ECMO with mediocre performance. Nevertheless, it identified longer ECMO duration and higher ECMO pump flow as the most important factors for ABI.

Perspective

Predicting ABI with ML in the ELSO Registry was substandard because of the lack of data granularity. Standardized neurological monitoring and more granular data collection across ELSO centers are important to detect the true prevalence of ABI. Nevertheless, ML identified longer ECMO duration and higher ECMO pump flow as the most important factors for ABI in patients on VA-ECMO.

Extracorporeal membrane oxygenation (ECMO) is increasingly used for cardiopulmonary support.1 Acute brain injury (ABI), which includes central nervous system (CNS) ischemia, intracranial hemorrhage (ICH), and hypoxic-ischemic brain injury, is reported to occur in up to 20% of adult patients receiving venoarterial (VA)-ECMO2 in the Extracorporeal Life Support Organization (ELSO) Registry. Furthermore, this rate is as high as 33% in patients on VA-ECMO using noninvasive multimodal neuromonitoring at a single institution.3 With greater ECMO use and more cases of ABI, accurately predicting ABI with modifiable risk factors such as hyperoxia,4 low pulse pressure,5,6 and hypercarbia7 is important to lessen its occurrence.

In VA-ECMO, there have been several scoring systems developed to predict survival outcomes,8, 9, 10, 11 but their generalizability is limited because they stem from single-center studies, are focused in a specific subset of patients (eg, only cardiogenic shock), and were created from logistic regression. Machine learning (ML) leverages big data to explore patterns and interactions without explicit programming from humans, thus offering distinct advantages to traditional regression.12 Furthermore, coupled with the large sample size of the ELSO Registry, ML may be the most promising technique to adequately synthesize demographic and laboratory information to effectively predict ABI.13 Additionally, identifying variables in the ML model that impact clinical outcomes will inform ECMO clinicians for mitigation of key risk factors for ABI.

Current literature applying ML to predict outcomes in patients receiving ECMO is sparse and primarily focused on non-neurological outcomes such as thrombosis/hemorrhage and mortality.14, 15, 16 An ELSO Registry analysis of patients on VA-ECMO (n = 23,812) demonstrated ML yielded better prediction for in-hospital mortality (area under the receiver operating characteristic curve [AUC-ROC] = 0.80) versus the SAVE score (AUC-ROC = 0.61).15 This study demonstrated the power of ML when applied to the ELSO Registry and provided the impetus for this study designed to test the capability of ML to predict ABI.

We aimed to leverage ML to predict ABI in a large international cohort (the ELSO Registry) of patients receiving ECMO.

Material and Methods

Study Design and Population

The Johns Hopkins Hospital Institutional Review Board approved this retrospective observational study (IRB00216321) with a waiver of informed consent on October 22, 2019. “Retrospective Analysis of Outcomes of Patients on Extracorporeal Membrane Oxygenation” is the study title. All procedures were followed in accordance with the Helsinki Declaration of 1975 and the ethical standards of the responsible committee on human experimentation (institutional or regional). The ELSO Registry is an international multicenter database from more than several hundred ECMO centers worldwide.17 It collects clinical characteristics and demographics, pre-ECMO, and on-ECMO laboratory values such as arterial blood gas (ABG), on-ECMO complications, and outcomes such as in-hospital mortality through voluntary participation. Comorbidity information was captured using the International Classification of Diseases, 10th Revision codes.

We included patients who were aged 18 years of age or more and supported with VA-ECMO for extracorporeal cardiopulmonary resuscitation (ECPR) and non-ECPR indications from 2009 to 2021. We excluded repeat ECMO runs within the same patient to avoid bias and complexity. VA-ECMO and ECPR cohorts were analyzed separately.

Data Collection

In total, 65 variables were collected (Figure 1) for ML. The ELSO Registry collects ABG and hemodynamics pre-ECMO support and on-ECMO. Both pre-ECMO ventilator settings and ABGs were drawn within 6 hours of starting ECMO cannulation. If multiple ABGs existed within a specific period, the pre-ECMO ABG that was nearest to the start of ECMO cannulation was chosen. On-ECMO hemodynamic and ABG information were drawn closest to 24 hours of ECMO support. Values that were meant to be obtained simultaneously such as systolic and diastolic blood pressure and oxygen saturation by pulse oximetry and by ABG were abstracted by a trained ELSO data manager/abstracter from each center and were collected concurrently.Figure 1 All 65 variables incorporated into our ML models including laboratory values, ECMO settings, demographics, other variables, and primary outcome (ABI). BP, Blood pressure; PaCO2, partial pressure of carbon dioxide; PaO2, partial pressure of oxygen; PCWP, pulmonary capillary wedge pressure; DPAP, diastolic pulmonary arterial pressure; SpO2, peripheral oxygen saturation; SvO2, mixed venous oxygen saturation; MPAP, mean pulmonary arterial pressure; SaO2, arterial blood gas oxygen saturation; SPAP, systolic pulmonary arterial pressure; HCO3, bicarbonate; ECMO, extracorporeal membrane oxygenation; FiO2, fraction of inspired oxygen; PEEP, positive-end expiratory pressure; PIP, peak inspiratory pressure; EEG, electroencephalogram.

Definitions

ABI was defined as the presence of infarction (ischemic stroke), diffuse ischemia (hypoxic-ischemic brain injury), intra/extraparenchymal hemorrhage, intraventricular hemorrhage, seizures determined by electroencephalograph or clinically, and neurosurgical intervention (examples include intracranial pressure monitor, external ventricular drain, and craniotomy) during ECMO support. CNS ischemia was defined as ischemic stroke (determined by ultrasound, computed tomography [CT], or magnetic resonance imaging [MRI]) and hypoxic-ischemic brain injury (determined by CT or MRI). ICH was defined as intra/extraparenchymal hemorrhage and intraventricular hemorrhage (both determined by CT or MRI). Definitions for other variables included in our analysis are in the Appendix E1.

Outcomes

The primary outcome was the occurrence of ABI during ECMO support. Secondary outcomes included subtypes of ABI such as CNS ischemia and ICH.

Statistical Analysis

Continuous variables were represented as median with interquartile range (IQR). Categorical variables were presented as frequency with percentages. The Wilcoxon rank-sum and Pearson's chi-square tests were used to compare continuous and categorical variables, respectively.

Data Preprocessing

All categorical variables were 1 hot-encoded before running ML algorithms. Multiple imputation was used for missing data. All missing variables are shown in Table E1.

Machine Learning Algorithm and Pipeline

We examined the suitability of 4 ML algorithms in predicting ABI from the ELSO Registry containing variables from pre-ECMO support and during ECMO support: Random Forest, CatBoost, LightGBM, and XGBoost. For each algorithm, we fine-tuned the hyperparameters and used a Bayesian optimization onto our dataset split randomly into training (70%) and test (30%) sets. Further details are noted in the Appendix E1.

Feature Importance Scores in Machine Learning

To better understand how these ML models were constructed and to determine which variables were most important in predicting ABI, we analyzed which variables were of highest importance in correctly predicting ABI. Specifically, we examined the ranked feature importance in the best performing models, which discloses the contribution of each variable in the composition of the boosted decision trees within the model. We primarily focused on the top 3 most important features for ease of comparison and interpretability for the reader. Furthermore, Feature Importance Scores and Shapley Additive Explanations (SHAP) values depict the contribution of a variable on the predictions of the model (Appendix E1). Both Feature Importance Scores and SHAP values add interpretability to the model framework and reveal pertinent clinical variables associated with ABI. All statistical analyses were performed using R Studio (R 4.1.2, www.r-project.org) and Python.

Results

Venoarterial Extracorporeal Membrane Oxygenation (Nonextracorporeal Cardiopulmonary Resuscitation)

Of 35,855 patients receiving VA-ECMO (non-ECPR), 2769 (8%) had ABI (Table E2, Figure 2). The median age was 57.8 years (IQR, 45.9-66.4), and 66% (n = 23,542) were male. The median duration of ECMO support was 4.3 days (IQR, 2-7.7).Figure 2 Flowchart of study cohort (VA-ECMO and ECPR patients) from the ELSO Registry in 2009-2020. ELSO, Extracorporeal Life Support Organization; ECMO, extracorporeal membrane oxygenation; VV, venovenous; Conversion, VA→VV or VV→VA; VVA, venovenoarterial; Other, mode not defined; VP, venopulmonary; VA, venoarterial; ECPR, extracorporeal cardiopulmonary resuscitation.

Model performance

Using the leave-1-out-cross-validation (LOOCV) 10-fold approach for predicting ABI in VA-ECMO patients, the model achieved an AUC-ROC of 0.67 (Figure 3, A). The accuracy of the model was 83%. The true-positive, true-negative, false-positive, and false-negative rates were 33%, 88%, 12%, and 67%, respectively (Table 1). The positive predictive value (PPV) and negative predictive value (NPV) were 18% and 94%, respectively. The area under the precision recall curve was 0.15. The precision, recall, and F1 were 0.15, 0.38, and 0.22, respectively.Figure 3 Receiver-operating characteristic curves for predicting (A) ABI, (B) CNS ischemia, and (C) ICH in patients receiving VA-ECMO. ECPR, Extracorporeal cardiopulmonary resuscitation.

Table 1 Model performance in the 30% test set of venoarterial extracorporeal membrane oxygenation patients for predicting acute brain injury, central nervous system ischemia, and intracranial hemorrhage

Variable	Accuracy	TPR	TNR	FPR	FNR	PPV	NPV	
ABI	83% (8928/10,757)	33% (3550/10,757)	88% (9466/10,757)	12% (1291/10,757)	67% (7207/10,757)	18% (1963/10,757)	94% (3550/10,757)	
CNS ischemia	86% (9251/10,757)	33% (3550/10,757)	88% (9466/10,757)	12% (1291/10,757)	67% (7207/10,757)	11% (1183/10,757)	97% (10,434/10,757)	
ICH	97% (10,434/10,757)	5% (538/10,757)	99% (10,649/10,757)	1% (108/10,757)	95% (10,219/10,757)	8% (861/10,757)	98% (10,542/10.757)	
ML produced a strong NPV but a poor PPV. Accuracy, True-positive + true-negative/true-positive + true-negative + false-positive + false-negative; TPR, true positive rate; TNR, true negative rate; FPR, false-positive rate; FNR, false-negative rate; PPV, positive predictive value; NPV, negative predictive value; ABI, Acute brain injury; CNS, central nervous system; ICH, intracranial hemorrhage.

For predicting CNS ischemia, the model achieved an AUC-ROC of 0.67 (Figure 3, B). The accuracy of the model was 86%. The true-positive, true-negative, false-positive, and false-negative rates were 33%, 88%, 12%, and 67%, respectively. The PPV and NPV were 11% and 97%, respectively. The area under the precision recall curve was 0.09. The precision, recall, and F1 were 0.11, 0.25, and 0.15, respectively.

For ICH, the model achieved an AUC-ROC of 0.62 (Figure 3, C). The accuracy of the model was 97%. The true-positive, true-negative, false-positive, and false-negative rates were 5%, 99%, 1%, and 95%, respectively. The PPV and NPV were 8% and 98%, respectively. The area under the precision recall curve was 0.03. The precision, recall, and F1 were 0.05, 0.11, and 0.07, respectively.

Feature importance

We identified the top 3 most important variables per Feature Importance Scores and depict the remaining variables (Figure 4, A, Figure E1, A, Table E3). The top 3 variables in predicting ABI were longer duration of ECMO support, higher ECMO pump flow rate at 24 hours, and higher on-ECMO partial pressure of oxygen (PaO2), in predicting CNS ischemia were higher ECMO pump flow rate at 24 hours, pre-ECMO cardiac arrest, and conventional ventilation at 24 hours of ECMO support, and in predicting ICH were longer duration of ECMO support, higher ECMO pump flow rate at 4 hours, and higher on-ECMO PaO2 (Appendix E1, Figure 4, B and C, and Figure E1, Table E3, Table E4, Table E5).Figure 4 Feature importance in increasing importance (ascending) for each neurological outcome: (A) ABI, (B) CNS ischemia, and (C) ICH in patients receiving VA-ECMO. VA-ECMO, Venoarterial extracorporeal membrane oxygenation.

Extracorporeal Cardiopulmonary Resuscitation

Of 10,775 patients receiving ECPR, 1787 (16.5%) had ABI (Figure 1, Table E6). The median age of the ECPR cohort was 57.1 years (IQR, 45.5-65.9), and 68% (n = 7388) were male. The median duration of ECMO support was 2.63 days (IQR, 0.88-5.33).

Model performance

For predicting ABI in patients receiving ECPR, the model achieved an AUC-ROC of 0.72 (Figure E2, A). The accuracy of the model was 69%. The true-positive, true-negative, false-positive, and false-negative rates were 61%, 70%, 30%, and 39%, respectively (Table E7). The PPV and NPV were 29% and 90%, respectively.

For predicting CNS ischemia, the model achieved an AUC-ROC of 0.73 (Figure E2, B). The accuracy of the model was 81%. The true-positive, true-negative, false-positive, and false-negative rates were 41%, 85%, 15%, and 59%, respectively. The PPV and NPV were 18% and 95%, respectively.

For ICH, the model achieved an AUC-ROC of 0.69 (Figure E2, C). The accuracy of the model was 88%. The true-positive, true-negative, false-positive, and false-negative rates were 28%, 89%, 11%, and 72%, respectively. The PPV and NPV were 7% and 98%, respectively.

Feature importance

The top 3 variables for predicting ABI were longer duration of ECMO support, older age, and higher ECMO pump flow rates at 24 hours, and further details are depicted in the Supplement (Figures E3 and E4, Table E10, Table E8, Table E9, Appendix E1).

Exploratory analysis: Features and mortality

A multivariable logistic regression model assessing mortality with the top 3 most important features for ABI in patients on VA-ECMO was constructed for comparison. A longer ECMO duration (adjusted odds ratio [OR], 1.019, 95% CI, 1.014-1.024) and higher on-ECMO PaO2 (adjusted OR, 1.214, 95% CI, 1.185-1.244, both P < .001) level were both associated with increased mortality; higher ECMO pump flow rate at 24 hours (adjusted OR, 1.027, 95% CI, 0.984-1.089, P = .275) was not associated with mortality.

Discussion

This is the first ML study leveraging a large international database to predict ABI in patients receiving ECMO, conveying the novelty and generalizability of our study's results (Figure 5).Figure 5 Summary of key study findings. ML identified longer ECMO duration (in days) and higher 24-hour ECMO pump flow rates as the most important risk factors for ABI in patients receiving VA-ECMO. Better standardized neurological monitoring is required to detect the true prevalence across ELSO centers. VA-ECMO, Venoarterial extracorporeal membrane oxygenation; ECPR, extracorporeal cardiopulmonary resuscitation.

Venoarterial Extracorporeal Membrane Oxygenation Versus Venovenous Extracorporeal Membrane Oxygenation Risk Factors

ML uniquely identified longer duration of ECMO support (in hours), higher ECMO pump flow rate at 24 hours of ECMO support, and higher on-ECMO 24-hour PaO2 as the top 3 most important variables associated with ABI. Although ECMO duration is not necessarily a modifiable risk factor, it is still an important feature to monitor because a difference in 12 hours is a clinically significant difference, as previously shown in another ELSO Registry analysis.18 Because patients receiving venovenous (VV)-ECMO have been shown to be cannulated longer than patients on VA-ECMO,19, 20, 21 the longer ECMO duration and lower risk of ABI associated may be due to the withdrawal of life-sustaining therapy for severely sick patients.22,23 Accordingly, this may have created a selection bias for patients who did undergo ABI and survived on ECMO support for longer. Furthermore, a higher ECMO pump flow rate and likely corresponding hemolysis24,25 were uniquely important for ABI in VA-ECMO and ECPR, but not in VV-ECMO. This finding may reflect the different hemodynamic/physiological states24, 25, 26 and use/disuse of an aortic cannula27 in VA-ECMO versus VV-ECMO populations and warrants further study. Although pre-ECMO cardiac arrest is a known risk factor for CNS ischemia in patients receiving ECPR,2,28 likely related to reperfusion injury and associated reactive oxygen species formation,28,29 we also note that this factor was highly important in patients receiving VV-ECMO,30 which has not been previously reported. These comparisons suggest there are similar underlying but overall divergent risk factors between these populations, which necessitates further investigation with prospective observational studies. Hyperoxia (PaO2 was treated as a continuous variable to avoid bias due to “data binning”31) is associated with increased risk of ABI due to generation of reactive oxygen species29 and impairment of hippocampal oxidative energy metabolism,32 which accentuate reperfusion injury, as suggested in a previous ELSO Registry analysis4 and at a tertiary academic ECMO center.33 Notably, central cannulation was the tenth most important feature for CNS ischemia, which is in line with previous literature demonstrating differences in rates of ABI based on cannulation strategy,34 although other studies demonstrate no significant differences in neurological injury between both strategies.35,36 Finally, older age was associated with an increased risk of ABI, which agrees with a 2017-2019 ELSO Registry analysis (n = 15,172) of patients on VA-ECMO that demonstrated older age was associated with higher complication rates.37

Machine Learning Methodologies

We chose tree-based ML algorithms to predict ABI, which are becoming more commonly used in healthcare studies38 because they provide an effective way to consider all different possible outcomes in a model. There are several specific advantages of tree-based ML algorithms over nontree-based models, including (1) the ability to input a wide variety of data (ie, both continuous and categorical); (2) the capability to handle data that is complex, nonlinear, and not normally distributed; (3) the ability to easily visualize complex data through Feature Importance and SHAP value plots; (4) they do not require extensive data cleaning and preparation because data variable transformations are not required; and (5) their ability to capture subtle data patterns by separating features into mutually exclusive and distinctive regions.39, 40, 41, 42 Additionally, recent data have suggested that tree-based ML models may be statistically significantly superior than nontree-based ML models with tabular data.43 Furthermore, these tree-based ML models demonstrate high power and good accuracy, and provide interpretability to the models.44 The primary difference between using Random Forest versus gradient boosting tree methods is that Random Forest trees are constructed in an independent fashion while gradient boosting methods are created sequentially. Accordingly, Random Forest can determine their outputs without restriction of order, whereas gradient boosting methods like XGBoost are restricted in a more fixed manner. There are also key differences within boosting methods: CatBoost may be most optimal for categorical data and can generate output more quickly than XGBoost or LightGBM. LightGBM demonstrates better accuracy and speed than XGBoost, but XGBoost is the more established ML algorithm, perhaps making it a reliable ML tree-based method. Nevertheless, despite implementing these 4 powerful and innovative methods with oversampling to enhance statistical power, ML could still not accurately predict ABI in the ELSO Registry. This finding may suggest that the ELSO Registry does not capture causative variables for ABI over the entire duration of ECMO support that are needed to fully glean the insights and advantages of ML and ultimately identify modifiable risk factors for ABI. Finally, we note that although ML did not predict ABI with high accuracy, it did produce a strong NPV (94% and 90% for ABI in VA-ECMO and ECPR, respectively), suggesting our models’ true utility may lie in its high sensitivity and capability to rule out patients who truly did not have ABI. Furthermore, our models also demonstrated high true-negative rates (88% and 70% for ABI, and 99% and 89% for ICH, in VA-ECMO and ECPR, respectively), which also suggests a high specificity and capability to rule patients in with ABI accurately. Therefore, implementing this model as a screening test may be warranted and useful for ECMO clinicians.

Lack of Standardized Neurological Monitoring

Given the relatively mediocre performance in predicting ABI and its subtypes in both cohorts, we reveal certain limitations using a heterogenous, large dataset such as the ELSO Registry to predict ABI with ML. Specifically, unlike the institutional protocol at Johns Hopkins Hospital that uses standardized neurological monitoring with proven efficacy,3 the protocols used to determine ABI across ECMO centers are neither standardized nor adjudicated/validated, and thus vary considerably. Accordingly, we observed only a 7.7% prevalence of ABI in patients on VA-ECMO and 16.5% prevalence of ABI in patients on ECPR within the ELSO Registry, which is considerably less than the prevalence of 33% at an experienced tertiary care ECMO center.3 Therefore, this study calls for more sensitive and accurate detection of ABI and more granular collection of variables across ECMO centers. ABI can precede mortality and therefore identifying risk factors for ABI can help clinicians mitigate their occurrence and their associated mortality risk. In fact, a single-center study of 106 pediatric patients on VA-ECMO and 68 pediatric patients on VV-ECMO using ML to predict CNS ischemia and ICH showed a superior AUC-ROC (0.76) than ours with the ELSO Registry (0.67).45 This result may not be surprising given the institution's rigorous advanced neuroimaging technique to determine ABI and adjudication system by multiple clinicians. Accordingly, their prevalence of ABI (51% in VA/VV-ECMO mixed population) was higher than ours with the ELSO Registry (7.7% in VA-ECMO and 16.5% in ECPR). Overall, an ELSO Registry addendum for neurological monitoring and imaging protocols may improve performance for ML to predict ABI. Furthermore, we suggest that all ELSO centers use standardized neurological monitoring protocols to better detect the true prevalence of ABI (and capture it more accurately in the ELSO Registry) and ultimately mitigate this devastating outcome for patients.

Study Limitations

The primary limitation of our analysis was the lack of standardized neurological monitoring protocols across ECMO centers and lack of ABI adjudication in the ELSO Registry. Because ABI is defined by imaging findings in the Registry, the quality control of ABI is likely good. However, there is still underestimation of ABI in the Registry because many patients do not obtain proper neuroimaging studies in the first place. A fundamental limitation of this study was that model performance in VA-ECMO for predicting ABI, CNS ischemia, and ICH was poor due to low PPV. Given the relatively low outcome rates of ABI and its subtypes, these outcome variables likely have substantial class imbalance and thus make ML models predicting ABI challenging. Accordingly, we saw improved performance with ML predicting ABI and CNS ischemia versus ICH in patients receiving VA-ECMO, likely due to their higher prevalence; similarly, patients receiving ECPR observed improved ML performance, which is logical because of their higher prevalence of ABI overall and its subtypes relative to patients without ECPR VA-ECMO. Furthermore, the ELSO Registry lacks granularity with laboratory measurements because ABGs are only collected at a singular time point instead of multiple times throughout the ECMO run and were not collected at the same exact time point at each center. We also acknowledge that cross-sectionally the ECMO pump flow rates were small and may not be clinically meaningful, but these differences were still statistically significant in our model and should be noted. Finally, because this was a retrospective study, only associations could be determined.

Conclusions

Using the largest database of ECMO patients globally, we present the first study to predict neurological outcomes on sufficiently powered international ECMO patient cohorts. ML identified longer ECMO duration and higher pump flow rates as the most important risk factors for ABI in both VA-ECMO and ECPR cohorts. Overall, performance of ML models to predict ABI in patients receiving VA-ECMO and ECPR was suboptimal likely because of the lack of data granularity in the ELSO Registry. This finding suggests that the detection and sensitivity rates for capturing ABI in patients receiving ECMO across ECMO centers worldwide is substandard. Accordingly, standardized neurological monitoring and imaging protocols are urgently needed.

Conflict of Interest Statement

D.B. receives research support from and consults for LivaNova; has been on the medical advisory boards for Xenios, Medtronic, Inspira, and Cellenkos; is the President-elect of the ELSO and the Chair of the Executive Committee of the International ECMO Network (ECMONet); and writes for UpToDate. C.E.V. has been a consultant or served on advisory boards for Merck, Janssen, and Regeneron, outside of the submitted work. S.M.C. is supported by the National Heart, Lung, and Blood Institute (1K23HL157610) and Hyperfine (SAFE MRI ECMO study). All other authors reported no conflicts of interest.

The Journal policy requires editors and reviewers to disclose conflicts of interest and to decline handling or reviewing manuscripts for which they may have a conflict of interest. The editors and reviewers of this article have no conflicts of interest.

Supplemental Results

Feature Importance in Venoarterial Extracorporeal Membrane Oxygenation

The median ECMO duration was higher in patients with ABI versus patients without ABI (4.8 vs 4.3 days, P < .001). The median ECMO pump flow rate at 24 hours was higher in patients with ABI versus patients without ABI (4 vs 3.95 L/min, P < .001). The median on-ECMO PaO2 was higher in patients with ABI versus patients without ABI (162 vs 141 mm Hg, P < .001). The median ECMO pump flow rate at 24 hours was higher in patients with CNS ischemia versus patients without CNS ischemia (4 vs 3.95 L/min, P < .001). The prevalence of CNS ischemia in patients with pre-ECMO cardiac arrest was higher than patients without cardiac arrest (5.8% vs 3.3%, P < .001). The prevalence of CNS ischemia in patients with conventional venting at 24 hours of ECMO support was higher than patients without conventional venting at 24 hours of ECMO support (8.6% vs 2.7%, P < .001). The median ECMO duration was higher in patients with ICH versus patients without ICH (6 vs 4.3 days, P < .001). The median ECMO pump flow rate at 4 hours was higher in patients with ICH versus patients without ICH (3.98 vs 3.82 L/min, P < .001). The median on-ECMO PaO2 was similar between patients with ICH versus patients without ICH (151 vs 142 mm Hg, P = .27).

Exploratory Analysis: Hyperoxia in Venoarterial Extracorporeal Membrane Oxygenation

Patients receiving VA-ECMO with ABI were more likely to have hyperoxia (>120 mm Hg at 24 hours of cannulation, n = 1475, 53%) than patients without ABI (n = 14,822, 45%, P < .001). The median MAP was slightly lower in patients with ABI with hyperoxia (12 mm Hg) versus the median MAP in patients with ABI without hyperoxia (13 mm Hg, P = .003).

Feature Importance in Extracorporeal Cardiopulmonary Resuscitation

The median ECMO duration was higher in patients with ABI versus patients without ABI (3.1 vs 2.5 days, P < .001). Patients with ABI were older versus patients without ABI (median age of 57.7 vs 54.4 years, P < .001). The median ECMO pump flow rate at 24 hours of ECMO support was higher in patients with ABI versus patients without ABI (3.8 vs 3.6 L/min, P < .001). The top 3 variables for predicting CNS ischemia were duration of ECMO support, serum bicarbonate level at 24 hours of ECMO support, and body mass index (Figure E2, B, Figure E3, B, Table E8). The median ECMO duration was higher in patients with CNS ischemia versus those without CNS ischemia (3.3 vs 2.5 days, P < .001). Patients with CNS ischemia had similar levels of serum bicarbonate at 24 hours of ECMO support as patients without CNS ischemia (23 vs 23 mEq/L, P = .47). Patients with CNS ischemia had a higher median body mass index than patients without CNS ischemia (29.1 vs 27.6 kg/m2, P < .001). The top 3 variables for predicting ICH were being supported on ECMO at a North American ELSO center, positive-end expiratory pressure at 24 hours of ECMO support, and being supported on ECMO at a European ELSO center (Figure E2, C, Figure E3, C, Table E9). The prevalence of ICH was higher in patients supported on ECMO at a North American ELSO Center versus those not supported on ECMO at a North American ELSO Center (3.3% vs 1.7%, P < .001). The median positive-end expiratory pressure at 24 hours of ECMO support for patients with ICH was not different than that of patients without ICH (8 vs 8 mm Hg, P = .25). The prevalence of ICH was lower in patients supported on ECMO at a European ELSO Center versus those not supported on ECMO at a European ELSO center (1.2% vs 3%, P < .001).Figure E1 SHAP value plots for (A) ABI, (B) CNS ischemia, and (C) ICH in patients receiving VA-ECMO. VA-ECMO, Venoarterial extracorporeal membrane oxygenation; ECMO, extracorporeal membrane oxygenation; PaO2, partial pressure of oxygen; HCO3, bicarbonate; BP, blood pressure; SaO2, arterial blood gas oxygen saturation; PIP, peak inspiratory pressure; SvO2, mixed venous oxygen saturation; DBP, diastolic blood pressure.

Figure E2 ROC curves for predicting (A) ABI, (B) CNS ischemia, and (C) ICH in patients receiving ECPR. ECPR, Extracorporeal cardiopulmonary resuscitation.

Figure E3 Feature importance scores for (A) ABI, (B) CNS ischemia, and (C) ICH in patients receiving ECPR. ECPR, Extracorporeal cardiopulmonary resuscitation; ECMO, extracorporeal membrane oxygenation; PaO2, partial pressure of oxygen; HCO3, bicarbonate; BP, blood pressure; SaO2, arterial blood gas oxygen saturation; PIP, peak inspiratory pressure; SvO2, mixed venous oxygen saturation; DBP, diastolic blood pressure.

Figure E4 SHAP value plots for (A) ABI, (B) CNS ischemia, and (C) ICH in patients receiving ECPR. ECPR, Extracorporeal cardiopulmonary resuscitation; ECMO, extracorporeal membrane oxygenation; PaO2, partial pressure of oxygen; HCO3, bicarbonate; BP, blood pressure; SaO2, arterial blood gas oxygen saturation; PIP, peak inspiratory pressure; SvO2, mixed venous oxygen saturation; DBP, diastolic blood pressure.

Table E1 Variables with missingness in Extracorporeal Life Support Organization Registry for all adult patients receiving extracorporeal membrane oxygenation from 2009 to 2021

Variable	Missing	X (%)	
Pulmonary capillary wedge pressure at 24 h	87,017	99	
Pre-ECMO pulmonary capillary wedge pressure	86,774	98	
Pre-ECMO cardiac index	82,670	94	
Cardiac index at 24 h	81,750	93	
Pre-ECMO mean pulmonary arterial pressure	80,178	91	
Pre-ECMO mixed venous oxygen saturation	79,730	90	
Pre-ECMO diastolic pulmonary arterial pressure	78,978	90	
Pre-ECMO systolic pulmonary arterial pressure	78,845	89	
Mixed venous oxygen saturation at 24 h	76,111	86	
Diastolic pulmonary arterial pressure at 24 h	75,479	86	
Systolic pulmonary arterial pressure at 24 h	75,388	86	
Mixed venous oxygen saturation at 24 h	66,204	75	
Pre-ECMO peripheral oxyhemoglobin saturation	65,314	74	
Peripheral oxyhemoglobin saturation at 24 h	60,599	69	
Pre-ECMO mean airway pressure	56,242	64	
Pre-ECMO lactate	53,670	61	
Lactate at 24 h	48,005	54	
Time to extubation	47,511	54	
Pre-ECMO peak inspiratory pressure	45,232	51	
Mean airway pressure at 24 h	43,657	50	
Pre-ECMO positive end-expiratory pressure	34,613	39	
Pre-ECMO mean blood pressure	34,500	39	
Pre-ECMO ventilation rate	34,263	39	
Peak inspiratory pressure at 24 h	32,346	37	
Pre-ECMO arterial oxyhemoglobin saturation	32,126	36	
Patient being transported to ELSO center	31,678	36	
Pre-ECMO percentage of inspired oxygen	28,816	33	
Height	26,604	30	
Pre-ECMO diastolic blood pressure	26,570	30	
Pre-ECMO systolic blood pressure	26,270	30	
Arterial oxyhemoglobin saturation at 24 h	24,642	28	
Mean blood pressure at 24 h	24,149	27	
Pre-ECMO serum bicarbonate	23,588	27	
Pre-ECMO PaO2	22,914	26	
Pre-ECMO partial pressure of carbon dioxide	22,713	26	
Ventilation rate at 24 h	22,255	25	
Positive end-expiratory pressure at 24 h	21,837	25	
Diastolic blood pressure at 24 h	20,687	23	
Pre-ECMO pH	20,641	23	
Systolic blood pressure at 24 h	20,582	23	
Percentage of inspired oxygen at 24 h	20,430	23	
PaO2 at 24 h	17,543	20	
Partial pressure of carbon dioxide at 24 h	17,432	20	
Serum bicarbonate at 24 h	16,402	19	
ECMO pump flow rate at 24 h	15,935	18	
pH at 24 h	15,283	17	
Time to intubation	14,839	17	
ECMO pump flow rate at 4 h	11,937	14	
Weight	3116	4	
ECMO duration	78	0	
Patient ID	0	0	
Run ID	0	0	
Run number	0	0	
Sex	0	0	
Race/ethnicity	0	0	
Age	0	0	
Primary diagnosis by ICD-10	0	0	
Primary diagnosis by ICD-9	0	0	
ECMO modality	0	0	
Support type	0	0	
Discontinuation of ECMO	0	0	
Discharged alive off of ECMO	0	0	
Discharge location	0	0	
Year on ECMO	0	0	
Pre-ECMO ventilation type	0	0	
Pre-ECMO handbagging	0	0	
Vent type at 24 h	0	0	
Handbagging at 24 h	0	0	
Pre-ECMO cardiac arrest	0	0	
Bridged to transplant as indication for ECMO	0	0	
ID of ELSO center	0	0	
Continent of chapter name	0	0	
Trauma as indication for ECMO	0	0	
Placement of artificial airway during ECMO	0	0	
ECMO, Extracorporeal membrane oxygenation; ELSO, Extracorporeal Life Support Organization; ICD-10, International Classification of Diseases, 10th Revision; ICD-9, International Classification of Diseases, 9th Edition.

Table E2 Baseline characteristics and clinical variables of patients on venoarterial extracorporeal membrane oxygenation stratified by presence of acute brain injury

Variable	Total VA-ECMO (no ECPR) (n = 35,855)	ABI (n = 2769, 8%)	No ABI (n = 33,086, 92%)	P value	
Demographics					
 Age (y)	57.80 (45.9-66.4)	56.1 (43.2-64.8)	57.9 (46.1-66.6)	<.001	
 Male sex	23,542 (66%)	1726 (62%)	21,817 (66%)	<.001	
 Body mass index, kg/m2	27.8 (24.1-32.6)	28.4 (24.5-33.1)	27.8 (24.1-32.5)	<.001	
Race/ethnicity				<.001	
 Asian	4763 (13%)	319 (12%)	4445 (13%)		
 Black	3560 (10%)	327 (12%)	3234 (10%)		
 Hispanic	1941 (5%)	160 (6%)	1782 (5%)		
 White	20,133 (56%)	1605 (58%)	18,529 (56%)		
 Others	5458 (15%)	358 (13%)	5096 (15%)		
Year ECLS				<.001	
 2009	319 (1%)	283 (10%)	36 (1%)		
 2010	448 (1%)	398 (14%)	50 (1%)		
 2011	646 (2%)	578 (21%)	68 (1%)		
 2012	1093 (3%)	991 (36%)	102 (1%)		
 2013	1339 (4%)	129 (5%)	1210 (4%)		
 2014	1796 (5%)	166 (6%)	1630 (5%)		
 2015	2483 (7%)	212 (8%)	2271 (7%)		
 2016	3090 (9%)	242 (9%)	2848 (9%)		
 2017	4128 (12%)	259 (9%)	3869 (12%)		
 2018	4651 (13%)	325 (12%)	4326 (13%)		
 2019	5581 (16%)	404 (15%)	5177 (16%)		
 2020	5189 (14%)	387 (14%)	4802 (15%)		
 2021	5092 (14%)	389 (14%)	4703 (14%)		
Medical history					
 Diabetes	2924 (8%)	252 (9%)	2672 (8%)	.06	
 Hypertension	4205 (12%)	382 (14%)	3823 (12%)	<.001	
 Atrial fibrillation	3083 (9%)	218 (8%)	2865 (9%)	.16	
 Cardiomyopathy	3413 (10%)	248 (9%)	3165 (10%)	.30	
 COPD	1083 (3%)	66 (2%)	1017 (3%)	.04	
Pre-ECMO support					
 Additional temporary mechanical circulatory support	11,730 (33%)	973 (35%)	10,757 (33%)	.005	
 Vasopressor infusions	22,584 (63%)	1876 (68%)	20,708 (63%)	<.001	
 Inotrope infusions	11,503 (32%)	824 (30%)	10,679 (32%)	.006	
Pre-ECMO blood pressure variables					
 Systolic blood pressure (mm Hg)	87 (72-104)	85 (70-103)	87 (72-104)	<.001	
 Diastolic blood pressure (mm Hg)	54 (43-65)	52 (42-64)	54 (44-65)	<.001	
 Mean blood pressure (mm Hg)	65 (54-76)	63 (53-75)	65 (54-76)	.001	
 Pulse pressure (mm Hg)	32 (20-45)	31 (20-43)	32 (20-45)	.053	
 Mean arterial pressure (mm Hg)	14 (10-18)	14 (11-19)	14 (10-18)	.03	
Pre-ECMO ABG					
 pH	7.29 (7.18-7.38)	7.26 (7.14-7.35)	7.29 (7.19-7.38)	<.001	
 HCO3- (mEq/L)	20 (16-23.2)	19 (15.1-22.9)	20 (16-23.4)	<.001	
 PaO2 (mm Hg)	103 (68-217.5)	93.95 (62-212)	104 (68-218)	<.001	
 PaCO2 (mm Hg)	41 (33.80-50)	42.2 (34-54)	41 (33.7-50)	<.001	
 Lactate (mmol/L)	6.1 (2.9-10.8)	6 (2.8-10.7)	8 (3.8-12)	<.001	
 SpO2 (%)	98 (92-100)	97 (89-100)	98 (93-100)	<.001	
 SaO2 (%)	97 (90-100)	96 (86-99)	97 (91-99)	<.001	
On-ECMO blood pressure variables					
 Systolic blood pressure (mm Hg)	96 (84-110)	94 (81-108)	96 (84-110)	<.001	
 Diastolic blood pressure (mm Hg)	64 (55-72)	64 (56-73)	64 (55-72)	.04	
 Mean blood pressure (mm Hg)	74 (67-81)	73 (66-81)	74 (67-81)	.001	
 Pulse pressure (mm Hg)	31 (18-46)	28 (15-44)	31 (18-46)	.053	
 Mean arterial pressure (mm Hg)	12 (10-15)	13 (10-15)	12 (10-15)	<.001	
On-ECMO ABG					
 pH	7.42 (7.37-7.46)	7.41 (7.36-7.46)	7.42 (7.37-7.47)	.005	
 HCO3- (mEq/L)	24.1 (21.7-27)	24 (21-27)	24.1 (21.8-27)	.02	
 PaO2 (mm Hg)	142 (91.8-250)	162 (94.1-297.57)	141 (91.5-244.2)	<.001	
 PaCO2 (mm Hg)	38 (33.3-42)	38 (33-42.5)	38 (33.3-42)	.50	
 Lactate (mmol/L)	2.3 (1.4-4.4)	3.1 (1.8-5.7)	2.3 (1.4-4.2)	<.001	
 SpO2 (%)	99 (97-100)	99 (97-100)	99 (97-100)	.30	
 SaO2 (%)	98 (97-99)	99 (97-100)	98 (97-99)	.007	
ΔPaCO2	−3 (−12 to 4.7)	−4 (−16 to 3)	−2.9 (−12 to 5)	<.001	
Pump flow rate (4 h, L/min)	3.83 (3.17-4.42)	3.9 (3.2-4.48)	3.82 (3.16-4.41)	.01	
Pump flow rate (24 h, L/min)	3.24 (3.96-4.5)	4 (3.34-4.6)	3.95 (3.22-4.5)	<.001	
Days on ECMO support	4.33 (2-7.71)	4.83 (2.5-8.67)	4.29 (2-7.63)	<.001	
Neurological complications on-ECMO					
 Composite ABI					
 Composite Ischemia	1459 (4%)	1459 (53%)	0 (0%)	<.001	
 Hypoxic-ischemic brain injury	280 (1%)	280 (10%)	0 (0%)	<.001	
 Ischemic stroke	1194 (3%)	1194 (43%)	0 (0%)	<.001	
 Composite ICH	792 (2%)	792 (29%)	0 (0%)	<.001	
 Intra-/extra-parenchymal hemorrhage	269 (1%)	269 (10%)	0 (0%)	<.001	
 Intraventricular hemorrhage	108 (1%)	108 (4%)	0 (0%)	<.001	
 Brain death	659 (2%)	659 (24%)	0 (0%)	<.001	
 Neurosurgical intervention	31 (1%)	31 (1%)	0 (0%)	<.001	
 Seizures confirmed by EEG	31 (1%)	31 (1%)	0 (0%)	<.001	
 Seizures clinically determined	188 (1%)	188 (7%)	0 (0%)	<.001	
Other complications on-ECMO					
 ECMO circuit mechanical failure	4413 (12%)	472 (17%)	3941 (12%)	<.001	
 Renal replacement theory	9446 (26%)	1092 (39%)	8354 (25%)	<.001	
 Hemolysis	1303 (4%)	159 (6%)	1144 (3%)	<.001	
 Cardiac arrhythmia	4152 (12%)	474 (17%)	3678 (11%)	<.001	
 Gastrointestinal hemorrhage	1338 (4%)	174 (6%)	1164 (4%)	<.001	
Outcomes					
 In-hospital mortality	19,030 (53%)	2320 (84%)	16,710 (51%)	<.001	
Bolded P values represent a statistically significant association (P <.05). VA-ECMO, Venoarterial extracorporeal membrane oxygenation; ECPR, extracorporeal cardiopulmonary resuscitation; ABI, acute brain injury; ECLS, Extracorporeal Life Support; COPD, chronic obstructive pulmonary disorder; ABG, arterial blood gas; HCO3, bicarbonate; PaCO2, partial pressure of carbon dioxide; SpO2, peripheral oxygen saturation; SaO2, arterial blood gas oxygen saturation; Δ, delta; ICH, intracranial hemorrhage; EEG, electroencephalogram.

Table E3 Comparisons among the top 3 most important features for acute brain injury in patients on venoarterial extracorporeal membrane oxygenation

Variable	With ABI	Without ABI	P value	
Median ECMO duration	4.8 d	4.3 d	<.001	
Median ECMO pump flow rate at 24 h	4 L/min	3.95 L/min	<.001	
Median on-ECMO PaO2	162 mm Hg	141 mm Hg	<.001	
ABI, Acute brain injury; ECMO, extracorporeal membrane oxygenation; PaO2, partial pressure of oxygen.

Table E4 Comparisons among the top 3 most important features for central nervous system ischemia in patients on venoarterial extracorporeal membrane oxygenation

Variable	With CNS ischemia	Without CNS ischemia	P value	
Median ECMO pump flow rate at 24 h	4 L/min	3.95 L/min	<.001	
Pre-ECMO cardiac arrest	5.8% (n = 633)	N/A	<.001	
Without pre-ECMO cardiac arrest	3.3% (n = 796)	N/A		
With conventional venting at 24 h	8.6% (n = 2342)	N/A	<.001	
Without conventional venting at 24 h	2.7% (n = 44)	N/A		
CNS, Central nervous system; ECMO, extracorporeal membrane oxygenation; N/A, not available.

Table E5 Comparisons among the top 3 most important features for intracranial hemorrhage in patients on venoarterial extracorporeal membrane oxygenation

Variable	With ICH	Without ICH	P value	
Median ECMO duration	6 d	4.3 d	<.001	
Median ECMO pump flow rate at 4 h	3.98 L/min	3.82 L/min	<.001	
Median on-ECMO PaO2	151 mm Hg	142 mm Hg	.27	
ICH, Intracranial hemorrhage; ECMO, extracorporeal membrane oxygenation.

Table E6 Baseline characteristics and clinical variables among patients on extracorporeal cardiopulmonary resuscitation stratified by presence of acute brain injury

Variable	Total ECPR (n = 10,775)	ABI (n = 1787, 17%)	No ABI (n = 8988, 83%)	P value	
Demographics					
 Age (y)	57.1 (45.5-65.9)	57.70 (46.30-66.50)	54.40 (41.50-63.00)	<.001	
 Male sex	7388 (68%)	1273 (71%)	6116 (68%)	.008	
 Body mass index, kg/m2	27.68 (24.22-32.46)	28.29 (24.91-33.44)	27.55 (24.22-32.19)	<.001	
Race/ethnicity				.002	
 Asian	2093 (19%)	319 (18%)	1775 (20%)		
 Black	993 (9%)	197 (11%)	797 (9%)		
 Hispanic	425 (4%)	89 (5%)	337 (4%)		
 White	5855 (54%)	956 (53%)	4900 (55%)		
 Others	1409 (13%)	226 (13%)	1179 (13%)		
Year ECLS				<.001	
 2009	83 (1%)	27 (2%)	56 (1%)		
 2010	102 (1%)	21 (1%)	81 (1%)		
 2011	147 (1%)	38 (2%)	109 (1%)		
 2012	241 (2%)	54 (3%)	187 (2%)		
 2013	442 (4%)	85 (5%)	357 (4%)		
 2014	497 (5%)	82 (5%)	415 (5%)		
 2015	813 (8%)	143 (8%)	670 (7%)		
 2016	927 (9%)	159 (9%)	768 (9%)		
 2017	1189 (11%)	158 (9%)	1031 (11%)		
 2018	1443 (13%)	215 (12%)	1228 (14%)		
 2019	1911 (18%)	301 (17%)	1580 (18%)		
 2020	1580 (15%)	272 (15%)	1308 (15%)		
 2021	1400 (13%)	232 (13%)	1168 (13%)		
Medical history					
 Diabetes	872 (8%)	173 (10%)	699 (8%)	.007	
 Hypertension	1148 (11%)	234 (13%)	914 (10%)	<.001	
 Atrial fibrillation	550 (5%)	93 (5%)	457 (5%)	.83	
 Cardiomyopathy	518 (5%)	104 (6%)	414 (5%)	.03	
 COPD	214 (2%)	42 (2%)	172 (2%)	.23	
Pre-ECMO support					
 Additional temporary mechanical circulatory support	1420 (13%)	231 (13%)	1189 (13%)	.73	
 Vasopressor infusions	6393 (59%)	1068 (60%)	5325 (59%)	.68	
 Inotrope infusions	1371 (13%)	215 (12%)	1156 (13%)	.34	
Pre-ECMO blood pressure variables					
 Systolic blood pressure (mm Hg)	82 (60-108)	80 (57-109)	83 (60-108)	.18	
 Diastolic blood pressure (mm Hg)	50 (33-66)	48 (30-67)	50 (33-66)	.3695	
 Mean blood pressure (mm Hg)	58 (40-74)	58 (41-80)	58 (40-74)	.04	
 Pulse pressure (mm Hg)	30 (19-47)	30 (19-44)	30 (19-47)	.2177	
 Mean arterial pressure (mm Hg)	14 (11-18)	13 (10-18)	14 (11-18)	.1473	
Pre-ECMO ABG					
 pH	7.16 (7.00-7.30)	7.090 (6.920-7.250)	7.170 (7-7.310)	<.001	
 HCO3- (mEq/L)	17.60 (13.00 -22.00)	17.00 (12.95-21.35)	17.7 (13.0-22.0)	.05333	
 PaO2 (mm Hg)	76.0 (51.0-137.4)	67.7 (45.0-118.5)	77.2 (52.0-144)	<.001	
 PaCO2 (mm Hg)	49.00 (36.00-68.00)	55.00 (39.00-76.20)	48.00 (35.30-66.00)	<.001	
 Lactate (mmol/L)	10.30 (5.00-14.60)	11.60 (7.425-15.475)	10.00 (5.80-14.32)	<.001	
 SpO2 (%)	94 (81-99)	91 (77-99)	94 (82-99)	.02	
 SaO2 (%)	92 (76-98)	88 (67-97)	93 (78-98)	<.001	
On-ECMO blood pressure variables					
 Systolic blood pressure (mm Hg)	94 (80-109.5)	91 (79-107)	95 (80-110)	<.001	
 Diastolic blood pressure (mm Hg)	64 (56-73)	65 (55-74)	64 (56-73)	.4142	
 Mean blood pressure (mm Hg)	72 (65-81)	73 (65-82)	72 (65-81)	.049	
 Pulse pressure (mm Hg)	28 (14-44)	25 (12-41)	29 (15-44)	<.001	
 Mean arterial pressure (mm Hg)	14 (11-18)	13 (10-18)	14 (11-18)	.93	
On-ECMO ABG					
 pH	7.4 (7.34-7.46)	7.4 (7.34-7.45)	7.41 (7.34-7.46)	.042	
 HCO3- (mEq/L)	23 (20-26)	23 (19.7-26)	23 (20-26)	.07	
 PaO2 (mm Hg)	138.4 (95.65-290)	152 (95.65-290)	135 (87.3-258)	<.001	
 PaCO2 (mm Hg)	37 (32-42)	37 (32-42)	37 (32-42)	.67	
 Lactate (mmol/L)	3.3 (1.8-7)	4 (2.25-7.4)	3.1 (1.8-6.8)	<.001	
 SpO2 (%)	99 (97-100)	99 (97-100)	99 (97-100)	.48	
 SaO2 (%)	98 (96-99)	98 (97-99)	98 (96-99)	.08	
ΔPaCO2	−11 (−29 to 1)	−15.65 (−38.20 to −1)	−10 (−27 to 1.2)	<.001	
Pump flow rate (4 h, L/min)	3.5 (2.9-4.1)	3.6 (3.0-4.2)	3.5 (2.86-4.1)	<.001	
Pump flow rate (24 h, L/min)	3.6 (3.0-4.24)	3.8 (3.15-4.36)	3.6 (2.91-4.2)	<.001	
Cannulation strategy					
Days on ECMO support	2.625 (0.875-5.333)	3.083 (1.583-5.625)	2.458 (0.6667-5.2917)	<.001	
Neurological complications on-ECMO					
 Composite ABI					
 Composite Ischemia	799 (7%)	799 (9%)	0 (0%)	<.001	
 Hypoxic-ischemic brain injury	357 (3%)	357 (4%)	0 (0%)	<.001	
 Ischemic stroke	462 (4%)	462 (5%)	0 (0%)	<.001	
 Composite ICH	281 (3%)	281 (3%)	0 (0%)	<.001	
 Intra/extra parenchymal hemorrhage	82 (1%)	82 (1%)	0 (0%)	<.001	
 Intraventricular hemorrhage	39 (0%)	39 (1%)	0 (0%)	<.001	
 Brain death	681 (6%)	681 (8%)	0 (0%)	<.001	
 Neurosurgical intervention	13 (0%)	13 (1%)	0 (0%)	<.001	
 Seizures confirmed by EEG	175 (2%)	175 (2%)	0 (0%)	<.001	
 Seizures clinically determined	152 (1%)	152 (2%)	0 (0%)	<.001	
Other complications on-ECMO					
 ECMO circuit mechanical failure	1217 (11%)	222 (12%)	995 (11%)	.10	
 Renal replacement theory	2450 (23%)	606 (34%)	1844 (21%)	<.001	
 Hemolysis	319 (3%)	228 (13%)	91 (1%)	<.001	
 Cardiac arrhythmia	1384 (13%)	1053 (59%)	331 (4%)	<.001	
 Gastrointestinal hemorrhage	457 (4%)	348 (19%)	109 (1%)	<.001	
Outcomes					
 In-hospital mortality	7490 (70%)	1579 (88%)	5911 (66%)	<.001	
Bolded P values represent a statistically significant association (P <.05). ECPR, Extracorporeal cardiopulmonary resuscitation; ABI, acute brain injury; ECLS, Extracorporeal Life Support; COPD, chronic obstructive pulmonary disorder; ECMO, extracorporeal membrane oxygenation; ABG, arterial blood gas; HCO3, bicarbonate; PaO2, partial pressure of oxygen; PaCO2, partial pressure of carbon dioxide; SpO2, peripheral oxygen saturation; SaO2, arterial blood gas oxygen saturation; Δ, delta; ICH, intracranial hemorrhage; EEG, electroencephalogram.

Table E7 Model performance in patients on extracorporeal cardiopulmonary resuscitation for predicting acute brain injury, central nervous system ischemia, and intracranial hemorrhage

Variable	Accuracy	TPR	TNR	FPR	FNR	PPV	NPV	
ABI	69%	61%	70%	30%	39%	29%	90%	
CNS ischemia	81%	41%	85%	15%	59%	18%	95%	
ICH	88%	28%	89%	11%	72%	7%	98%	
Accuracy, True-positive + true-negative/true-positive + true-negative + false-positive + false-negative; TPR, true positive rate; TNR, true negative rate; FPR, false-positive rate; FNR, false-negative rate; PPV, positive predictive value; NPV, negative predictive value; ABI, acute brain injury; CNS, central nervous system; ICH, intracranial hemorrhage.

Table E8 Comparisons among the top 3 most important features for acute brain injury in patients receiving extracorporeal cardiopulmonary resuscitation

Variable	With ABI	Without ABI	P value	
Median ECMO duration	3.1 d	2.5 d	<.001	
Age	57.7 y	54.4 y	<.001	
Median ECMO pump flow rate at 24 h	3.8 L/min	3.6 L/min	<.001	
ABI, Acute brain injury; ECMO, extracorporeal membrane oxygenation.

Table E9 Comparisons among the top 3 most important features for central nervous system ischemia in patients receiving extracorporeal cardiopulmonary patients

Variable	With CNS ischemia	Without CNS ischemia	P value	
Median ECMO duration	3.3 d	2.5 d	<.001	
Serum bicarbonate at 24 h	23 mEq/L	23 mEq/L	.47	
Body mass index	29.1 kg/m2	27.6 kg/m2	<.001	
CNS, Central nervous system; ECMO, extracorporeal membrane oxygenation.

Table E10 Comparisons among the top 3 most important features for intracranial hemorrhage in patients receiving extracorporeal cardiopulmonary resuscitation

Variable	With ICH	Without ICH	P value	
Supported at North American ELSO center	3.3% (n = 195)	N/A	<.001	
Not supported at North American ELSO center	1.7% (n = 86)	N/A		
Median positive-end expiratory pressure at 24 h	8 mm Hg	8 mm Hg	.25	
Supported at North American ELSO center	1.2% (n = 29)	N/A	<.001	
Not supported at North American ELSO center	3% (n = 252)	N/A		
ICH, Intracranial hemorrhage; ELSO, Extracorporeal Life Support Organization; N/A, not available.

Appendix E1 Supplemental Methods

Definitions

On-ECMO pulse pressure was computed as “systolic blood pressure at 24 hours” – “diastolic blood pressure at 24 hours.” Pre-ECMO and on-ECMO ventilator settings included conventional ventilation, high-frequency oscillatory ventilation, other high-frequency ventilation (eg, high-frequency jet ventilation or percussive ventilation), other ventilation (not specified), and absence of ventilation. Pre-ECMO additional temporary mechanical circulatory support was defined as the intra-aortic balloon pump, Impella, and left and right ventricular assist devices. Pre-ECMO cardiac arrest was defined as an episode that necessitated the use of cardiopulmonary resuscitation and performance of external cardiac massage within 24 hours of ECMO support. Central cannulation was outlined as the placement of cannula in the aorta. Peripheral cannulation was outlined as the placement of cannula in the peripheral vessels. Bridge to transplant was defined as a patient being placed on ECMO for “bridging” the patient to heart or lung transplant. Trauma was defined as a patient undergoing ECMO because of traumatic injury. Chapter name included the location of the ELSO center: Asia-Pacific, Europe, Latin America, North America, and South and West Asia. ECMO duration was defined as the number of hours patients received ECMO once cannulated.

Machine Learning Algorithm and Pipeline

With the fine-tuned hyperparameters, all of the 4 selected models were fitted onto the training dataset and evaluated on the test set with the best performing model being selected for further optimization. Given the low prevalence of ABI in our dataset, random oversampling of patients with ABI in the training set was performed at different frequencies; for each oversampling frequency, the model was evaluated with a 10-CV approach. Upon identification of the optimal oversampling rate, we applied our best performing model to the entirety of the cohort with a LOOCV approach. The LOOCV works by including all observations in the training set except 1 singular observation to be used in the test set. The LOOCV stepwise approach was repeated for the entire dataset. Each observation was used as the test set at 1 point, producing a total of “N” models that were trained and then tested on the holdout “N” observations. These observations were then combined to form 1 singular test set of size “N” observations. This LOOCV approach mitigates the risk of bias by testing the ML algorithm on the entire cohort and ensuring reproducibility of these results. Our tree-based ML models have built-in mechanisms to account for binary features and nonbinary features in our training set and modeling. At nodes at a branch point, for continuous variables, it is arbitrarily discretized into less than versus greater than at a particular number and it does this until each bin/leaf is optimized.

Subsequently, we calculated the AUC-ROC, area under the precision recall curve, and a Brier score on these observations to assess the predictive performance of our models. After choosing a threshold that maximizes the F1 score, further model metrics including accuracy, true-positive, true-negative, false-positive, false-negative rates, PPV, NPV, precision, and recall were calculated. The accuracy represents how often the ML model correctly predicted the outcome of interest (number of correct predictions/total number of predictions); clinically, this represents the quality of the model in predicting ABI. Precision calculates how often the model correctly predicts the positive class (true-positives/true-positive + false-positives); clinically, this metric tells us how often ABIs that are captured by the model are truly ABIs (this is important because a false-positive measurement of ABI may be unnecessarily treated and lead to increased resource use for the hospital and patient). Recall determines how often the model correctly identifies all true-positives that are indeed actual positives (true-positives/true-positives + false-negatives); this metric is important clinically when it is important to not miss any positive outcome as an undetected ABI can be devastating and lead to mortality. The F1 score represents the harmonic mean of both the precision and recall of the model (2∗precision∗recall/precision + recall). A higher F1 score represents a well-balanced performance by the model and can thus achieve both high precision and high recall, accurately identifying true ABIs and not under detecting any ABIs. The true-positive rate represents the proportion of positive instances that were correctly predicted by the ML model (true-positives/true-positives + false-negatives) and has similar clinical implications as recall. The false-positive rate represents the proportion of negative instances that are incorrectly classified by the ML model (false-positives/false-positives + true-negatives) and the similar clinical implications as precision. The true-negative rate represents the specificity of the model, determining the probability that a true-negative sample will actually test negative (true-negatives/true-negatives + false-positives). Clinically, this is important in “ruling in” ABIs, with similar implications to precision and the false-positive rate. The false-negative rate (“miss rate”) is the probability that a true-positive sample will indeed be missed by the model (false-negatives/false-negatives + true-positives). This has similar clinical implications as recall and the true-positive rate. The PPV is the probability that if a sample is recognized as a positive result, then the sample truly has the disease (true-positives/true-positives + false-positives), whereas the NPV is the probability that if a sample is recognized as a negative result, then the sample truly does not have the disease (true-negatives/true-negatives + false-negatives).

Feature Importance Scores in Machine Learning

The Feature Importance Scores show the relative contribution of each feature ranked from highest (top bar) to lowest (bottom bar). In the SHAP plot, red values denoted features of high importance versus blue values denoted features of low importance. Each dot represents the feature attribution value of each patient and is plotted as a SHAP value on the x-axis. SHAP values quantify the predictive impact of each feature. SHAP values greater than zero represent a greater likelihood of having ABI.

Preprint is on ResearchSquare (https://doi.org/10.21203/rs.3.rs-3848514/v1).
==== Refs
References

1 Thiagarajan R.R. Barbaro R.P. Rycus P.T. Extracorporeal life support organization registry international report 2016 ASAIO J 63 1 2017 60 67 27984321
2 Cho S.M. Canner J. Chiarini G. Modifiable risk factors and mortality from ischemic and hemorrhagic strokes in patients receiving venoarterial extracorporeal membrane oxygenation: results from the extracorporeal life support organization registry Crit Care Med 48 10 2020 e897 e905 32931195
3 Ong C.S. Etchill E. Dong J. Neuromonitoring detects brain injury in patients receiving extracorporeal membrane oxygenation support J Thorac Cardiovasc Surg 165 2023 2104 2110.e1 34865837
4 Shou B.L. Ong C.S. Premraj L. Arterial oxygen and carbon dioxide tension and acute brain injury in extracorporeal cardiopulmonary resuscitation patients: analysis of the extracorporeal life support organization registry J Heart Lung Transplant 42 4 2023 503 511 36435686
5 Shou B.L. Wilcox C. Florissi I. Early low pulse pressure in VA-ECMO is associated with acute brain injury Neurocrit Care 38 2023 612 621 36167950
6 Kalra A. Kang J.K. Wilcox C. Impact of pulse pressure on acute brain injury in venoarterial ECMO patients with cardiogenic shock during the first 24 hours of ECMO cannulation: analysis of the extracorporeal life support organization registry Res Sq 2023
7 Shou B.L. Ong C.S. Zhou A.L. Arterial carbon dioxide and acute brain injury in venoarterial extracorporeal membrane oxygenation ASAIO J 68 12 2022 1501 1507 35671442
8 Schmidt M. Zogheib E. Rozé H. The preserve mortality risk score and analysis of long-term outcomes after extracorporeal membrane oxygenation for severe acute respiratory distress syndrome Intensive Care Med 39 10 2013 1704 1713 23907497
9 Schmidt M. Burrell A. Roberts L. Predicting survival after ECMO for refractory cardiogenic shock: the survival after veno-arterial-ECMO (SAVE)-score Eur Heart J 36 33 2015 2246 2256 26033984
10 Akin S. Caliskan K. Soliman O. A novel mortality risk score predicting intensive care mortality in cardiogenic shock patients treated with veno-arterial extracorporeal membrane oxygenation J Crit Care 55 2020 35 41 31689611
11 Becher P.M. Twerenbold R. Schrage B. Risk prediction of in-hospital mortality in patients with venoarterial extracorporeal membrane oxygenation for cardiopulmonary support: the ECMO-ACCEPTS score J Crit Care 56 2020 100 105 31896442
12 Yoon J.H. Pinsky M.R. Clermont G. Artificial intelligence in critical care medicine Crit Care 26 1 2022 75 35337366
13 Mamdani M. Slutsky A.S. Artificial intelligence in intensive care medicine Intensive Care Med 47 2 2021 147 149 32767073
14 Ayers B. Wood K. Gosev I. Prasad S. Predicting survival after extracorporeal membrane oxygenation by using machine learning Ann Thorac Surg 110 4 2020 1193 1200 32454016
15 Stephens A.F. Šeman M. Diehl A. ECMO PAL: using deep neural networks for survival prediction in venoarterial extracorporeal membrane oxygenation Intensive Care Med 49 9 2023 1090 1099 37548758
16 Abbasi A. Karasu Y. Li C. Sodha N.R. Eickhoff C. Ventetuolo C.E. Machine learning to predict hemorrhage and thrombosis during extracorporeal membrane oxygenation Crit Care 24 1 2020 689 33302954
17 Kalra A. Bachina P. Shou B.L. Utilizing machine learning to predict neurological injury in venovenous extracorporeal membrane oxygenation patients: an ELSO registry analysis Res Sq [Preprint] 2023 Dec 22 rs.3.rs-3779429 10.21203/rs.3.rs-3779429/v1
18 Smith M. Vukomanovic A. Brodie D. Thiagarajan R. Rycus P. Buscher H. Duration of veno-arterial extracorporeal life support (VA ECMO) and outcome: an analysis of the Extracorporeal Life Support Organization (ELSO) registry Crit Care 21 1 2017 45 28264702
19 Jaber B. Bembea M.M. Loftis L.L. Venovenous versus venoarterial extracorporeal membranous oxygenation in inotrope dependent pediatric patients with respiratory failure ASAIO J 67 4 2021 457 462 33770001
20 Kalra A. Shou B.L. Zhao D. Racial and ethnical discrepancy in hypoxemia detection in patients on extracorporeal membrane oxygenation JTCVS Open 14 2023 P145 P170
21 Kalra A. Kang J.K. Khanduja S. Long-term neuropsychiatric, neurocognitive, and functional outcomes of patients receiving ECMO: a systematic review and meta-analysis Neurology 102 3 2024 e208081
22 Carlson J.M. Etchill E. Whitman G. Early withdrawal of life sustaining therapy in extracorporeal cardiopulmonary resuscitation (ECPR): results from the extracorporeal life support organization registry Resuscitation 179 2022 71 77 35934132
23 Carlson J.M. Etchill E.W. Enriquez C.A.G. Population characteristics and markers for withdrawal of life-sustaining therapy in patients on extracorporeal membrane oxygenation J Cardiothorac Vasc Anesth 36 3 2022 833 839 34088552
24 Kalra A. Shou B.L. Zhao D. ECMO physiological factors influence pulse oximetry and arterial oxygen saturation discrepancies Ann Thorac Surg 117 2024 P1221 P1228
25 Kalra A. Wilcox C. Holmes S.D. Characterizing the racial discrepancy in hypoxemia detection in venovenous extracorporeal membrane oxygenation: an extracorporeal life support organization registry analysis Lung 2024 https://link.springer.com/article/10.1007/s00408-024-00711-4
26 Makdisi G. Wang I.W. Extra Corporeal Membrane Oxygenation (ECMO) review of a lifesaving technology J Thorac Dis 7 7 2015 E166 E176 26380745
27 Pavlushkov E. Berman M. Valchanov K. Cannulation techniques for extracorporeal life support Ann Transl Med 5 4 2017 70 28275615
28 Wilcox C. Choi C.W. Cho S.-M. Brain injury in extracorporeal cardiopulmonary resuscitation: translational to clinical research J Neurocrit Care 14 2 2021 63 77
29 Hafner S. Beloncle F. Koch A. Radermacher P. Asfar P. Hyperoxia in intensive care, emergency, and peri-operative medicine: Dr. Jekyll or Mr. Hyde? A 2015 update Ann Intensive Care 5 1 2015 42 26585328
30 Booke H. Zacharowski K. Adam E.H. Raimann F.J. Bauer F. Flinspach A.N. Cardiopulmonary resuscitation in veno-venous-ECMO patients-a retrospective study on incidence, causes and outcome PLoS One 18 8 2023 e0290083
31 Altman D.G. Royston P. The cost of dichotomising continuous variables BMJ 332 7549 2006 1080 16675816
32 Richards E.M. Fiskum G. Rosenthal R.E. Hopkins I. McKenna M.C. Hyperoxic reperfusion after global ischemia decreases hippocampal energy metabolism Stroke 38 5 2007 1578 1584 17413048
33 Al-Kawaz M.N. Canner J. Caturegli G. Duration of hyperoxia and neurologic outcomes in patients undergoing extracorporeal membrane oxygenation Crit Care Med 49 10 2021 e968 e977 33935164
34 Biancari F. Kaserer A. Perrotti A. Central versus peripheral postcardiotomy veno-arterial extracorporeal membrane oxygenation: systematic review and individual patient data meta-analysis J Clin Med 11 24 2022 7406 36556021
35 Raffa G.M. Kowalewski M. Brodie D. Meta-analysis of peripheral or central extracorporeal membrane oxygenation in postcardiotomy and non-postcardiotomy shock Ann Thorac Surg 107 1 2019 311 321 29959943
36 Mariscalco G. Salsano A. Fiore A. Peripheral versus central extracorporeal membrane oxygenation for postcardiotomy shock: multicenter registry, systematic review, and meta-analysis J Thorac Cardiovasc Surg 160 5 2020 1207 1216.e44 31864699
37 Fernando S.M. MacLaren G. Barbaro R.P. Age and associated outcomes among patients receiving venoarterial extracorporeal membrane oxygenation-analysis of the extracorporeal life support organization registry Intensive Care Med 49 2023 1456 1466 37792052
38 Hu L. Li L. Using tree-based machine learning for health studies: literature review and case series Int J Environ Res Public Health 19 23 2022 16080 36498153
39 2011 4 Key advantages of using decision trees for predictive analytics http://www.simafore.com/blog/bid/62333/4-key-advantages-of-using-decision-trees-for-predictive-analytics 2024
40 Breiman L. Random forests Mach Learn 45 2001 5 32
41 James G. Witten D. Hastie T. Tibshirani R. An Introduction to Statistical Learning 2013 Springer
42 Cortes C. Vapnik V. Support-vector networks Mach Learn 20 1995 273 297
43 Uddin S. Lu H. Confirming the statistically significant superiority of tree-based machine learning algorithms over their counterparts for tabular data PLoS One 19 4 2024 e0301541
44 Habehh H. Gohel S. Machine learning in healthcare Curr Genomics 22 4 2021 291 300 35273459
45 Shah N. Farhat A. Tweed J. Neural networks to predict radiographic brain injury in pediatric patients treated with extracorporeal membrane oxygenation J Clin Med 9 9 2020 2718 32842683
