
==== Front
PLoS One
PLoS One
plos
PLOS ONE
1932-6203
Public Library of Science San Francisco, CA USA

10.1371/journal.pone.0309748
PONE-D-24-20146
Research Article
Medicine and Health Sciences
Health Care
Health Care Facilities
Hospitals
Intensive Care Units
Medicine and Health Sciences
Medical Conditions
Infectious Diseases
Bacterial Diseases
Bacteremia
Computer and Information Sciences
Artificial Intelligence
Machine Learning
Research and Analysis Methods
Mathematical and Statistical Techniques
Statistical Methods
Forecasting
Physical Sciences
Mathematics
Statistics
Statistical Methods
Forecasting
Medicine and Health Sciences
Epidemiology
Medical Risk Factors
Cancer Risk Factors
Medicine and Health Sciences
Oncology
Cancer Risk Factors
Computer and Information Sciences
Artificial Intelligence
Machine Learning
Support Vector Machines
Physical Sciences
Mathematics
Applied Mathematics
Algorithms
Machine Learning Algorithms
Research and Analysis Methods
Simulation and Modeling
Algorithms
Machine Learning Algorithms
Computer and Information Sciences
Artificial Intelligence
Machine Learning
Machine Learning Algorithms
Biology and Life Sciences
Biochemistry
Proteins
C-Reactive Proteins
A machine learning model for early candidemia prediction in the intensive care unit: Clinical application
Candidemia prediction model in the intensive care unit
https://orcid.org/0000-0002-0132-0001
Meng Qiang Funding acquisition Visualization Writing – original draft 1
Chen Bowang Conceptualization 1
Xu Yingyuan Validation 2
Zhang Qiang Data curation Validation 2
Ding Ranran Software 1
Ma Zhen Software 1
Jin Zhi Data curation Writing – original draft 1
Gao Shuhong Supervision 1 *
https://orcid.org/0009-0008-6043-7665
Qu Feng Supervision 1 *
1 Jining No. 1 People’s Hospital Affiliated to Shandong First Medical University, Jining, Shandong, China
2 Pulmonary and Critical Care Medicine, Tengzhou Central People’s Hospital, Tengzhou City, Shandong Province, People’s Republic of China
Giordano Giovanni Editor
Sapienza University of Rome: Universita degli Studi di Roma La Sapienza, ITALY
Competing Interests: The authors have declared that no competing interests exist.

* E-mail: rmyyzzeq@163.com (FQ); mengqiang1985@126.com(SG)
9 9 2024
2024
19 9 e030974830 5 2024
17 8 2024
© 2024 Meng et al
2024
Meng et al
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Candidemia often poses a diagnostic challenge due to the lack of specific clinical features, and delayed antifungal therapy can significantly increase mortality rates, particularly in the intensive care unit (ICU). This study aims to develop a machine learning predictive model for early candidemia diagnosis in ICU patients, leveraging their clinical information and findings. We conducted this study with a cohort of 334 patients admitted to the ICU unit at Ji Ning NO.1 people’s hospital in China from Jan. 2015 to Dec. 2022. To ensure the model’s reliability, we validated this model with an external group consisting of 77 patients from other sources. The candidemia to bacteremia ratio is 1:1. We collected relevant clinical procedures and eighteen key examinations or tests features to support the recursive feature elimination (RFE) algorithm. These features included total bilirubin, age, platelet count, hemoglobin, CVC, lymphocyte, Duration of stay in ICU and so on. To construct the candidemia diagnosis model, we employed random forest (RF) algorithm alongside other machine learning methods and conducted internal and external validation with training and testing sets allocated in a 7:3 ratio. The RF model demonstrated the highest area under the receiver operating characteristic (AUC) with values of 0.87 and 0.83 for internal and external validation, respectively. To evaluate the importance of features in predicting candidemia, Shapley additive explanation (SHAP) values were calculated and results revealed that total bilirubin and age were the most important factors in the prediction model. This advancement in candidemia prediction holds significant promise for early intervention and improved patient outcomes in the ICU setting, where timely diagnosis is of paramount crucial.

Key R&D Program of Jining 2023YXNS151 https://orcid.org/0000-0002-0132-0001
Meng Qiang Key R&D Program of Jining, grant number 2023YXNS151. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data AvailabilityAll relevant data are within the manuscript and its Supporting Information files.
Data Availability

All relevant data are within the manuscript and its Supporting Information files.
==== Body
pmcIntroduction

Fever is a common symptom among patients admitted to the intensive care unit (ICU), often attributed to infections caused by variable microorganisms. Candida spp. is the leading cause of invasive fungal infections, ranking as the fourth most common microorganisms responsible for bloodstream infections (BSIs) [1]. Both invasive fungal infections and candidemia predominantly affect critically ill patients, with up to one-third episodes of candidemia cases occurring within the ICU [2]. Candidemia usually causes life-threatening conditions, resulting in increased mortality rate, healthcare costs, and prolonged ICU stays [3,4]. However, distinguishing candidemia from bacteremia is challenging due to the lack of specific clinical symptoms and presentations [5,6]. The current gold standard for candidemia diagnosis is a positive blood culture test [7]. However, blood culture tests have low sensitivity and require extended incubation times, potentially delaying antifungal therapy [8]. Therefore, physicians usually need to decide whether to wait for blood culture result which may delay initiation of antifungal therapy and prescribe empirical antifungal drugs when they suspect candidemia infection based on their experience. Retrospective studies have showed that such delays can substantially enhance mortality rates, with percentages rising from 35% to 80% [9,10].Conversely, initiating empirical antifungal treatment too early may induce antifungal resistance which has become a significant global public health threat.

New diagnostic biomarkers such as (1, 3)-β-D-glucan (BGD) and galactomannan (GM), glycan antigens derived from the fungal cell wall, or antibody recognizing mannan, have been introduced in clinical practice [6]. However, these biomarker tests are often costly and require complex infrastructure, limiting hospitals’ ability to conduct such around-the-clock tests [11]. So, they may not provide early diagnosis of candidemia in ICU patients, leaving clinicians without timely information when suspecting a candidemia infection [12]. Furthermore, biomarkers could not accurately predict candidemia in many conditions. BGD yields a high percentage of false-positive results due to transfusion of human blood products, hemodialysis, infection by some Gram-positive bacteria, use of certain beta-lactam antibiotics, cellulose dressings or enteral nutrition, and disruptions of gastrointestinal (GI) tract integrity [13,14].Therefore, the demand for the development of more rapid and accurate approaches to differentiate candidemia from bacterial infections is not only evident but also urgent, such approaches could aid ICU physicians in making informed decision regarding the initiation of antifungal therapy [15,16].

Recently, clinical decision support based on machine learning (ML) has received increasing attention for early prediction of candidemia. ML techniques possess a unique capacity to process vast quantities of data, enabling them to employ advanced mathematical models to grasp intricate patterns within clinical datasets. This capability equips ML to excel in tasks such as pattern detection and differentiation [17]. Previous studies have unveiled the potential of ML in early candidemia diagnosis. Bhavani SV et al. [16] trained a gradient boosting machine (GBM) model for early detection of candidemia in high and low risk groups. Their model outperformed the traditional logistic regression model, demonstrating the effectiveness of ML approaches with a higher area under the receiver operating characteristic (AUC). Similarly, Yoo J et al. [18] developed the CanDETEC model, utilizing random forest (RF) algorithms for candidemia risk prediction in patients with malignancy, and validates its efficacy through a single-center retrospective stud applied random forest (RF) models for the early prediction of candidemia in cancer patients. Atamna A et al. [19] and Gao Y et al. [20] constructed a logistic regression (LR) model to pinpoint risk factors of candidemia in internal medicine wards, offering valuable insights for the clinical management of patients with candidemia and bacteremia. Besides, Hu WH et al. [21] developed a two-level stacked generalisation model using multiple machines learning algorithms, including Naïve Bayesian (NB), k-Nearest Neighbour (KNN), logistic regression (LR), and random forest (RF) as base classifiers, with a support vector machine (SVM) as the meta-classifier, to predict 14-day mortality in candidemia patients. However, the aforementioned studies primarily focused on general populations of patients outside the ICU setting. Risk factors for Candida bloodstream infections in adult patients have been extensively described and identified, often observed in hospitalized patients. Nevertheless, critically ill ICU patients often present with distinct risk factors that are less applicable to distinguishing from bacteremia patients. Furthermore, the majority of studies have relied on traditional linear models to construct their predictive models, which offer only moderate predictive accuracy even in the best performing models. The demand for a more accurate and rapid approach to differentiate candidemia from bacteremia in ICU patients has grown, which will guide timely and effective therapeutic interventions in clinical settings. This study focuses on developing an early prediction model for fungal bloodstream infections in ICU patients, employing various machine learning algorithms. Through extensive retrospective analysis and external validation, the study validates the effectiveness and generalizability of the models. Additionally, the research elucidates the contribution of individual features in the model using SHAP values and constructs a predictive nomogram for diagnosing candidemia, providing clinicians with a valuable decision-making tool.

Materials and methods

Human ethics

This study adhered to the guidelines set forth in the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) [22] for the construction of the multivariable prediction model and its subsequent validation. The study protocol received approval from the Human Ethics Review Committee of Ji Ning NO.1 people’s Hospital, assigned the reference number KYLL-202308-134. The ethics review board at Ji Ning No.1 people’s Hospital granted an exemption from the acquisition of informed consent, given the retrospective nature of the study. All patients’ information was handled confidentially during the data collection and manuscript preparation.

Patient inclusion and exclusion criteria

Clinical data for model development and internal validation were retrospectively collected from the electronic medical record system at Ji Ning No. 1 People’s Hospital, spanning from Jan. 2015 to Dec. 2022. The clinical data for this study were accessed from September 2023 to October 2023. Patients in the candidemia group were included if they met the following criteria: (1) aged ≥ 18 years; (2) requirement for intensive care; (3) expected duration of stay in ICU >48 hours; and (4) at least one blood peripheral positive for Candida spp. Candidemia group excluded criteria: (1) patients who co-infected with bacteremia within 7 days before and after the detection of candidemia; (2) patients with fungal infections other than those caused by Candida species, or (iv) patients who died within 48 hours. (3) patients who were pregnant or lactating. To ensure a comparable periods of risk exposure for both groups, each control patient had a hospitalization duration similar to the time at risk of cases, defined as the number of days from hospital admission to the occurrence of candidemia. To ensure comparable risk exposure periods between candidemia cases and control patients, we implemented a meticulous matching process. Candidemia cases were retrospectively identified using the microbiology laboratory database, and only the first episode was considered if a patient experienced multiple episodes. Control patients diagnosed with bacteremia were selected in a 1:1 ratio and matched to candidemia cases based on several criteria, including age (±5 years), sex, date of hospital admission, and duration of hospitalization at the time of the first positive blood culture. The hospitalization duration for candidemia cases was defined as the number of days from hospital admission to the occurrence of candidemia. Similarly, for control patients, hospitalization duration was defined as the time from admission to the onset of bacteremia, ensuring alignment with the candidemia cases’ risk period. This matching strategy aims to create a balanced and representative dataset for the analysis [23].

For external validation, clinical data were obtained from Teng Zhou Central People’s Hospital, covering the period from Jan. 2020 to Jun. 2023. The external validation data for this study were accessed from January 2024 to February 2024.

Data collection

We determined the potential risk factors for Candida infection through reviewing the past literature and discussions among the experts in the research team. BDG and GM tests, as new biomarkers, have been rarely used in the past, so we did not include those as risk factors in our study. Data collected at baseline encompassed various characteristics, including the duration of stay in ICU, antibiotic usage within the previous two weeks, presence of solid cancer, diabetes, history of chemotherapy, recent surgical procedures, use of central venous catheter (CVC), receipt of total parenteral nutrition (TPN), immunosuppressant treatments and other clinical laboratory data. Prior to analysis, we applied data processing techniques. Cases with missing data excessing 50% of features were excluded from the dataset. To ensure the quality and completeness of the dataset for subsequent analysis, missing values in the remaining cases were imputed using the mean value of the training set and testing set, respectively.

Microbiological test

Blood culture was processed using the automated blood culture system (Bio-Merieux SA, Marcy l’etoile, France). Fungal isolates and bacterial isolates were cultured at 35°C for 48–72 hours. Simultaneously, we conducted gram staining and microscopic examination. Strain identification was carried out using the VITEK2 Compact system (Bio-Merieux SA, Marcy l’etoile, France) for both fungal and bacterial isolates.

Descriptive analysis

Continuous data were presented as mean± standard deviation (SD) or median (interquartile range, IQR), while categorical data were reported as frequencies and percentages. Univariate analyses were conducted, with continues data analyzed using independent t-test or non-parametric tests, and categorical data analyzed using the Chi-square or Fisher’s exact test. The descriptive analyses were performed using IBM SPSS Statistics for Windows, Version 25.0. A significance level of p <0.05 was considered statistically significant.

Feature selection techniques

Feature selection is a crucial process aimed at identifying the most relevant subset of variables within a dataset. This practice serves to reduce the number of features, enhance the model’s generalization ability, and mitigate the risk of overfitting [24]. While machine learning algorithms in our study can automatically rank features of importance, they may occasionally introduce a small number of redundant features due to the inherent randomness in feature selection [25]. To address this problem, we employed recursive feature elimination (RFE) on the training set for feature selection. RFE involves the iterative removing of the least important features, allowing us to identify the optimal feature combination that can improve overall generalization performance [26].

Model development and external validation

We used a range of machine learning algorithms, including extreme gradient boosting (XGB), logistic regression (LR), support vector machines (SVM), recurrent neural network (RNN) and random forest (RF), to establish candidemia prediction models with the selected variables. Extreme gradient boosting (XGBoost) operates by sequentially constructing an ensemble of decision trees, where each tree corrects the errors of its predecessors. This method uses gradient descent to minimize the loss function and enhance model performance [27]. Logistic Regression (LR) is a form of linear regression aimed at establishing the relationship between a binary response variable (indicating the occurrence of an event) and a set of explanatory variables. It achieves this by using a logistic function to model the probability of the binary outcome [28]. Support Vector Machine (SVM) works by constructing an optimal hyperplane that maximizes the margin between different classes. For nonlinear problems, it employs kernel functions to project data into higher-dimensional spaces for effective classification [29]. Recurrent Neural Networks (RNN) simulate the interconnected structure of neurons. Input data passes through multiple layers of nodes (neurons), which apply weighted sums and activation functions to learn complex patterns and make predictions [30]. Random Forest (RF) is an ensemble method that trains multiple regression trees on different random subsets of the data. At each node, a randomly selected subset of predictors is used, which enhances robustness to predictor correlations [31].

The entire dataset was randomly divided into ten groups. In each iteration, seven groups of patients were assigned to the training cohort, while the remaining three groups were assigned to the testing cohort. These models (XGB, LR, SVM, RNN and RF) underwent ten-fold cross-validation on the training set to determine the optimal hyperparameters. Using the optimal hyperparameters for each model, we trained the models on the training set. Subsequently, we applied these optimized machine learning models to the testing cohort and external cohort for internal and external evaluation.

External validation was performed using an independent set of electronic records of healthy individuals obtained from Teng Zhou Central People’s Hospital spanning from 2020 to 2023. Through machine learning algorithm of RFE, we have obtained features for constructing an early stage candidemia predict model. Then, we collected data of the selected features from Teng Zhou Central People’s Hospital for external validation of candidemia predict model.

Evaluating performance and interpretation of models

We use receiver operating characteristic curve (ROC) to predict power of all the models for candidemia diagnosis. The area under the ROC curve (AUCROC) served as a metric to assess their capacity to distinguish candidemia from bacteremia. The best predict model was defined as the one with the highest AUC value. Furthermore, we assessed performance using metrics such as accuracy, precision, recall, and F1-score to comprehensively evaluate the predictive model’s effectiveness [30]. Score and accuracy were calculated using a threshold of 0.5. Model calibration was visualized through the calibration curve [32]. These curves plot the mean decile prediction probability for patients in each group. Calibration results were evaluated by examining the proximity of the calibration curve to the identity line (y = x), representing the standard calibration. In addition, we conducted a decision curve analysis to evaluate the potential benefits of clinical decision- making based on RF model predictions.

Given that machine learning model are often considered as black-boxes, understanding the impact of each risk variable on predict model can be challenging. To address this problem, we used Shapley additive explanation (SHAP) value [33] to explain the contribution of each feature to specific prediction made by the optimal model. SHAP value was conducted using the “create_explainer” and “shap” function in “shapper” R package (version 4.3.2). Finally, we used nomograms to estimate probability of individual candidemia infection using the “nomogram “function in the “rms” R package (version 4.3.2)[34]. The nomogram analysis considered multiple variables while offers a more accurate prediction and enhances decision-making for candidemia prediction.

Results

Significant differences in patient parameters

In this study, we included a total of 334 patients, comprising 167 patients with candidemia and 167 with bacteremia, to construct a risk prediction model for early candidemia diagnosis in ICU setting. The patient cohort was divided into a training set that consisted of 70% of the patients (234 patients), and a testing set that comprised the remaining 30% (100 patients). Additionally, data from 77 patients, with 38 diagnosed with candidemia and 39 with bacteremia, were recruited from Teng Zhou Central People’s Hospital to serve as an external validation set. The procedure for patient’s selection is detailed in Fig 1.

10.1371/journal.pone.0309748.g001 Fig 1 Flowchart of candidemia and bacteremia patients selected in the ICU for model development and external validation.

(A)Patients select from Jining NO. 1 people’s hospital for model development which including the training (n = 234) and internal validation sets (n = 100). (B) Patients select from Teng Zhou People’s Hospital for external validation set (n = 77).

Comparisons of the baseline features between the candidemia and bacteremia group were provided in Table 1. As shown in Table 1, age is significantly difference between candidemia and bacteremia patients. There is no statistically significant difference in comorbidities between candidemia and bacteremia patients. The key characteristics of the patients in the training and testing dataset are presented in Table 2. As shown in Table 2, several features, including age, presence of central venous catheter (CVC), platelet count, use of total parenteral nutrition (TPN), and duration of stay in the ICU, exhibited significant differences between patients with candidemia and those with bacteremia patients, both in training and test sets. However, there was no difference in gender, presence of fever, alanine transaminase (ALT), aspartate transaminase (AST), albumin, glutamyl transpeptidase (GGT), total bilirubin, creatinine, diabetes, history of solid cancer, chemotherapy, use of immunosuppressive drugs, antibiotic therapy, prior abdominal surgery, white blood count (WBC), neutrophils count, lymphocyte count, C-reactive protein (CRP), procalcitonin (PCT) levels, and hemoglobin level between candidemia and bacteremia patients. A detailed breakdown of the selected features for candidemia and bacteremia in external validation can be found in S1 Table. The comparison of selected features between internal and external validation can be found in S2 Table, WBC and CRP exhibited significant difference.

10.1371/journal.pone.0309748.t001 Table 1 Baseline features between the candidemia and bacteremia groups.

	Candidemia
(n = 167)	Bacteremia
(n = 167)	P value	
Gender (male, n (%))	112(67.6%)	106(63.7%)	0.367	
Age (y, mean (SD))	70.6(17.49)	64.5(16.85)	0.036	
CHD disease (n (%))	48(28.7%)	46(27.5%)	0.852	
High blood pressure disease (n (%))	68(40.7%)	74(44.3%)	0.623	
Diabetes (n (%))	58(34.7%)	48(28.7%)	0.239	
Chronic pulmonary disease (n (%))	48(28.7%)	46(27.5%)	0.851	
Liver disease (n (%))	8(4.79%)	12(7.14%)	0.363	
Renal disease (n (%))	15(45.5%)	18(54.5%)	0.582	
Cerebrovascular disease (n (%))	49(29.3%)	34(20.3%)	0.118	
This table presents the baseline characteristics of patients diagnosed with candidemia (n = 167) and bacteremia (n = 167) in our study. The dataset was randomly split into a training set (70%, n = 234) and a test set (30%, n = 100) to develop and validate the diagnostic models. Statistical comparisons between the two groups were performed using t-tests for continuous variables and chi-square tests for categorical variables. The analysis results showed a significant difference in age between the two groups (p < 0.05), while other baseline disease parameters showed no statistical significance. Therefore, these baseline disease parameters were not considered in the model variable selection. Definition of abbreviations: y: Years; CHD: Coronary atherosclerotic heart disease.

10.1371/journal.pone.0309748.t002 Table 2 Comparison of patient parameters between candidemia and bacteremia groups in the training and the testing sets.

	Infection patients(n = 334)	
Training set(n = 234)	P value	Testing set (n = 100)	P value	
Candidemia (n = 113)	Bacteremia
(n = 121)	Candidemia
(n = 54)	Bacteremia
(n = 46)	
Gender (male, n (%))	74(65.4%)	78(64.5%)	0.570	31(57.4%)	29(63.0%)	0.551	
Age (y,mean (SD))	70.2(16.72)	64.2(16.31)	0.027	71.7(12.81)	65.8(15.56)	0.019	
Fever (°C, mean (SD))	38.2(0.68)	38.6(0.54)	0.795	38.2(0.68)	38.6(0.56)	0.124	
ALT (U/L, median (IQR))	25.9(9.5,58.2)	23.9(15.1,31.5)	0.779	28.5(12.58,91.78)	30.95(13.73,60.38)	0.828	
AST (U/L, median (IQR))	28.0(17.5, 69.9)	47.5(18.6, 97.0)	0.221	43.6(27.3, 73.7)	33.2(21.1, 68.8)	0.258	
Albumin (g/l, mean (SD))	28.5(5.60)	29.4(5.11)	0.339	28.1(6.55)	31.0(5.92)	0.068	
GGT (U/L, median (IQR))	55.0(30.1,107.6)	49.0(25.5,135.5)	0.979	58.0(37.50,111.5)	42.95(22.0, 115.5)	0.079	
Total bilirubin(μmol/L, median (IQR))	26.8(10.7, 41.7)	15.5(9.1, 26.9)	0.037	27.3(8.9, 45.8)	10.8(8.9, 16.1)	0.028	
Creatinine (μmol/L, median (IQR))	91.4(68.3, 72.3)	175.0(43.0,241.8)	0.046	99.0(63.0, 138.0)	172.5(52.3, 237.0)	0.035	
Diabetes (n, %)	38(33.6%)	35(28.9%)	0.481	20(37.0%)	13(28.3%)	0.398	
Solid cancer (n, %)	32(28.3%)	18(14.9%)	0.012	15(27.8%)	6(13.0%)	0.071	
Chemotherapy (n, %)	26(23.0%)	14(11.6%)	0.094	14(25.9%)	6(13.0%)	0.126	
Immunosuppressive drugs (n, %)	35(31.0%)	21(16.4%)	0.008	19(35.2%)	8(17.4%)	0.046	
Antibiotic therapy (n, %)	105(92.9%)	100(82.6%)	0.017	49(90.7%)	35(76.1%)	0.046	
CVC (n, %)	68(60.2%)	31(25.6%)	<0.001	31(57.4%)	10(21.7%)	0.001	
Abdominal surgery (n, %)	70(61.95%)	55(45.45%)	0.012	34(63.0%)	19(41.3%)	0.044	
WBC count (109/L, median (IQR))	9.9(6.5,17.3)	17.4(7.9,20.8)	0.041	11.2(7.6, 20.6)	19.1(6.5, 24.7)	0.043	
CRP (mg/l, mean (SD))	85.3(55.32)	130.4(80.61)	0.014	91.6(51.57)	138.6(66.45)	0.034	
PCT (ng/ml, median (IQR))	1.9(0.7,10.9)	3.9(0.3,17.3)	0.034	2.0(0.6, 14.0)	3.3(0.8, 10.4)	0.044	
Neutrophils count (109/L, median (IQR))	7.0(5.6, 14.9)	10.3(6.1, 14.2)	0.344	7.31(4.90, 10.72)	10.27(5.75, 17.77)	0.101	
Lymphocyte count (109/L, mean (SD))	0.8(0.62)	1.3(0.73)	0.036	0.67(0.60)	1.23(0.90)	0.048	
Platelet count (109/L, mean (SD))	139.1(109.71)	186.8(113.31)	0.011	123.6(97.97)	208.4(119.52)	0.003	
Monocyte count (109/L, median (IQR))	0.35(0.24, 0.79)	0.66(0.36, 0.88)	0.023	0.34(0.19, 0.73)	0.66(0.43, 1.12)	0.425	
Hemoglobin(g/L, mean (SD))	84.1(19.22)	100.9(21.52)	0.028	79.9(16.41)	98.9(19.53)	0.027	
TPN (n, %)	62(54.9%)	27(22.3%)	<0.001	28(51.9%)	10(21.7%)	0.004	
Duration of ICU stay (days, mean (SD))	24.0(21.6)	14.4(15.5)	0.046	27.5(37.1)	16.7(35.9)	0.030	
This table presents a comparison of key patient parameters, including demographic factors, laboratory values, and clinical characteristics between patients with candidemia and bacteremia in both the training (model development, n = 234) and testing (internal validation, n = 100) datasets. The training set was used for model development, while the testing set was used for internal validation. Statistical analyses were conducted using t-tests for continuous variables and chi-square tests for categorical variables to assess differences between the two groups. Differences with p<0.05 were considered statistically significant. This consistency across both sets suggests that the data splitting did not impact the subsequent model development and validation processes. The inclusion of these parameters in the comparison aims to provide a comprehensive overview of the demographic and clinical characteristics of the patients involved in the study. Understanding these characteristics is essential for evaluating the potential impact on the model’s performance and its clinical applicability. Definition of abbreviations: y: Years; ALT: Alanine transaminase; AST: Aspartate transaminase; GGT: Glutamyl transpeptidase; TPN: Total parenteral nutrition; CVC: Central venous catheter; WBC: White blood cell; PCT: Procalcitonin; CRP: C-reactive protein; ICU: Intensive care unit; IQR: Interquartile range; SD: Standard deviation.

Model development and selected features

To identify the most relevant features for our prediction model, the Recursive Feature Elimination (RFE) algorithm, a tree-based method, was utilized to rank the best feature combination on the training set [35]. Finally, the most significant 18 features were selected according to their importance, which include total bilirubin, age, platelet count, hemoglobin, CVC, lymphocyte, duration of stay in ICU, neutrophils, monocyte, antibiotic therapy, PCT, CRP, WBC, immunosuppressive drugs, TPN, abdominal surgery, chemotherapy and solid cancer (Fig 2). These selected features were used for the subsequent model development.

10.1371/journal.pone.0309748.g002 Fig 2 Recursive feature optimization using the training set.

The dotted line shows the highest accuracy is achieved with 18 features. The horizontal axis is the number of feature selections, and the vertical axis is the prediction accuracy.

Model performance for predicting candidemia

We employed multiple prediction models, including Random Forest (RF), Logistic Regression (LR), Support Vector Machine (SVM), extreme gradient boosting (XGB), and Recurrent Neural Network (RNN), to forecast the recurrence of candidemia with the 18 features which were selected. In Table 3, we present a comparative analysis of these models, demonstrating their performance metrics regarding accuracy, precision, recall, F1-score and Area Under the Curve (AUC). The ROC curves were constructed based on the AUC value (Fig 3). For the ML models, we conduct repeated 10-folding cross validation based on the optimization of AUC to identify the optimal model parameters. Our analysis demonstrated the effectiveness of all models, as each exhibited strong predictive performance with an AUC greater than 0.80. Notably, the RF model outperformed the others with an AUC of 0.87. The highest accuracy attained for predicting candidemia was 0.79 achieved by both the RF and RNN models.

10.1371/journal.pone.0309748.g003 Fig 3 ROC curves among machine learning models in candidemia diagnostic among ICU patients.

ROC: Receiver operating characteristic curve; auc: Area under the curve; LR: Logistic regression; SVM: Support vector machine; RF: Random forest; XGB: Extreme gradient boosting. RNN: Recurrent neural network.

10.1371/journal.pone.0309748.t003 Table 3 Performance of the machine-learning algorithms in internal validation cohort.

Model	Accuracy	Precision	Recall	Sensitivity	Specificity	F1 score	AUC	
Logistic regression	0.73	0.69	0.78	0.78	0.68	0.74	0.82	
Random forest	0.79	0.85	0.69	0.80	0.77	0.76	0.87	
Support vector machine	0.73	0.69	0.78	0.69	0.88	0.74	0.82	
Extreme Gradient boosting	0.76	0.79	0.69	0.56	0.88	0.73	0.83	
Recurrent neural network	0.79	0.78	0.78	0.78	0.79	0.78	0.84	
This table summarizes the performance metrics of various machine learning models evaluated using the internal validation cohort (n = 100). Metrics include Accuracy, Precision, Recall, F1 Score, Sensitivity, Specificity and AUC. All models were trained using the same set of variables to ensure comparability. Definition of abbreviations: AUC: Area Under the Curve.

The calibration curves for our predictive models were presented in Fig 4. The Random Forest and Logistic Regression models showed excellent calibration, with their curves closely aligning with the diagonal line. The mean absolute errors (MAE) were 0.088 and 0.094, respectively. Likewise, the Recurrent Neural Network and Support Vector Machine models exhibit strong calibration performance. However, the calibration curve for the Extreme Gradient Boosting model noticeably deviates from diagonal line, suggesting a potential inaccuracy in comparison to the other models. In addition, we also provide the decision curves that assess the clinical usefulness of the RF model on the test set in Fig 5. These curves show that the RF model offers substantial benefit when the threshold probability of candidemia falls within the range of 55% to 85%.

10.1371/journal.pone.0309748.g004 Fig 4 Calibration curve for model validation of LR, RF, SVM, XGB and RNN on the testing set.

The calibration curve plotted means predicted probabilities for each model against the observed event frequencies on the testing set (n = 100), using decile-binned data.

10.1371/journal.pone.0309748.g005 Fig 5 Decision curve analysis (DCA) of the random forest model on the test set.

X-axis indicates the threshold probability for candidemia diagnosis and Y-axis indicates the net benefit. The red line represents the random forest model.

The Random Forest model continued to perform well again in our external validation cohort from Teng Zhou Central People’s Hospital. As shown in Table 4, the RF model achieved a good predictive performance with an AUC of 0.83, slightly lower than the testing set. The RF model exhibited a precision of 0.83 and accuracy of 0.81 in the external validation cohort. These results confirm the model’s reliability in correctly discriminating true positives and true negatives.

10.1371/journal.pone.0309748.t004 Table 4 Performance of machine-learning algorithms in external validation cohort.

Model	Accuracy	Precision	Recall	Sensitivity	Specificity	F1	AUC	
Logistic regression	0.73	0.63	0.63	0.74	0.65	0.63	0.80	
Random forest	0.81	0.83	0.63	0.78	0.75	0.71	0.83	
Support vector machine	0.71	0.59	0.63	0.66	0.82	0.61	0.78	
Extreme Gradient boosting	0.80	0.68	0.81	0.59	0.82	0.74	0.83	
Recurrent neural network	0.73	0.63	0.63	0.74	0.67	0.63	0.76	
This table summarizes the performance metrics of various machine learning models evaluated using the external validation cohort (n = 77). Metrics include Accuracy, Precision, Recall, F1 Score, Sensitivity, Specificity and AUC. All models were trained using the same set of variables to ensure comparability. Definition of abbreviations: AUC: Area Under the Curve.

Interpretation of random forest predictive model

We used SHAP values to elucidate the contribution of individual features in the Random Forest model. As shown in Fig 6, all features are ranked by their importance. The most influential features in red are positioned at the top, while less significant ones in blue are at the bottom. There are a few key insights from SHAP analysis. First, total bilirubin emerges as the most crucial predictor for candidemia in the RF model, appearing that a higher level of total bilirubin is associated with an elevated risk of candidemia. Second, several other factors positively correlate with the risk of candidemia, including age, antimicrobial therapy, neutrophil count, use of immunosuppressive drugs, total parenteral nutrition, abdominal surgery, chemotherapy, the presence of a central venous catheter, a history of solid cancer, and duration of ICU stay. Third, lower levels of platelets, hemoglobin, lymphocytes, white blood cells, C-reactive protein, procalcitonin, and monocytes are negatively associated with the risk of candidemia.

10.1371/journal.pone.0309748.g006 Fig 6 Feature importance estimated using the Shapley additive explanations (SHAP) values.

The SHAP values were calculated using the training data (n = 234). The plot sorts features by the sum of SHAP value magnitude over all samples. The blue to red color represents the feature value (red high, blue low). The x-axis measures the impacts on the model output (right positive, left negative).

The predictive nomogram for candidemia probability

In this study, we constructed a predictive nomogram based on 18 selected features to estimate the probability of candidemia. The primary goal of this analysis is to provide clinicians with a valuable resource for assessing and quantifying the likelihood of candidemia based on specifical clinical characteristics. As shown in Fig 7, each feature is assigned a probability score within a range from 0 to 100. The cumulative scores for all the features are then summed to generate a total score. This total score can then be projected onto the lower risk axis of the nomogram, enabling the prediction of the risk of candidemia diagnosis in ICU patients. To further facilitate clinical use, we plan to convert the content of Fig 5 into a list of point values and include it in the S3 Table.

10.1371/journal.pone.0309748.g007 Fig 7 Nomogram of candidemia predicting using 18 clinical features.

The nomogram was developed using the training data (n = 234). To use the nomogram, we first score each patient’s variables to the score axis, added points of all the variables, draw a line from the total score axis to determine candidemia predicting probability on the lower line of the nomogram.

Discussion

In this study, we aimed to develop a predicative model for candidemia in intensive care unit patients using machine learning techniques. Our study cohort included 167 patients with candidemia infection and 167 patients with bacteremia infection admitted to the ICU at Ji Ning NO.1 people’s hospital from Jan. 2015 to Dec. 2022. Our approach distinguishes itself from prior researches that mainly focused on identifying risk factor for candidemia susceptibility, where healthy people or non-infected patients in medical ward served as the control group. Here, we not only identified risk factors but also constructed a diagnostic model, with bacteremia patients serving as the control model.

When compared our study with Li, et al.’s predictive model for candidemia in cancer patients [36], it is worth noting that while they achieved a slightly higher AUC value, variance in patient selection criteria plays a significant influence. Our study exclusively focused on ICU patients, a group with widely distributed risk factors. By using bacteremia patients within the same ICU as the control group in our model development, we aimed in order to better predicate the candidemia especially in ICU setting. Li et al. also employed bacteremia patients as the control group in their predictive model for invasive candidiasis [37], constituting 81.63% of candidemia and 64.79% of bacteremia cases from ICU, which yielded an AUC value of 0. 92. The slightly lower AUC values in this study may be attributed to the differences in the selected patient populations. An important distinction in our study is the use of data from an external hospital for validation, a practice not commonly observed in previous candidemia prediction study. This external validation provides additional credibility to the generalizability of our model.

We employed a range of machine learning algorithms, including LR, RF, SVM, XGB, and RNN, to construct diagnostic models using routine clinical data. These models displayed robust performance in distinguishing candidemia from bacteremia among ICU patients. Among these models, the RF model stood out as the most promising candidate. It achieved impressive precision, recall, and F1 scores of 0.85, 0.69, and 0.76, respectively. These metrics were either equal to or greater than the corresponding scores obtained by the RNN model (0.78). When subjected to external validation, the RF model maintained its excellence, yielding an accuracy of 0.81, precision of 0.83, recall of 0.63, and F1 score of 0.71 reflective of its exceptional diagnostic performance. The decision to select the Random Forest (RF) model as the final machine learning model was driven by its remarkable diagnostic performance, highlighted by a high AUC value of 0.87 and these outstanding evaluation metrics. In addition, RF algorithm displays an advantage in handling missing data points, whether they are numerical or factor variables. This capability is particularly valuable in the context of clinical studies, where missing data can be a common occurrence.

The recurrent neural network (RNN) model also showed strong diagnostic performance, with an AUC of 0.84 and the highest accuracy (0.79) among the models tested. RNNs are recognized for their ability to model sequential data and complex temporal patterns, suggesting potential improvements with larger datasets and time-series features. Although not outperforming the RF model, the RNN’s capacity to capture intricate patterns remains a noteworthy advantage, particularly in scenarios involving larger, more dynamic datasets. Notably, prior studies like Cleophas et al. [38] have highlighted the benefits of RNNs in capturing complex relationships, though performance can vary significantly with data quality and quantity.

Logistic regression, traditionally favored for its interpretability, exhibited an AUC of 0.82, which is slightly lower than the more complex models like RF and RNN. This result is consistent with existing literature indicating that logistic regression often provides a baseline level of performance but may not fully capture the complexities of clinical data, especially when interactions between variables are nonlinear and multifaceted. Studies such as Holmes et al. [39] have found logistic regression effective but limited when dealing with complex, high-dimensional data. Nevertheless, its simplicity and ease of interpretation remain beneficial for understanding individual predictors’ influence.

Extreme gradient boosting (XGB) achieved an AUC of 0.83, showcasing its ability to handle structured data and select relevant features effectively. While its performance was competitive, the RF model consistently outperformed it across key metrics, highlighting the importance of model selection based on dataset characteristics and clinical needs. Similar studies, such as those conducted by Baxter et al. [40], have demonstrated the strength of XGB in various clinical contexts, although the choice between XGB and RF often depends on specific data characteristics and computational considerations.

The support vector machine (SVM) achieved an AUC of 0.82, similar to logistic regression. Despite SVMs’ effectiveness in high-dimensional spaces, their performance was limited, likely due to kernel selection and clinical data complexity. The results are consistent with other studies in the field that report SVM as a powerful classifier, particularly when paired with appropriate kernel functions and parameter tuning [36].

SHAP values were used to identify the crucial predictors of candidemia. Notably, the following factors emerged as positively associated with candidemia risk: high levels of total bilirubin, patient age, total parenteral nutrition (TPN), duration of ICU stay, the presence of a central venous catheter (CVC), and the influence of glucocorticoids. Among these factors, high level of total bilirubin is of most valuable feature. Serum bilirubin is a stable and well-established marker of liver impairment [41,42]. The level of bilirubin is negatively correlated with antimicrobial immunity, rendering individuals more susceptible to candidiasis [43]. Albumin act as an antioxidant and free radical scavenger which can offer protection from inflammatory processes [44]. Patients with hypoalbuminemia are more susceptible to infect opportunistic pathogens like candidemia. However, hypoalbuminemia was not identified as a predict feature for distinguishing between candidemia and bacteremia in this model. In inflammatory states, capillary escape and breakdown of albumin increased which can lead to hypoalbuminemia [44]. Besides, server inflammation often combined with liver failure, which could decrease serum albumin mass because albumin synthesis rate is mainly decreased in liver failure [45]. Low serum albumin levels have been associated with an increased risk of mortality in patients with bloodstream infections because of bacterial or fungal infection [46]. Several studies have demonstrated a correlation between hypoalbuminemia and higher mortality rates in these patients. For example, a study by Lao et al. [47] found that low serum albumin levels were independently associated with increased mortality in patients with candidemia. Similarly, a study by A. Capelastegui et al. [48] showed that hypoalbuminemia was a significant predictor of mortality in patients with bacteremia.

The remaining positive predicators identified in our study have been supported by previous research in various contexts. For example, both TPN and CVC presence have been recognized as an independent risk factor for candidemia [2]. TPN induced candidemia may arise from alternations in gut microbiota [49], leading to changes in epithelial barrier function and subsequent intestinal inflammation [50]. The strong association of CVC presence with candidemia, even after adjusting for TPN, underscores its significance for candidemia [19].

Duration of stay in the ICU has also been proposed as an essential indicator of candidemia [51]. This metric is influenced by various factors, including patient demographics, treatment complexity and comorbidities. Moreover, glucocorticoids, which are used in the management of various immune diseases, carry several side effects, such as insulin resistance, increased susceptibility to severe infections, Cushing’s syndrome, and peptic ulcers [52]. Prolonged treatment with high-dose systemic glucocorticoids elevates the risk of candidemia [53].

In addition, our study highlighted the association of recent abdominal surgery, chemotherapy, solid cancer and antimicrobial therapy with candidemia. Surgery may compromise the natural body barrier [54], cytotoxic chemotherapy can disrupt the mucosal integrity [55], and antimicrobial therapy may induce alterations in intestinal flora [2]. All these factors collectively create an environment conducive to Candida invasion through mesenteric circulation [2,55].

Platelets are widely recognized as integral components of the host’s defense system against invasive infections [56]. In our study, low platelet count was positively associated with candidemia prediction. This finding is significant since platelets paly a protective role in integrating vascular endothelial cells against Candida–induced damage [56]. The decreased presence of platelets may then increase the risk of candidemia. PCT and CRP are well-known for their predictive value in assessing the prognosis of patients with candidiasis and bacterial infections. Some studies have even attempted to use these two markers to distinguish candidemia from bacteremia [57]. Like PCT and CRP, lymphocytes, WBC and monocytes were negatively correlated with candidemia in this study. These findings are particularly intriguing, as white blood play a pivotal role in initiation and maintenance of phagocyte-dependent fungal eliminations [58]. Furthermore, lymphocyte subtyping has even been used as a good marker in the early diagnosis of candidemia [59], which we did not included in this study. Interestingly, our study uncovers an association between low hemoglobin level and high level of bilirubin with an increased incidence of candidemia. This observation might be attributed to hemolytic factors produced by Candida, which can lead to rupture of red blood cells upon binding to the cell membrane [60].

This study has several clinical application prospects. Firstly, the developed machine learning models provide a reliable early prediction method for candidemia in intensive care unit (ICU) patients, which could assist clinicians in making timely decisions regarding the initiation of antifungal therapy. By identifying high-risk patients in advance, hospitals can arrange treatment and monitoring equipment in advance, thereby avoiding additional costs and adverse outcomes resulting from delayed diagnosis. Accurately identifying candidemia patients helps hospitals allocate resources more effectively and can reduce unnecessary resource wastage. Secondly, the identification of key risk factors through SHAP analysis reveals potential mechanisms underlying the development of candidemia, providing important insights for implementing personalized interventions for high-risk patients. Additionally, the predictive nomogram offers a convenient tool for clinicians to estimate the probability of candidemia based on patients’ clinical characteristics, aiding in the optimization of individualized treatment plans. Overall, this study provides new insights and approaches for improving patient outcomes, optimizing resource utilization, and enhancing the management strategies for fungal bloodstream infections.

There are several limitations in our study. First, the sample size was constrained by the fact that the data was collected from a single medical center. The relatively small sample size may have implications for the generalizability of our findings. Second, our study design was retrospective in nature. This choice was necessitated by the low incidence of candidemia, which limited our ability to incorporate the latest biomarkers into our predictive model. Third, although our model demonstrated good performance and achieved acceptable results during external validation, it is essential to acknowledge that retrospective studies inherently carry the potential bias. Therefore, there is a need for multiple-center prospective cohorts to expand our understanding and refine the predicative model.

Conclusions

In this study, we have demonstrated that the machine learning model on the RF algorithm outperforms other algorithms in predicting candidemia. Our findings emphasize the critical importance of considering total bilirubin levels as a key predictor for candidemia, particularly among ICU patients. The RF model’s predictive capabilities could serve as a valuable tool for clinicians in the early diagnosis and treatment of candidemia. The results of this study suggest the potential utility of AI-assisted diagnosis for enhancing the clinical management of candidemia, representing a promising avenue for improving patient care in this context.

Supporting information

S1 Table Characteristics of selected features in the external validation group.

Definition of abbreviations: y: Years; TPN: Total parenteral nutrition; CVC: Central venous catheter; WBC: White blood cell; PCT: Procalcitonin; CRP: C-reactive protein; ICU: Intensive care unit; IQR: Interquartile range; SD: Standard deviation; *The t-test for metric variable if data are normally distributed; **the Chi-square test (big sample size) and Fisher’s exact test (small sample size) for categorical variables; ***the Mann–Whitney U-test for metric variables if data are not normally distributed.

(DOCX)

S2 Table Comparison of selected features between internal and external validation.

Definition of abbreviations: y: Years; TPN: Total parenteral nutrition; CVC: Central venous catheter; WBC: White blood cell; PCT: Procalcitonin; CRP: C-reactive protein; ICU: Intensive care unit; IQR: Interquartile range; SD: Standard deviation; *The t-test for metric variable if data are normally distributed; **the Chi-square test (big sample size) and Fisher’s exact test (small sample size) for categorical variables; ***the Mann–Whitney U-test for metric variables if data are not normally distributed.

(DOCX)

S3 Table Nomogram points system for clinical variables.

Definition of abbreviations: TPN: Total parenteral nutrition; CRP: C-reactive protein; WBC: White blood cell; CVC: Central venous catheter.

(DOCX)

10.1371/journal.pone.0309748.r001
Decision Letter 0
Giordano Giovanni Academic Editor
© 2024 Giovanni Giordano
2024
Giovanni Giordano
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Submission Version0
28 Jun 2024

PONE-D-24-20146A machine learning model for early candidemia prediction in the intensive care unit: Clinical applicationPLOS ONE

Dear Dr. Meng,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Aug 12 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Giovanni Giordano

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, we expect all author-generated code to be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse.

3. Thank you for stating the following financial disclosure: 

Key R&D Program of Jining, grant number 2023YXNS151. 

  

Please state what role the funders took in the study.  If the funders had no role, please state: "The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript." 

If this statement is not correct you must amend it as needed. 

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

4. PLOS requires an ORCID iD for the corresponding author in Editorial Manager on papers submitted after December 6th, 2016. Please ensure that you have an ORCID iD and that it is validated in Editorial Manager. To do this, go to ‘Update my Information’ (in the upper left-hand corner of the main menu), and click on the Fetch/Validate link next to the ORCID field. This will take you to the ORCID site and allow you to create a new iD or authenticate a pre-existing iD in Editorial Manager. Please see the following video for instructions on linking an ORCID iD to your Editorial Manager account: https://www.youtube.com/watch?v=_xcclfuvtxQ

Additional Editor Comments:

Dear Authors,

please address the following:

Reviewer 1:

1. The introduction and especially the literature review is not sufficient. Authors need to add more state-of-the-art methods.

2. The model development section is not clear, it should include an explaination of the working principle of all the models utilized in this work.

3. The features used as inputs for the SHAP analysis are not mentioned.

4. The results should be compared with state-of-the-art methods.

Reviewer 2:

Major Comments:

1. The authors carried out 10-fold cross-validation of the algorithms by randomly assigning 7/10 groups for training the models and the remaining 3/10 for internal validation. This is considered standard best practices. However, the results in Table 2 are inconsistent with this premise and suggest that there was a single static pair of training and testing data.

2. Related to the previous point, the results in Figure 2 do not clarify which data set was used. Additionally, if the training or internal validation data were used, then there should be multiple curves for each model corresponding to each fold of the cross-validation. The same lack of clarity also applies to Table 3.

3. The nomogram is a nice addition that can improve the clinical utility of the models here. This is a graphical device, which can lack precision when used. I suggest that the authors also include a table showing the coefficients of the nomogram formula for more precise calculation of risk by clinicians.

4. The authors opted to exclude patients with 50% or more missing features but without providing adequate justification or performing sensitivity analyses to assess the impact of using lower or higher cutoffs.

5. Missing features of the remaining patients were imputed using mean values instead of using more refined approaches such as multiple imputation.

6. On the topic of missing data, please explain how clinicians should deal with missing values when using the nomogram or its formula when attempting to calculate the risk of candidemia infection. This is particularly critical since the model uses 18 features, each having a possibility of being missing, and the overall probability of at least one missing feature is likely moderate to high.

7. The bivariate analysis suggests that many (if not most) features are not marginally associated with infection. I suggest trying to build a more compact model and comparing its performance to the RF model presented. This is likely to have sub-optimal performance relative to the RF model, but if the performance is still comparable (say, AUC>0.8) then this might be more useful to clinician, especially given the likelihood of missing discussed previously.

8. Another reason to consider a more parsimonious model is that the sample size may not allow for adequate power to support 18 variables, and it’s likely that the predictive strength is being driven by only a handful of variables.

9. The data was matched based on hospitalization duration, which was defined as the number of days from hospital admission to the occurrence of candidemia. This is unclear since we do not know the exact time of candidemia occurrence, and especially since testing can take a long time.

Minor Comments

There are many grammatical and organizational errors that need to be revised to improve the readability of the paper. I highlighted these as comments on the manuscript. Moreover, the figure and table captions should be further improved to clearly indicate which sample was used and the sample size.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors have done a good job, the work is interesting, and the results have been presented well however, there are some observations:

1. The introduction and especially the literature review is not sufficient. Authors need to add more state-of-the-art methods.

2. The model development section is not clear, it should include an explaination of the working principle of all the models utilized in this work.

3. The features used as inputs for the SHAP analysis are not mentioned.

4. The results should be compared with state-of-the-art methods.

Reviewer #2: This paper presents and contrasts predictive models for early detection of candidemia infection that leverage more readily available clinical information in ICU patients. The results suggest that the machine learning algorithms demonstrated strong predictive strengths against testing and validation data, thus allowing clinicians to identify potential infections in a timely manner.

Major Comments:

1. The authors carried out 10-fold cross-validation of the algorithms by randomly assigning 7/10 groups for training the models and the remaining 3/10 for internal validation. This is considered standard best practices. However, the results in Table 2 are inconsistent with this premise and suggest that there was a single static pair of training and testing data.

2. Related to the previous point, the results in Figure 2 do not clarify which data set was used. Additionally, if the training or internal validation data were used, then there should be multiple curves for each model corresponding to each fold of the cross-validation. The same lack of clarity also applies to Table 3.

3. The nomogram is a nice addition that can improve the clinical utility of the models here. This is a graphical device, which can lack precision when used. I suggest that the authors also include a table showing the coefficients of the nomogram formula for more precise calculation of risk by clinicians.

4. The authors opted to exclude patients with 50% or more missing features but without providing adequate justification or performing sensitivity analyses to assess the impact of using lower or higher cutoffs.

5. Missing features of the remaining patients were imputed using mean values instead of using more refined approaches such as multiple imputation.

6. On the topic of missing data, please explain how clinicians should deal with missing values when using the nomogram or its formula when attempting to calculate the risk of candidemia infection. This is particularly critical since the model uses 18 features, each having a possibility of being missing, and the overall probability of at least one missing feature is likely moderate to high.

7. The bivariate analysis suggests that many (if not most) features are not marginally associated with infection. I suggest trying to build a more compact model and comparing its performance to the RF model presented. This is likely to have sub-optimal performance relative to the RF model, but if the performance is still comparable (say, AUC>0.8) then this might be more useful to clinician, especially given the likelihood of missing discussed previously.

8. Another reason to consider a more parsimonious model is that the sample size may not allow for adequate power to support 18 variables, and it’s likely that the predictive strength is being driven by only a handful of variables.

9. The data was matched based on hospitalization duration, which was defined as the number of days from hospital admission to the occurrence of candidemia. This is unclear since we do not know the exact time of candidemia occurrence, and especially since testing can take a long time.

Minor Comments

There are many grammatical and organizational errors that need to be revised to improve the readability of the paper. I highlighted these as comments on the manuscript. Moreover, the figure and table captions should be further improved to clearly indicate which sample was used and the sample size.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

Attachment Submitted filename: PONE-D-24-20146_reviewer.pdf

10.1371/journal.pone.0309748.r002
Author response to Decision Letter 0
Submission Version1
10 Aug 2024

Repones to editor:

1. We have revised the manuscript according to the journal's formatting requirements.

2. The code generated for this study is shared on GitHub at the following URL: https://github.com/mengqiang1985/candidemia-code.

3. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

4. we have added and validated the ORCID ID for the corresponding author, Feng Qu, in the Editorial Manager system.

Repones to Reviewer 1:

1. We understand the importance of providing a comprehensive introduction and literature review. In line 72, we have added a review of Naïve Bayesian (NB), k-Nearest Neighbour (KNN), logistic regression (LR), random forest (RF), and support vector machine (SVM) in the introduction.

2. We appreciate your suggestions to enhance the clarity of the model development section. In the revised section of line 148-159, we have provided the following explanations:

Extreme gradient boosting (XGBoost) operates by sequentially constructing an ensemble of decision trees, where each tree corrects the errors of its predecessors. This method uses gradient descent to minimize the loss function and enhance model performance [27]. Logistic Regression (LR) is a form of linear regression aimed at establishing the relationship between a binary response variable (indicating the occurrence of an event) and a set of explanatory variables. It achieves this by using a logistic function to model the probability of the binary outcome [28]. Support Vector Machine (SVM) works by constructing an optimal hyperplane that maximizes the margin between different classes. For nonlinear problems, it employs kernel functions to project data into higher-dimensional spaces for effective classification [29]. Recurrent Neural Networks (RNN) simulate the interconnected structure of neurons. In-put data passes through multiple layers of nodes (neurons), which apply weighted sums and activation functions to learn complex patterns and make predictions [30]. Random Forest (RF) is an ensemble method that trains multiple regression trees on different random subsets of the data. At each node, a randomly selected subset of predictors is used, which enhances robustness to predictor correlations [31].

3. We appreciate your suggestion to specify the input features used in the SHAP analysis. To address this concern, we have updated the manuscript to explicitly list the input features utilized in the SHAP analysis in line 184 of the revised manuscript.

4. We appreciate the reviewer's suggestion to compare our results with state-of-the-art methods. In the revised discussion section, we have revised the discussion section to include detailed comparisons of not only the Random Forest (RF) model but also the Recurrent Neural Network (RNN), Logistic Regression (LR), Extreme Gradient Boosting (XGB), and Support Vector Machine (SVM) models with results from the existing literature:

The recurrent neural network (RNN) model also showed strong diagnostic performance, with an AUC of 0.84 and the highest accuracy (0.79) among the models tested. RNNs are recognized for their ability to model sequential data and complex temporal patterns, suggesting potential improvements with larger datasets and time-series features. Although not outperforming the RF model, the RNN's capacity to capture intricate patterns remains a note-worthy advantage, particularly in scenarios involving larger, more dynamic datasets. Notably, prior studies like Cleophas et al. [38] have highlighted the benefits of RNNs in capturing complex relationships, though performance can vary significantly with data quality and quantity.

Logistic regression, traditionally favored for its interpretability, exhibited an AUC of 0.82, which is slightly lower than the more complex models like RF and RNN. This result is consistent with existing literature indicating that logistic regression often provides a baseline level of performance but may not fully capture the complexities of clinical data, especially when interactions between variables are nonlinear and multifaceted. Studies such as Holmes et al. [39] have found logistic regression effective but limited when dealing with complex, high-dimensional data. Nevertheless, its simplicity and ease of interpretation remain beneficial for understanding individual predictors' influence.

Extreme gradient boosting (XGB) achieved an AUC of 0.83, showcasing its ability to handle structured data and select relevant features effectively. While its performance was competitive, the RF model consistently outperformed it across key metrics, highlighting the importance of model selection based on dataset characteristics and clinical needs. Similar studies, such as those conducted by Baxter et al. [40], have demonstrated the strength of XGB in various clinical contexts, although the choice between XGB and RF often depends on specific data characteristics and computational considerations.

The support vector machine (SVM) achieved an AUC of 0.82, similar to logistic regression. Despite SVMs' effectiveness in high-dimensional spaces, their performance was limited, likely due to kernel selection and clinical data complexity. The results are consistent with other studies in the field that report SVM as a powerful classifier, particularly when paired with appropriate kernel functions and parameter tuning [36].

Repones to Reviewer 2:

1. Thank you for your observation. We apologize for any confusion caused by our initial description. Here we provide a detailed explanation of our model training and validation process to clarify our methodology:

In our study, we first divided the entire dataset into a training set (70%) and a validation set (30%). We then applied 10-fold cross-validation on the training set to determine the optimal hyperparameters for each model (RF, SVM, RNN, LR, XGB). Therefore, Table 2 only presents a single static pairing of the training and testing data.

2. We appreciate your feedback and agree that Figure 2 and Table 3 need more clarity. For Figure 2, we utilized the training set data to illustrate the performance of the models. The figure aims to provide a visual representation of model behavior based on the training dataset. We recognize the potential confusion regarding which dataset was used and will ensure this is clarified in the figure caption and accompanying text in the revised manuscript. Regarding Table 3, metrics such as AUC, among others, reflect the model performance after parameter tuning using cross-validation. We employed 10-fold cross-validation to optimize the model parameters, which is standard practice to ensure robust performance assessment. The single result for each metric in Table 3 reflects the aggregate performance across all folds rather than results from each individual fold, providing a clear summary of model performance.

3. Thank you for your insightful comments regarding the inclusion of a nomogram to improve the clinical utility of our models. We appreciate the suggestion to provide a table with the coefficients of the nomogram formula for precise risk calculation. However, our nomogram is based on a random forest model, which inherently differs from models that provide linear or non-linear formulas, such as logistic regression. Random forest is an ensemble learning method that operates by constructing multiple decision trees during training and outputting the class that is the mode of the classes (classification) or mean prediction (regression) of the individual trees. To further facilitate clinical use, we plan to convert the content of Figure 5 into a list of point values and include it in the S3 Table. This addition will make it easier for clinicians to reference and apply these point values for risk assessment, enhancing the model's practicality.

4. We appreciate your suggestion to explore the impact of different missing data thresholds. We chose to exclude patients with 50% or more missing features to maintain data integrity and minimize the bias introduced by imputing a large amount of missing data. Retaining samples with higher data completeness ensures the reliability and interpretability of the model's predictions. However, we acknowledge the importance of sensitivity analyses and plan to explore the impact of different thresholds in future studies to assess their influence on model performance.

5. We used mean imputation for missing features to simplify the data processing and reduce computational complexity. However, we recognize that using more refined methods, such as multiple imputation, could improve the accuracy and robustness of the model. Therefore, we plan to apply methods like multiple imputation in future studies to evaluate their impact on model performance.

6. Thank you for highlighting this critical aspect of clinical application. To enhance the accuracy and reliability of the nomogram or its formula in calculating the risk of candidemia infection, we recommend that clinicians ensure the completeness of all 18 features. For the 10 binary variables, clinicians should obtain accurate information based on the patient's actual condition, reference common values from similar patient cohorts, or extract insights from medical records. For other clinical indicators, it is advisable to prioritize obtaining the most recent test data. If real-time data is unavailable, historical data from comparable patients can be used to estimate missing values, and consultation with specialists is encouraged to ensure precision. We acknowledge that incomplete data may occur in clinical practice; thus, clinical judgment and multidisciplinary collaboration are essential for the effective application of the model. We will continue to explore and validate optimal strategies for handling missing data to enhance the model's applicability and reliability across various clinical settings.

7. We appreciate your suggestion to explore a more compact model. We attempted to build a more compact model using variables closely related to infection, such as CRP, WBC, PCT, platelet count, lymphocytes, neutrophils, and monocytes. However, the simplified model achieved an AUC of only 0.645, which is below the clinically useful threshold of AUC > 0.8. This performance gap suggests that the reduced feature set may not capture all relevant interactions present in the Random Forest model. We acknowledge the importance of balancing simplicity and performance and plan to explore additional feature combinations and modeling techniques to achieve this.

8. Thank you for this insight. We attempted to create a more parsimonious model by selecting key variables related to infection. However, the simplified model did not achieve the desired diagnostic accuracy, with an AUC below the clinical utility threshold. This may be due to the inability to capture complex interactions between variables and the limitation of our sample size. Besides, we have employed 10-fold cross-validation to maximize the utility of our data, ensuring robust model validation and reducing the risk of overfitting. We will continue to explore other feature combinations and modeling approaches to enhance model performance, and consider increasing the sample size to improve predictive capability.

9. Thank you for your thorough review and valuable feedback on our study. Regarding your concerns about the unclear definition of hospitalization duration used in the data matching process, we have carefully considered your comments and have provided clarification in line 106-116 of the revised manuscript.:

To ensure a comparable periods of risk exposure for both groups, each control patient had a hospitalization duration similar to the time at risk of cases, defined as the number of days from hospital admission to the occurrence of candidemia. To ensure comparable risk exposure periods between candidemia cases and control patients, we implemented a meticulous matching process. Candidemia cases were retrospectively identified using the microbiology laboratory database, and only the first episode was considered if a patient experienced multiple episodes. Control patients diagnosed with bacteremia were selected in a 1:1 ratio and matched to candidemia cases based on several criteria, including age (±5 years), sex, date of hospital admission, and duration of hospitalization at the time of the first positive blood culture. The hospitalization duration for candidemia cases was defined as the number of days from hospital admission to the occurrence of candidemia. Similarly, for control patients, hospitalization duration was defined as the time from admission to the onset of bacteremia, ensuring alignment with the candidemia cases' risk period.

This matching approach was designed to address potential diagnostic delays in detecting candidemia, thereby ensuring that the hospitalization duration accurately reflects the true period of risk for each patient.

Minor Comments

1. Thank you for your feedback on the grammatical and organizational issues in our manuscript. We have carefully reviewed the comments you highlighted in the manuscript and made the necessary revisions to improve the readability of the paper. Specifically, we have corrected grammatical errors, including sentence structure, punctuation, and verb tense consistency. Additionally, we reorganized sections to ensure a smooth flow of information, making it easier for readers to understand our research process and findings. We also simplified complex or ambiguous sentences to enhance clarity and readability, ensuring that our scientific arguments are presented clearly and concisely.

2. Thank you for your comments regarding the selection of variables in Tables 1 and 2. In Table 1, we present the baseline characteristics of age and chronic diseases for all patients. The results show that, apart from age, there are no statistically significant differences in chronic diseases between the candidemia and bacteremia groups. This suggests that chronic diseases do not significantly affect the incidence of candidemia. Based on these findings, we did not include chronic diseases as screening variables for candidemia in Table 2. We believe this approach provides a clear and focused analysis of factors that are more directly relevant to candidemia incidence. We appreciate your feedback and will ensure that these considerations are clearly communicated in the manuscript.

3. Thank you for your valuable feedback. To address your concerns and provide a more comprehensive evaluation of the model's performance, we have added sensitivity and specificity results to Table 3 and Table 4.

4. We have revised the captions to clearly indicate which sample was used and the sample size, ensuring that readers can easily understand the context and details of each figure and table. The specific modifications are as follows:

Fig 1: Flowchart of candidemia and bacteremia patients selected in the ICU for model development and external validation. (A)Patients select from Jining NO. 1 people’s hospital for model development which including the training and internal validation sets. (B) Patients select from Teng Zhou People's Hospital for external validation set.

Table 1：Baseline features between the candidemia and bacteremia groups.

This table presents the baseline characteristics of patients diagnosed with candidemia (n = 167) and bacteremia (n = 167) in our study. The dataset was randomly split into a training set (70%, n=234) and a test set (30%, n=100) to develop and validate the diagnostic models. Statistical comparisons between the two groups were performed using t-tests for continuous variables and chi-square tests for categorical variables. The analysis results showed a significant difference in age between the two groups (p < 0.05), while other baseline disease parameters showed no statistical significance. Therefore, these baseline disease parameters were not considered in the model variable selection. Definition of abbreviations: y: years; CHD: coronary atherosclerotic heart disease.

Table 2：Comparison of patient parameters between candidemia and bacteremia groups in the training and the testing sets.

This table presents a comparison of key patient parameters, including demographic factors, laboratory values, an

Attachment Submitted filename: Response to Reviewers.docx

10.1371/journal.pone.0309748.r003
Decision Letter 1
Giordano Giovanni Academic Editor
© 2024 Giovanni Giordano
2024
Giovanni Giordano
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Submission Version1
19 Aug 2024

A machine learning model for early candidemia prediction in the intensive care unit: Clinical application

PONE-D-24-20146R1

Dear Dr. Qu,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Giovanni Giordano

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

10.1371/journal.pone.0309748.r004
Acceptance letter
Giordano Giovanni Academic Editor
© 2024 Giovanni Giordano
2024
Giovanni Giordano
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
30 Aug 2024

PONE-D-24-20146R1

PLOS ONE

Dear Dr. Qu,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Giovanni Giordano

Academic Editor

PLOS ONE
==== Refs
References

1 Wisplinghoff H , Bischoff T , Tallent SM , Seifert H , Wenzel RP , Edmond MB . Nosocomial bloodstream infections in US hospitals: analysis of 24,179 cases from a prospective nationwide surveillance study. Clin Infect Dis. 2004;39 (3 ):309–17. Epub 2004/08/13. doi: 10.1086/421946 .15306996
2 Poissy J , Damonti L , Bignon A , Khanna N , Von Kietzell M , Boggian K , et al . Risk factors for candidemia: a prospective matched case-control study. Crit Care. 2020;24 (1 ):109. Epub 2020/03/20. doi: 10.1186/s13054-020-2766-1 ; PubMed Central PMCID: PMC7081522.32188500
3 Zaoutis TE , Argon J , Chu J , Berlin JA , Walsh TJ , Feudtner C . The epidemiology and attributable outcomes of candidemia in adults and children hospitalized in the United States: a propensity analysis. Clin Infect Dis. 2005;41 (9 ):1232–9. Epub 2005/10/06. doi: 10.1086/496922 .16206095
4 Fisher BT , Boge CLK , Xiao R , Shuster S , Chin-Quee D , Allen J , et al . Multicenter Prospective Study of Biomarkers for Diagnosis of Invasive Candidiasis in Children and Adolescents. Clin Infect Dis. 2022;75 (2 ):248–59. Epub 2022/02/09. doi: 10.1093/cid/ciab928 ; PubMed Central PMCID: PMC9890499.35134165
5 Pappas PG , Lionakis MS , Arendrup MC , Ostrosky-Zeichner L , Kullberg BJ . Invasive candidiasis. Nat Rev Dis Primers. 2018;4 :18026. Epub 2018/05/12. doi: 10.1038/nrdp.2018.26 .29749387
6 Bassetti M , Giacobbe DR , Vena A , Wolff M . Diagnosis and Treatment of Candidemia in the Intensive Care Unit. Semin Respir Crit Care Med. 2019;40 (4 ):524–39. Epub 2019/10/05. doi: 10.1055/s-0039-1693704 .31585478
7 Magadia RR , Weinstein MP . Laboratory diagnosis of bacteremia and fungemia. Infect Dis Clin North Am. 2001;15 (4 ):1009–24. Epub 2002/01/10. doi: 10.1016/s0891-5520(05)70184-7 .11780265
8 Playford EG , Lipman J , Jones M , Lau AF , Kabir M , Chen SC , et al . Problematic Dichotomization of Risk for Intensive Care Unit (ICU)-Acquired Invasive Candidiasis: Results Using a Risk-Predictive Model to Categorize 3 Levels of Risk From a Multicenter Prospective Cohort of Australian ICU Patients. Clin Infect Dis. 2016;63 (11 ):1463–9. Epub 2016/09/08. doi: 10.1093/cid/ciw610 .27601224
9 Morrell M , Fraser VJ , Kollef MH . Delaying the empiric treatment of candida bloodstream infection until positive blood culture results are obtained: a potential risk factor for hospital mortality. Antimicrobial agents and chemotherapy. 2005;49 (9 ):3640–5. Epub 2005/08/30. doi: 10.1128/AAC.49.9.3640-3645.2005 ; PubMed Central PMCID: PMC1195428.16127033
10 Zilberberg MD , Kollef MH , Arnold H , Labelle A , Micek ST , Kothari S , et al . Inappropriate empiric antifungal therapy for candidemia in the ICU and hospital resource utilization: a retrospective cohort study. BMC Infect Dis. 2010;10 :150. Epub 2010/06/08. doi: 10.1186/1471-2334-10-150 ; PubMed Central PMCID: PMC2890008.20525301
11 Liesenfeld O , Lehman L , Hunfeld KP , Kost G . Molecular diagnosis of sepsis: New aspects and recent developments. Eur J Microbiol Immunol (Bp). 2014;4 (1 ):1–25. Epub 2014/03/29. doi: 10.1556/EuJMI.4.2014.1.1 ; PubMed Central PMCID: PMC3955828.24678402
12 Zhang J , Cheng W , Li D , Chen J , Zhao G , Wang H , et al . Development and Validation of a Risk Score for Predicting Invasive Candidiasis in Intensive Care Unit Patients by Incorporating Clinical Risk Factors and Lymphocyte Subtyping. Front Cell Infect Microbiol. 2022;12 :829066. Epub 2022/05/17. doi: 10.3389/fcimb.2022.829066 ; PubMed Central PMCID: PMC9091371.35573797
13 Paiva JA , Charles PE . Biomarker-guided antifungal therapy in patients with suspected invasive candidiasis: Ready for prime time? Intensive Care Med. 2017;43 (12 ):1889–91. Epub 2017/11/10. doi: 10.1007/s00134-017-4990-y .29119220
14 Posteraro B , Tumbarello M , De Pascale G , Liberto E , Vallecoccia MS , De Carolis E , et al . (1,3)-β-d-Glucan-based antifungal treatment in critically ill adults at high risk of candidaemia: an observational study. J Antimicrob Chemother. 2016;71 (8 ):2262–9. Epub 2016/04/30. doi: 10.1093/jac/dkw112 .27125554
15 Chertoff J , Ataya A . The Timing of Early Antibiotics and Hospital Mortality in Sepsis: Playing Devil’s Advocate. Am J Respir Crit Care Med. 2017;196 (7 ):934–5. Epub 2017/05/10. doi: 10.1164/rccm.201703-0657LE .28485616
16 Bhavani SV , Lonjers Z , Carey KA , Afshar M , Gilbert ER , Shah NS , et al . The Development and Validation of a Machine Learning Model to Predict Bacteremia and Fungemia in Hospitalized Patients Using Electronic Health Record Data. Crit Care Med. 2020;48 (11 ):e1020–e8. Epub 2020/08/17. doi: 10.1097/CCM.0000000000004556 ; PubMed Central PMCID: PMC7554143.32796184
17 Leiner T , Rueckert D , Suinesiaputra A , Baessler B , Nezafat R , Isgum I , et al . Machine learning in cardiovascular magnetic resonance: basic concepts and applications. J Cardiovasc Magn Reson. 2019;21 (1 ):61. Epub 2019/10/09. doi: 10.1186/s12968-019-0575-y ; PubMed Central PMCID: PMC6778980.31590664
18 Yoo J , Kim SH , Hur S , Ha J , Huh K , Cha WC . Candidemia Risk Prediction (CanDETEC) Model for Patients With Malignancy: Model Development and Validation in a Single-Center Retrospective Study. JMIR Med Inform. 2021;9 (7 ):e24651. Epub 2021/07/27. doi: 10.2196/24651 ; PubMed Central PMCID: PMC8367162.34309570
19 Atamna A , Eliakim-Raz N , Mohana J , Ben-Zvi H , Sorek N , Shochat T , et al . Predicting candidemia in the internal medicine wards: a comparison with gram-negative bacteremia-a retrospectives study. Diagnostic microbiology and infectious disease. 2019;95 (1 ):80–3. Epub 2019/05/28. doi: 10.1016/j.diagmicrobio.2019.04.007 .31129007
20 Gao Y , Tang M , Li Y , Niu X , Li J , Fu C , et al . Machine-learning based prediction and analysis of prognostic risk factors in patients with candidemia and bacteraemia: a 5-year analysis. PeerJ. 2022;10 . doi: 10.7717/peerj.13594 35726257
21 Hu WH , Lin SY , Hu YJ , Huang HY , Lu PL . Application of machine learning for mortality prediction in patients with candidemia: Feasibility verification and comparison with clinical severity scores. Mycoses. 2024;67 (1 ):e13667. Epub 2023/11/02. doi: 10.1111/myc.13667 .37914666
22 Moons KG , Altman DG , Reitsma JB , Ioannidis JP , Macaskill P , Steyerberg EW , et al . Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162 (1 ):W1–73. Epub 2015/01/07. doi: 10.7326/M14-0698 .25560730
23 Ripoli A , Sozio E , Sbrana F , Bertolino G , Pallotto C , Cardinali G , et al . Personalized machine learning approach to predict candidemia in medical wards. Infection. 2020;48 (5 ):749–59. Epub 2020/08/03. doi: 10.1007/s15010-020-01488-3 .32740866
24 Remeseiro B , Bolon-Canedo V . A review of feature selection methods in medical applications. Comput Biol Med. 2019;112 :103375. Epub 2019/08/06. doi: 10.1016/j.compbiomed.2019.103375 .31382212
25 Lin H , Xue Y , Chen K , Zhong S , Chen L . Acute coronary syndrome risk prediction based on gradient boosted tree feature selection and recursive feature elimination: A dataset-specific modeling study. PLoS One. 2022;17 (11 ):e0278217. Epub 2022/11/30. doi: 10.1371/journal.pone.0278217 ; PubMed Central PMCID: PMC9707772.36445881
26 Guyon I , Elisseeff A . An Introduction to Variable and Feature Selection. Journal of machine learning research. 2003;(3 ):1157–82.
27 Lv CX , An SY , Qiao BJ , Wu W . Time series analysis of hemorrhagic fever with renal syndrome in mainland China by using an XGBoost forecasting model. BMC Infect Dis. 2021;21 (1 ):839. Epub 2021/08/21. doi: 10.1186/s12879-021-06503-y ; PubMed Central PMCID: PMC8377883.34412581
28 Cox DRJJotRSSSBSM. The regression analysis of binary sequences. 1958;20 (2 ):215–32.
29 Cortes C , Vapnik VJMl . Support-vector networks. Machine Learning. 1995;20 :273–97. doi: 10.1007/bf00994018
30 Zeng Z , Tang X , Liu Y , He Z , Gong X . Interpretable recurrent neural network models for dynamic prediction of the extubation failure risk in patients with invasive mechanical ventilation in the intensive care unit. BioData Min. 2022;15 (1 ):21. Epub 2022/09/27. doi: 10.1186/s13040-022-00309-7 ; PubMed Central PMCID: PMC9513908.36163063
31 Breiman L. Random forests. Machine learning. 2001;45 :5–32.
32 Alba AC , Agoritsas T , Walsh M , Hanna S , Iorio A , Devereaux PJ , et al . Discrimination and Calibration of Clinical Prediction Models: Users’ Guides to the Medical Literature. JAMA. 2017;318 (14 ):1377–84. Epub 2017/10/20. doi: 10.1001/jama.2017.12126 .29049590
33 Li J , Gong M , Joshi Y , Sun L , Huang L , Fan R , et al . Machine Learning Prediction Model for Acute Renal Failure After Acute Aortic Syndrome Surgery. Frontiers in medicine. 2021;8 :728521. Epub 2022/02/04. doi: 10.3389/fmed.2021.728521 ; PubMed Central PMCID: PMC8801502.35111767
34 Giorgi FM , Ceraolo C , Mercatelli D . The R Language: An Engine for Bioinformatics and Data Science. Life (Basel). 2022;12 (5 ). Epub 2022/05/29. doi: 10.3390/life12050648 ; PubMed Central PMCID: PMC9148156.35629316
35 Deberneh HM , Kim I . Prediction of Type 2 Diabetes Based on Machine Learning Algorithm. Int J Environ Res Public Health. 2021;18 (6 ). Epub 2021/04/04. doi: 10.3390/ijerph18063317 ; PubMed Central PMCID: PMC8004981.33806973
36 Li J , Li Y , Gao Y , Niu X , Tang M , Fu C , et al . Prediction of Prognostic Risk Factors in Patients with Invasive Candidiasis and Cancer: A Single-Centre Retrospective Study. BioMed Research International. 2022;2022 :1–9. doi: 10.1155/2022/7896218 35692595
37 Li Y , Wu Y , Gao Y , Niu X , Li J , Tang M , et al . Machine-learning based prediction of prognostic risk factors in patients with invasive candidiasis infection and bacterial bloodstream infection: a singled centered retrospective study. BMC Infect Dis. 2022;22 (1 ):150. Epub 2022/02/15. doi: 10.1186/s12879-022-07125-8 ; PubMed Central PMCID: PMC8841094.35152879
38 Cleophas TJ , Cleophas TF . Artificial intelligence for diagnostic purposes: principles, procedures and limitations. cclm. 2010;48 (2 ):159–65. doi: 10.1515/CCLM.2010.045 20001439
39 Holmes JH , Sacchi L , Bellazzi R , Peek N . Artificial Intelligence in Medicine AIME 2015. Artif Intell Med. 2017;81 :1–2. Epub 2017/07/25. doi: 10.1016/j.artmed.2017.06.011 .28733119
40 <embc44109.2020.9175287.pdf>.
41 Perez Ruiz de Garibay A , Kortgen A , Leonhardt J , Zipprich A , Bauer M . Critical care hepatology: definitions, incidence, prognosis and role of liver failure in critically ill patients. Critical Care. 2022;26 (1 ). doi: 10.1186/s13054-022-04163-1 36163253
42 Hadem J , Stiefel P , Bahr MJ , Tillmann HL , Rifai K , Klempnauer J , et al . Prognostic implications of lactate, bilirubin, and etiology in German patients with acute liver failure. Clin Gastroenterol Hepatol. 2008;6 (3 ):339–45. Epub 2008/03/11. doi: 10.1016/j.cgh.2007.12.039 .18328438
43 Chang Y-C , Chen J-S , Yin C-H , Shin-Jung Lee S , Chen W-C . Candidemia in hospitalized cirrhotic patients with bloodstream infection: A retrospective analysis and brief summary of published studies. Journal of the Chinese Medical Association. 2022;85 (3 ):295–303. doi: 10.1097/JCMA.0000000000000695 35259132
44 Soeters PB , Wolfe RR , Shenkin A . Hypoalbuminemia: Pathogenesis and Clinical Significance. JPEN J Parenter Enteral Nutr. 2019;43 (2 ):181–93. Epub 2018/10/06. doi: 10.1002/jpen.1451 ; PubMed Central PMCID: PMC7379941.30288759
45 Rafferty MJ , Mcmillan DC , Preston TC , Hamid R , Small AC , Joshi N , et al . Reprioritisation of Liver Export Protein Synthesis in Patients with Decompensated Alcoholic Liver Disease. 2016;(3 ).
46 Wiedermann CJ . Hypoalbuminemia as Surrogate and Culprit of Infections. Int J Mol Sci. 2021;22 (9 ). Epub 2021/05/01. doi: 10.3390/ijms22094496 ; PubMed Central PMCID: PMC8123513.33925831
47 Lao M , Li C , Li J , Chen D , Ding M , Gong Y . Opportunistic invasive fungal disease in patients with type 2 diabetes mellitus from Southern China: Clinical features and associated factors. Journal of diabetes investigation. 2020;11 (3 ):731–44. Epub 2019/11/24. doi: 10.1111/jdi.13183 ; PubMed Central PMCID: PMC7232281.31758642
48 Capelastegui A , Zalacain R , Bilbao A , Egurrola M , Iturriaga LA , Quintana JM , et al . Pneumococcal pneumonia: differences according to blood culture results. BMC Pulm Med. 2014;14 :128. Epub 2014/08/07. doi: 10.1186/1471-2466-14-128 ; PubMed Central PMCID: PMC4127038.25096919
49 Demehri FR , Barrett M , Ralls MW , Miyasaka EA , Feng Y , Teitelbaum DH . Intestinal epithelial cell apoptosis and loss of barrier function in the setting of altered microbiota with enteral nutrient deprivation. Front Cell Infect Microbiol. 2013;3 :105. Epub 2014/01/07. doi: 10.3389/fcimb.2013.00105 ; PubMed Central PMCID: PMC3870295.24392360
50 Harris JK , El Kasmi KC , Anderson AL , Devereaux MW , Fillon SA , Robertson CE , et al . Specific microbiome changes in a mouse model of parenteral nutrition associated liver injury and intestinal inflammation. PLoS One. 2014;9 (10 ):e110396. Epub 2014/10/21. doi: 10.1371/journal.pone.0110396 ; PubMed Central PMCID: PMC4203793.25329595
51 Zhang Z , Zhu R , Luan Z , Ma X . Risk of invasive candidiasis with prolonged duration of ICU stay: a systematic review and meta-analysis. BMJ Open. 2020;10 (7 ):e036452. Epub 2020/07/15. doi: 10.1136/bmjopen-2019-036452 ; PubMed Central PMCID: PMC7359383.32660950
52 Yang R , Yu Y . Glucocorticoids are double-edged sword in the treatment of COVID-19 and cancers. Int J Biol Sci. 2021;17 (6 ):1530–7. Epub 2021/04/29. doi: 10.7150/ijbs.58695 ; PubMed Central PMCID: PMC8071771.33907516
53 Lionakis MS , Kontoyiannis DP . Glucocorticoids and invasive fungal infections. Lancet. 2003;362 (9398 ):1828–38. Epub 2003/12/05. doi: 10.1016/S0140-6736(03)14904-5 .14654323
54 Teoh F , Pavelka N . How Chemotherapy Increases the Risk of Systemic Candidiasis in Cancer Patients: Current Paradigm and Future Directions. Pathogens. 2016;5 (1 ). Epub 2016/01/20. doi: 10.3390/pathogens5010006 ; PubMed Central PMCID: PMC4810127.26784236
55 Nucci M , Anaissie E . Should vascular catheters be removed from all patients with candidemia? An evidence-based review. Clin Infect Dis. 2002;34 (5 ):591–9. Epub 2002/01/26. doi: 10.1086/338714 .11810600
56 Yeaman MR , Cheng D , Desai B , Kupferwasser LI , Xiong YQ , Gank KD , et al . Susceptibility to thrombin-induced platelet microbicidal protein is associated with increased fluconazole efficacy against experimental endocarditis due to Candida albicans. Antimicrobial agents and chemotherapy. 2004;48 (8 ):3051–6. Epub 2004/07/27. doi: 10.1128/AAC.48.8.3051-3056.2004 ; PubMed Central PMCID: PMC478484.15273120
57 Wang Q , Wang C , Yang M , Li X , Cui J , Wang C . Diagnostic efficacy of serum cytokines and chemokines in patients with candidemia and bacteremia. Cytokine. 2020;130 :155081. Epub 2020/04/05. doi: 10.1016/j.cyto.2020.155081 .32247169
58 Hatinguais R , Willment JA , Brown GD . PAMPs of the Fungal Cell Wall and Mammalian PRRs. Curr Top Microbiol Immunol. 2020;425 :187–223. Epub 2020/03/18. doi: 10.1007/82_2020_201 .32180018
59 Zhang J , Cui N , Wang H , Han W , Li Y , Xiao M , et al . Invasive Fungal Disease in Critically Ill Patients at High Risk: Usefulness of Lymphocyte Subtyping. J Intensive Care Med. 2020;35 (9 ):909–18. Epub 2018/09/21. doi: 10.1177/0885066618800690 .30231674
60 Roy U , Kornitzer D . Heme-iron acquisition in fungi. Current opinion in microbiology. 2019;52 :77–83. Epub 2019/07/03. doi: 10.1016/j.mib.2019.05.006 .31265986
