
==== Front
BMJ Open
BMJ Open
bmjopen
bmjopen
BMJ Open
2044-6055
BMJ Publishing Group BMA House, Tavistock Square, London, WC1H 9JR

39117411
10.1136/bmjopen-2023-081172
bmjopen-2023-081172
Original Research
Neurology
1713
1506
Diagnostic prediction models for bacterial meningitis in children with a suspected central nervous system infection: a systematic review and prospective validation study
http://orcid.org/0000-0002-3519-4015
Groeneveld Nina S 1n.s.groeneveld@amsterdamumc.nl

Bijlsma Merijn W 2m.w.bijlsma@amsterdamumc.nl

van Zeggeren Ingeborg E 1i.e.vanzeggeren@amsterdamumc.nl

Staal Steven L 1s.l.staal@amsterdamumc.nl

http://orcid.org/0000-0001-9828-4459
Tanck Michael W T 3m.w.tanck@amsterdamumc.nl

van de Beek Diederik 1D.vandeBeek@amsterdamumc.nl

Brouwer Matthijs C 1m.c.brouwer@amsterdamumc.nl

1 Department of Neurology, Amsterdam UMC Location AMC, Amsterdam, The Netherlands
2 Department of Pediatrics, Amsterdam UMC Location AMC, Amsterdam, The Netherlands
3 Department of Epidemiology and Data Science, Amsterdam UMC—Locatie AMC, Amsterdam, The Netherlands
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

We do not have any association with commercial entities that provided support for the work reported in the submitted manuscript. We do not have any association with commercial entities that could be viewed as having an interest in the general area of the submitted manuscript. We do not have any association involving spouses or children under 18 years of age.

Professor; m.c.brouwer@amsterdamumc.nl
2024
7 8 2024
14 8 e08117220 10 2023
17 7 2024
Copyright © Author(s) (or their employer(s)) 2024. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Abstract

Objectives

Diagnostic prediction models exist to assess the probability of bacterial meningitis (BM) in paediatric patients with suspected meningitis. To evaluate the diagnostic accuracy of these models in a broad population of children suspected of a central nervous system (CNS) infection, we performed external validation.

Methods

We performed a systematic literature review in Medline to identify articles on the development, refinement or validation of a prediction model for BM, and validated these models in a prospective cohort of children aged 0–18 years old suspected of a CNS infection.

Primary and secondary outcome measures

We calculated sensitivity, specificity, predictive values, the area under the receiver operating characteristic curve (AUC) and evaluated calibration of the models for diagnosis of BM.

Results

In total, 23 prediction models were validated in a cohort of 450 patients suspected of a CNS infection included between 2012 and 2015. In 75 patients (17%), the final diagnosis was a CNS infection including 30 with BM (7%). AUCs ranged from 0.69 to 0.94 (median 0.83, interquartile range [IQR] 0.79–0.87) overall, from 0.74 to 0.96 (median 0.89, IQR 0.82–0.92) in children aged ≥28 days and from 0.58 to 0.91 (median 0.79, IQR 0.75–0.82) in neonates.

Conclusions

Prediction models show good to excellent test characteristics for excluding BM in children and can be of help in the diagnostic workup of paediatric patients with a suspected CNS infection, but cannot replace a thorough history, physical examination and ancillary testing.

neurology
infectious diseases
diagnostic microbiology
decision making
http://dx.doi.org/10.13039/501100001826 ZonMw 917.17.308 918.19.627 European Research Council ERC Consolidator grant 101001237 ERC Starting grant
==== Body
pmcSTRENGTHS AND LIMITATIONS OF THIS STUDY

A strength of our study is that we present an extensive overview of published prediction models that predict the likelihood of bacterial meningitis. We furthermore validated these models in data from the Paediatric and Adult Causes of Encephalitis and Meningitis study. This was a multicentre prospective study in three hospitals in which patients aged 0–18 years old with a suspected CNS infection in which cerebrospinal fluid (CSF) examination was performed, were included.

Another strength of this study is that we evaluated the performance of these prediction models by assessing both discrimination and calibration.

A limitation of this study is that the median number of missing values per variable was 12% (IQR 4%–42%). Missing data were handled by multiple imputation.

Introduction

Bacterial meningitis (BM) in children is lethal and debilitating, with mortality rates between 4% and 21% and neurological sequelae occurring in up to one-third of survivors.13 Early start of treatment is crucial for the prognosis as delay in antibiotic treatment is associated with adverse outcomes.4 However, limiting unnecessary use of antibiotics is important to minimise antibiotic resistance, adverse reactions, hospital admissions and healthcare costs.5

Recognition of BM can be difficult. The typical triad of fever, neck stiffness and altered mental status is present in only 41% of adult patients and is even less common in children and infants.6 7 Diagnostic prediction models have been developed to help identify which child should be treated for BM and in which a watchful waiting approach can be applied.8 The majority of these models combine clinical and laboratory findings and predict the probability of acute BM, compared with viral meningitis or no meningitis. However, substantial differences between these models exist, especially with respect to patient populations and diagnostic criteria. Validation of prediction models in a broader population of patients suspected of a central nervous system (CNS) infection is necessary because this is the population in which these models will be of clinical use, however is often lacking. External validation of 16 diagnostic prediction models for BM in a cohort of 363 adult patients with a suspected CNS infection showed that none of the existing models performed well enough to recommend routine use in individual patient management. However, these models were mostly developed for children and might therefore perform better in a paediatric population.

Our aim was to perform a systematic review of prediction models for BM and validate these models using a multicentre cohort of paediatric patients with a suspected CNS infection in whom a lumbar puncture was performed.

Methods

Systematic review

We systematically reviewed the literature in Medline to identify models that predict the probability of acute BM. The Standards for Reporting Diagnostic accuracy studies 2015 guidelines and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis: checklist for systematic reviews and meta-analyses (TRIPOD-SRMA) were applied.9 10 We used a previously validated search filter for prediction models.11 We combined this filter with terms for meningitis and prediction models and searched for full-text articles in scientific peer-reviewed journals from January 1980 to 1 September 2022 in languages English, German, French, Spanish or Dutch. Prediction models were included if they contained at least three variables obtained from history, physical examination or simple laboratory tests and included children or adults. Publications describing the development, refinement or validation of a prediction model were included. Article screening and data extraction were performed by one researcher (NSG) and doubts were discussed and resolved by a second and third researcher (MCB and MWB). Quality of the reporting of the included studies (that were not previously included) was assessed according to the TRIPOD criteria, containing 6 domains with in total 22 items.12 Each item was scored as reported, reported incompletely and not reported. Characteristics and model performance of studies previously described (published before August 2018) were reported in the online supplemental material, and studies included after August 2018 were reported in the main manuscript.8

Validation cohort

Data from the Paediatric and Adult Causes of Encephalitis and Meningitis (PACEM) study were used for validation of the included prediction models. This was a multicentre prospective study in three hospitals (the Amsterdam University Medical Centers, Onze Lieve Vrouwe Gasthuis in Amsterdam and the Flevoziekenhuis in Almere) in which patients were included if (1) aged 0–18 years old, (2) presented to the emergency department or admitted to the paediatric ward between January 2012 and July 2015 with a suspected CNS infection and (3) cerebrospinal fluid (CSF) examination was performed.13 A detailed description of the cohort was described previously.13

Change in mental status was defined as a Paediatric Glasgow Coma Scale (GCS) <14; coma was defined as a GCS <8.14 Episodes were categorised into six categories regarding final diagnosis: BM, other CNS infections, inflammatory CNS diseases, systemic infections, other neurological diseases and other systemic disease.15 All episodes were independently assessed by two clinicians (NSG and SLS) and discrepancies were discussed and resolved by a third and fourth clinician (MCB and MWB). BM was defined as (1) a positive CSF culture, or (2) a negative CSF culture but positive blood culture and elevated CSF leucocyte count, or (3) a negative CSF and blood culture, elevated CSF leucocyte count, elevated infection parameters in blood (CRP >80 mg/L or leucocytes >14 × 109 /L), clinical parameters suggesting a bacterial infection and the final diagnosis of the treating physician. Age-specific cut-off values for abnormal CSF leucocyte count, protein and glucose were used. In children below 3 months >9 leucocytes/mm3 was considered elevated; in children of 3 months or older >6 leucocytes/mm3 was used as cut-off.1618 CSF protein >1000 mg/L and CSF glucose levels <60% of blood glucose levels were considered abnormal. CSF leucocyte count was corrected for CSF erythrocytes by subtracting one leucocyte for every 700 erythrocytes/mm3.

Statistics

The differences in baseline characteristics between BM and non-BM patients were identified with parametric and non-parametric tests. χ2 tests and Fisher’s exact tests were used to compare categorical outcomes.

The performance of the prediction models was assessed by evaluating discrimination and calibration.1921 The different prediction models were considered as index test; diagnosis of BM was considered reference standard. Discrimination was assessed by calculating the area under the receiver operating characteristic curves (AUC) with 95% confidence intervals (CIs). Calibration was evaluated with the calibration curve, assessing the calibration slope and calculating the calibration in the large. Discriminative ability was categorised as follows: excellent discrimination in case of an AUC of ≥0.90; good discrimination for 0.80≤ AUC <0.90; fair discrimination for 0.70≤ AUC <0.80; and poor discrimination in case of an AUC <0.70.22 When cut-off values for defining high risk of BM had been provided in the original article, sensitivity, specificity and predictive values for the high-risk categories were calculated.

For some of the multivariable logistic regression prediction models, beta coefficients from the original publication were not reported. In this case, we used the observed proportions in the risk categories as reported in the original cohort data, as expected proportions in those risk groups in the validation data.

In case of unclear or missing information about the specifics of the prediction model, authors were contacted for additional information.

In models that reported the complete multivariable logistic regression model, proportions of BM assigned to the different risk categories, defined by the original model, were calculated to display the clinical significance of this spread. A sensitivity analysis in neonates (age <28 days) and in children ≥28 days of age was performed, because of the different presentation of BM at the neonatal age.23

The median number of missing values per variable was 12% (IQR 4%–42%). Missing data were handled by multiple imputation using the R package MICE. We used 51 variables (Supplementary methods) from medical history, physical examination and laboratory results as predictors to impute missing values.24 Missing data were regarded as missing at random and different regression models were used depending on the type of variable: logistic regression for dichotomous variables, polytomous regression for unordered categorical variables with >2 levels and predictive mean matching for numerical variables. For skewed numerical variables and derived variables (e.g., age <7 days), the so-called passive imputation was applied with log or square-root transformation where appropriate. A total of 30 imputation sets were generated based on the observed percentage missingness. If only one predictor from the model was not available in the PACEM dataset, the prediction model was validated without that particular variable. When two or more predictors from the model were not available in our dataset, the model was not validated. If a continuous predictor was missing but the median or mean value (of the original cohort) of that predictor was reported in the original article, this mean/median value was set as observed value in the validation of the prediction model. For discrimination and calibration, we used R packages pROC 2225 and predictABEL.26

We used Rubin’s rule and bootstrapping to estimate proportions and c-statistics based on the imputation sets.27 All statistical tests were two-tailed and p values of <0.05 were considered statistically significant.

Patient and public involvement

None.

Results

Systematic review

Our literature search yielded 7724 articles of which 40 publications on diagnostic prediction models for acute BM were included. In total, 29 publications described the derivation of a total of 32 prediction models and 8 articles described an update, validation or explanation of a model.2865 In total, 20 articles on 23 prediction models were included for validation in our study (online supplemental figure S1). Thirteen publications validated one or more existing models in their dataset (tables1 2).2831 34 43 45 47 49 54 5660 66 All models were based on clinical characteristics and/or laboratory test results from blood and CSF. Characteristics of the derivation cohorts and performance measures of original models published after August 2018 are presented in tables1 2, and models published before August 2018 were described in detail previously, with the exception of one study published in 2011 that was not described before (online supplemental tables S1 and S2).8 45 A total of 23 models were developed in children293133 of which 7 in neonates,34 42 53 64 65 with a median cohort size in the derivation studies of 398 (IQR 158–908) patients. Five models were developed in both adults and children40 50 63 67 and three in adults.32 33 52 The most frequent quality limitations were retrospective derivation cohorts, lack of reporting on handling of missing data and little information about differences in distribution of important variables between the derivation and validation cohort (online supplemental table S3).

Table 1 Characteristics of included prediction models (published after August 2018)

	Population	Modelling method	Items	Cut-off	
Boum et al31	Children 2 months–12 years, history of fever and ≥1 sign of CNS involvement	LRM	Neck stiffness, positive Brudzinski or Kernig sign, bulging fontanel, CSF leucocyte count >100 cells/µL or peripheral neutrophils >10 000 cells/mm3.	BM likely if ≥1 items present	
Chen et al34	Full-term neonates aged ≤28 days with sepsis and an LP	SRA	Fever, absent source of infection, neurological manifestation (seizure, abnormal muscle tone, irritability or bulging anterior fontanelle) CRP >25 mg/L and PCT >2.5 ng/mL, CRP >50 mg/L and PCT>30 ng/mL	High risk if presence of ≥1 items	
Cheng et al64	Preterm infants with an LP	LR	0.1258× PCT level on first day after birth+1.2542 × prenatal glucocorticoid use (yes=1, no=0)–0.1089 × albumin level –0.2557×1 min Apgar –0.1838 × BiPAP days –0.0172 × haemoglobin level on first day after birth –0.8131 × sex (female=2, boy=1)	NR	
Dalai et al65	Neonates with an LP	LRM	−6.42+0.59 if no apnoea+2.05 if no irritability+0.18 if no high-pitched cry+1.01 if no seizures+1.26 if no neutrophilia+0.66 if normal CRP – 0.002 × SNAPPE-II score+1.75 if no leucomalacia	BM unlikely if score 0	
Huang et al41	Term neonates with an LP	LRM	Model 1: −8.47839+0.19469 × CSF WBC+0.00190 × CSF protein+0.01667 × CSF glucose – 0.00503×LDHModel 2 = −6.34939+0.14552 × CSF WBC+0.00069 × CSF protein – 0.05915×CSF glucose.	BM likely if score > −1.7032

BM likely if score > −2.8547

	
Li et al42	Full-term infants aged ≤28 days or premature infants <40 weeks PMA with an LP	SRA	Gender, birth weight, level of glucose in CSF, WBCs in CSF, hsCRP level in blood and LDH level in blood.*	NR	
Mentis et al63	Patients of all ages with meningitis	ML	Group 1 (G1)=CSF neutrophils and CSF lymphocytes; group 2 (G2)=G1 + CSF NLR; group 3 (G3)=G2 + blood albumin; group 4 (G4)=G3 + gender + age; group 5 (G5)=G4 + blood glucose; group 6 (G6)=G5 + blood CRP; group 7 (G7)=G6 + blood soluble urokinase-type plasminogen activator receptor; group 8 (G8)=G7 + lymphocytes-to-blood CRP ratio. Cut-off CSF neutrophils=287 cells/µl, cut-off CSF NLR=2. Other cut-offs not reported.	NR	
Mintegi et al43	Children 29 days–14 years with meningitis	LRM	Serum PCT >1.20 ng/mL = 3 points, CSF protein >80 mg/dL = 2 points, CSF ANC >1000/μL=1 point, serum CRP >40 mg/L = 1 point	BM likely if score ≥1	
Mirkhani et al44	Patients suspected of meningitis	DTA	Rule 1: CSF WBC count >32, CSF protein level >51, CSF glucose level ≤50 Rule 2: CSF WBC>32, CSF protein level>73, CSF glucose level>50	BM likely if all items of rule 1 or 2 are present	
Mwaniki et al45	Infants <90 days of age with meningitis	LRM	History of fever, history of convulsions, bulging fontanel, irritability, axillary temperature ≥39	BM likely if ≥1 item present	
Pelkonen et al49	Infants <90 days of age with an LP	LRM	Age >7 days, weight <2500g, ill>7 days, seizures on admission, signs of shock or prolonged capillary refill and unclear CSF	BM likely if ≥1 items present	
Wang et al53	Full-term neonates 0–28 days with CSF pleiocytosis	LRM	Peripheral blood ANC ≥10×109 cells/L, CSF protein level ≥1650 mg/L, CSF ANC ≥84×106 cells/L, a positive CSF gram stain and a history of seizure before or at the time of presentation.	Low risk if no item present	
* Variables were not clearly specified for this model.

ANC, absolute neutrophil countBiPAP, biphasic positive airway pressure; BM, bacterial meningitis; BMS, Bacterial Meningitis Score; CNS, central nervous system; CRP, C reactive protein; CSF, cerebrospinal fluid; DTA, decision tree algorithm; hsCRP, high-sensitive C reactive protein; LDH, lactate dehydrogenase; LP, lumbar puncture; LR, Lasso regression; LRM, logistic regression model; ML, machine learning; NLR, neutrophil-to-lymphocyte ratio; NR, not reported; PCT, procalcitonin; PMA, post menstrual age; SNAPPE-II, score for acute neonatal physiology and perinatal extension II; SRA, stepwise logistic regression analysis; WBC, white blood cell count

Table 2 Derivation and previous validation of identified prediction models (published after August 2018)

	Population	Source of data	Sample size	Original model performance	Internal/external validation	
Boum et al31	Children 2 months–12 years with a suspected CNS infection, Uganda, 2009–2012	Prospective database cohort, n=459	60 BM	Sens 93.3%, spec 64.4%	NA	
Chen et al34	Full-term neonates aged ≤28 days with sepsis and an LP, China, 2010–2019	Prospective and retrospective database cohort, n=689	102 BM	Sens 96.2%, accuracy 98.9%	Chen et al, 202134 (p, n=383): sens 95.9%, accuracy 98.9%	
Cheng et al64	Preterm infants with an LP, China, 2017–2020	Prospective cohort, n=168	77 BM, 91 AM	Brier score: 0.17, c- slope:0.966, C-index: 0.82 (95% CI 0.75 to 0.89)	NA	
Dalai et al65	Neonates with suspected sepsis and an LP, India, 2014–2016	Prospective cohort, n=300	121 definite/possible BM, 191 no BM	C-statistic: 0.67 (95% CI 0.60 to 0.73),P value <0.001. Accuracy: 66.2%.	NA	
Huang et al41	Term neonates with an LP, China, 2000–2017	Retrospective database cohort, n=1830	105 BM, 1725 AM	1: Sens 95.1%, spec 99.7%, AUC 0.982: Sens 95.1%, spec 99.7%, AUC 0.98	NA	
Li et al42	Full-term infants ≤28 days or premature <40 weeks PMA, with an LP, China, 2012–2018	Retrospective database cohort, n=997	236 BM	Sens 52.97%, spec 96.98%, AUC 0.91	NA	
Mentis et al*63	Patients of all ages with CSF samples at the National Meningitis Reference Laboratory, Greece	Retrospective database cohort, n=4339 cases	1758 BM, 2581 VM	Median sens MLR : 63% (IQR 61%–65%)Median sens RF : 67% (IQR 67%–70%)Median sens NB : 60% (58%–63%)	NA	
Mintegi et al43	Children 29 days–14 years with meningitis, Spain, 2011–2018	Retrospective and prospective database cohort, n=819	61 BM, 758 AM	Sens 100%, spec 83.2%	Mintegi et al, 202043 (p, n=190): Sens 100%, spec 77.4%, AUC=0.986, p<0.0001	
Mirkhani et al44	Individualssuspected of meningitis, Iran, 2009–2011	Retrospective database cohort, n=7945	2219 BM, 2291 AM	Sens 87%, spec 70%, AUC 0.84	NA	
Mwaniki et al45	All admissions aged <60 days with an LP, Kenya, 2001–2007	Prospective database cohort, n=3923	Age <7 days: 31 BMAge 7–59 days: 76 BM	Age <7 days: sens 84%, spec 67% Age 7–59 days: sens 94%, spec 30%	Obiero (Arch) (r, n=4809), sens: 79%, spec 51%;Mwaniki et al, 201145 (p, n=1512): sens 77.8%, spec 55.2%; Pelkonen 2021 (p, n=568): sens 45%, spec 68%	
Pelkonen et al49	Infants <90 days of age with BM or sepsis signs/symptoms with an LP, Angola, 2016–2017	Prospective database cohort, n=1088	212 confirmed BM, 88 probable BM	Sens 97%, spec 16%	NA	
Wang et al53	Full-term neonates 0–28 days with CSF pleiocytosis, China, 2001–2019	Prospective and retrospective database cohort, n=475	94 BM (20%), 381 AM (80%)	Sens 100%, spec 70.9%	NA	
* Only results from patients aged 0–14 years are shown here; medians of the different groups (G1–-G8, table 1) were calculated per statistical model.

AM, aseptic meningitis; AUC, area under the curve; BM, bacterial meningitis; C-index, concordance index; CNS, central nervous system; CSF, cerebrospinal fluid; c-slopecalibration-slopeLP, lumbar puncture; MLR, multiple logistic regression; NA, not applicable; NB, naïve-BayesPMA, postmenstrual age; p, prospective study; r, retrospective study; RF, random forest; sens, sensitivity; spec, specificity

Description of cohort

Between 2012 and 2015, a total of 468 episodes were included, of which 450 episodes could be used in the analysis (table 3). Reasons for exclusion were lack of information in online and paper files (n=14), multiple admissions of one patient in a short timeframe (n=2) and age at admission of 18 years or older (n=2). Included patients were female in 194 out of 450 (43%) cases, and median age at admission was 1.5 months (IQR 0.4–12). A total of 75% of children were <1 year old, 40% of children were <28 days old and 92 of 176 (52%) of neonates were born prematurely. For the analyses, three cohorts were used: the entire cohort of all children (n=450), neonates only (<28 days of age, n=176) and children aged ≥28 days (n=274). The neonate cohort included 16 (9%) BM cases and the children aged ≥28 days included 14 (5%) BM cases.

Table 3 Baseline characteristics validation cohort*

	BM (n=30)	No BM (n=420)	P value	
Female sex	11/30 (37%)	183/420 (44%)	0.57	
Age, months	0.5 (0.1–13.5)	1.6 (0.5–12.1)	0.10	
Prematurity	8/26 (31%)	84/334 (25%)	0.53	
Symptoms <24 hours	15/29 (52%)	212/373 (57%)	0.59	
AB started before LP	6/27 (22%)	28/393 (7%)	0.005	
Otitis media	4/30 (13%)	24/371 (6%)	0.15	
Sinusitis	4/30 (13%)	49/380 (13%)	0.78	
Pneumonia	1/30 (3%)	4/386 (1%)	0.31	
Endocarditis	0/30 (0%)	0/399 (0%)	>0.99	
General symptoms	
 Fever	17/27 (63%)	251/393 (64%)	>0.99	
 Irritability	16/16 (100%)	177/401 (44%)	<0.001	
 Vomiting	6/25 (24%)	81/371 (22%)	0.80	
 Diarrhoea	3/27 (11%)	48/377 (13%)	>0.99	
 Headache	3/21 (14%)	36/318 (11%)	0.72	
 Purpura/petechial rash	4/30 (13%)	10/420 (2%)	<0.001	
Vital signs	
 Heart rate (beats/min)†	161 (±43)	156 (±33)	0.448	
 Systolic BP (mm Hg)‡	93 (±21)	100 (±23)	0.307	
 Diastolic BP (mm Hg)‡	53 (±12)	58 (±19)	0.312	
Neurological symptoms	
 Seizures§	3/30 (10%)	48/420 (11%)	0.82	
 Focal	0/3 (0%)	6/48 (13%)	>0.99	
 Generalised	2/3 (67%)	19/48 (40%)	0.56	
 GCS <14	5/30 (17%)	43/420 (10%)	0.27	
 GCS <8	1/30 (3%)	20/420 (5%)	>0.99	
 Bulging fontanel	0/6 (0%)	6/43 (14%)	>0.99	
 Meningeal irritation	5/15 (33%)	43/234 (18%)	0.15	
 Focal deficits	1/24 (4%)	23/394 (6%)	0.73	
Laboratory findings blood	
 CRP (mg/L)¶	47 (5–155)	8 (2–28)	<0.001	
 Thrombocyte count (×109/L)**	207 (142–289)	279 (211–373)	<0.05	
 Leucocyte count (×109/L)††	16.6 (9.1–21.1)	9.8 (7.1–14.1)	<0.001	
Laboratory findings CSF	
 Leucocyte count (cells/mm3)‡‡	191 (54–2244	3 (1–6)	0.001	
 Protein level (g/L)§§	1.23 (0.6–2.1)	0.40 (0.22–0.74)	 <0.001	
 CSF:blood glucose ratio¶¶	0.51 (±0.30)	0.67 (±0.26)	<0.006	
Data are presented as n/N (%), median (IQR) or mean (±SD). Corrected for CSF erythrocyte count if possible, data on CSF leucocyte count available in 429 patients and data on CSF erythrocyte count available in 271 patients.

* Data are presented as no. of patients/no. of patients in which these data are available. Type of seizure unknown in

† Data available in 331 patients.Data available in , data available in , data available in , data available in , data available in , data available in , data available in .

‡ Data available in 123 patients.Corrected for CSF erythrocyte count if possible, data on CSF leukocyte count available in , data on CSF erythrocyte count available in .

§ Type of seizure unknown in 20/48 (42%) patients.

¶ Data available in 425 patients.

** Data available in 427 patients.

†† Data available in 431 patients.

‡‡ Corrected for CSF erythrocyte count if possible, data on CSF leucocyte count available in 429 patients, data on CSF erythrocyte count available in 271 patients.

§§ Data available in 421 patients.

¶¶ Data available in 265 patients.

AB, antibiotics; BM, bacterial meningitis; BP, blood pressure; CRP, C reactive protein; CSF, cerebrospinal fluid; GCS, Glasgow Coma Scale; LP, lumbar puncture

Symptoms were present <24 hours in 227 of 402 (56%) patients. Most common symptoms were fever in 268 of 420 (64%), irritability in 193 of 429 (45%), meningeal irritation in 48 of 249 (19%) and a decreased level of consciousness 80 in 450 (18%). Median CSF leucocyte count in all children was 4 (IQR 1–9). CSF examination showed elevated leucocytes (corrected for CSF erythrocyte count) in 70 of 258 (27%) patients below 3 months old and in 32 of 166 (19%) patients of 3 months old or older. CSF protein was elevated in 73 of 419 (17%) in all patients and CSF to blood glucose ratio was decreased in 104 of 263 (39%) in all patients.

CNS infection was diagnosed in 74 of 450 (16%) of patients, of which 30 (41%) had BM, 39 (53%) viral meningitis and 5 (7%) infectious encephalitis. Other diagnose categories included CNS inflammatory disease (3%), systemic infection (61%), other neurological disease (14%) and other systemic disease (6%). CSF culture was positive in 10 of 30 patients (30%) clinically diagnosed with BM and showed Streptococcus pneumoniae in 3, S. agalactiae in 3, Neisseria meningitidis in 2, Haemophilus influenzae in 1 and Escherichia coli in 1. Blood culture was positive in 11 of 30 BM patients (37%) and showed S. agalactiae in 4, H. influenza in 1, S. pneumoniae in 2, K. pneumoniae in 1, N. meningitidis in 2 and E. coli in 1. One patient had a positive CSF culture for S. pneumonia and a negative blood culture, one patient had a positive blood culture for K. pneumoniae and a negative CSF culture and in one patient CSF culture failed due to a traumatic lumbar puncture and blood culture was positive for S. agalactiae

Validation of prediction models

We validated 23 prediction models in our cohort: discrimination and calibration could be calculated for 13 models and sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) could be calculated for 20 models. The models of Cheng, Chen, Dalai, Dubos, Freedman, Mintegi and Pelkonen were excluded for validation because these models included two or more predictors that were not available in our PACEM dataset.34 38 39 43 49 64 65 The model of Mentis and the model of Li were not validated because these model were not reported in sufficient detail to perform validation.42 63 In total, 36 (80%) of the total number of 45 predictors from the 25 models were available in our dataset (online supplemental table S4). For the model of Boum, we were unable to assign points for a positive Brudzinski or Kernig sign; however, since the variable neck stiffness was considered a good proxy, this model was validated in our cohort. For the model of Spanos, the logistic regression model was used to calculate the AUC and calibration, and the CSF predictors (Spanos criteria) were used to calculate sensitivity, specificity, NPV and PPV. For the model of Huang, the median of lactate dehydrogenase as observed in the original cohort was imputed to validate the model in our cohort; however, this led to a negligible difference in predictions when compared with leaving out this variable. For the model of Mintegi, three out of a total of seven points could be appointed for procalcitonin, but this value was not available in our dataset and the contribution of this variable to the total model was considered high. Therefore, this model was not validated by using observed values from the original cohort. For the model of Bonsu (2008) and de Cauwer, the predicted probabilities could not be calculated because the betacoefficients of the original modal could not be retrieved. However, observed proportions per risk intervals from the original cohort were available and were used as the expected proportion for validation.30 35 Finally, we could assign no more than two points for duration of symptoms in days for the model of Oostenbrink because these data were not available in our dataset.48 We adjusted the original cut-off for the model of Oostenbrink by reducing the maximum amount of points with the percentage of points that were not available due to the missing values. Moreover, predictive values were calculated for the combined Oostenbrink model only.

All children

Discrimination was excellent in 2 models in all children and good in 6 of 13 models (table 4). The AUCs in these models ranged from 0.69 to 0.94 (median 0.83, IQR 0.79–0.87). The models of Bonsu, Nigrovic, the clinical model of Oostenbrink and the LRM model of Spanos showed an AUC below 0.80, indicating fair discrimination. In all children, the second model of Huang scored best in terms of discrimination with an AUC of 0.94 (CI 0.91 to 0.97). Moreover, sensitivity of Huang was 83% (95% CI 80% to 87%), with 90% specificity (95% CI 87% to 93%), 37% PPV (95% CI 33% to 42%) and 99% NPV (95% CI 99% to 100%). The calibration slopes indicated poor fit of all the models and none of the calibration curves showed reasonable agreement between the predicted and observed probability. Only the model of Boyer and Nigrovic showed a p value of the calibration slope >0.05. Moreover, calibration-in-the-large showed overestimation or underestimation in all of the models (table 4, online supplemental figure S2). As an example, the proportions of BM in low-risk and high-risk groups are shown in online supplemental table S5.

Table 4 Discrimination and calibration for all children

	AUC (95% CI)	Calibration in the large (95% CI)	Calibration slope (95% CI), p value	
Bonsu	0.75 (0.65 to 0.86)	34% (29% to 39%)	Slope: 0.06 (0.03–0.09), p<0.001	
Bonsu 2	0.87 (0.79 to 0.95)	−5% (−8% to −2%)	Slope: 1.6 (1.1–2.0), p<0.001	
Boyer	0.83 (0.72 to 0.94)	2% (−1% to 6%)	Slope: 0.9 (0.6–1.2), p=0.34	
Chavanet—adults	0.88 (0.80 to 0.95)	NA	NA	
Chavanet—children	0.82 (0.73 to 0.90)	NA	NA	
De Cauwer	0.85 (0.78 to 0.91)	16% (12% to 21%)	Slope: 1.9 (1.3 to 2.6), p<0.001	
Hoen	0.84 (0.76 to 0.91)	11% (7% to 15%)	Slope: 0.07 (0.04 to 0.11), p<0.001	
Huang—model 1	0.94 (0.90 to 0.97)	NA	NA	
Huang—model 2	0.94 (0.91 to 0.97)	NA	NA	
Nigrovic	0.79 (0.71 to 0.86)	18% (13% to 22%)	Slope: 1.4 (0.9 to 1.9), p=0.11	
Oostenbrink clinical	0.69 (0.57 to 0.81)	24% (19% to 29%)	Slope: 0.14 (0.09 to 0.2), p<0.001	
Oostenbrink CSF	0.77 (0.66 to 0.87)	12% (8% to 17%)	Slope: 0.68 (0.43 to 0.93), p<0.001	
Spanos LRM	0.74 (0.64 to 0.84)	20% (15% to 25%)	Slope: 0.4 (0.3 to 0.5), p<0.001	
AUC, area under the curve; CSF, cerebrospinal fluid; LRM, logistic regression modelNA, not applicable

Median sensitivity of the 20 models was 80% (IQR 68%–91%) overall (table 5). NPV was ≥99% in 3/20 (15%) models overall.30 35 46 None of the models showed a sensitivity and NPV of 100% in all children.

Table 5 Sensitivity, specificity and predictive values for all children

	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	
Bonsu	92% (89% to 94%)	20% (16% to 24%)	8% (5% to 11%)	97% (95% to 99%)	
Bonsu 2	90% (89% to 93%)	60% (55% to 64%)	14% (11% to 17%)	99% (97% to 100%)	
Boyer	87% (84% to 90%)	60% (56% to 65%)	14% (10% to 17%)	98% (97% to 100%)	
Boum	83% (79% to 87%)	35% (30% to 39%)	8% (6% to 11%)	97% (95% to 98%)	
Brivet	94% (92% to 97%)	11% (8% to 14%)	7% (5% to 9%)	96% (95% to 98%)	
Chavanet—adults	71% (66% to 75%)	91% (89% to 94%)	37% (32% to 41%)	98% (96% to 99%)	
Chavanet—children	77% (73% to 81%)	80% (76% to 84%)	22% (18 to 26%)	98% (96% to 99%)	
De Cauwer	97% (95% to 98%)	50% (45% to 54%)	12% (9% to 15%)	100% (99% to 100%)	
Deivanayagam	24% (20% to 28%)	100% (99% to 100%)	85% (81% to 88%)	95% (93 t%o 97%)	
Hoen	76% (71% to 80%)	77% (74% to 81%)	19% (16% to 23%)	98% (96% to 99%)	
Huang—model 1	80% (76% to 84%)	91% (88% to 94%)	39% (35% to 44%)	98% (97% to 100%)	
Huang—model 2	83% (80% to 87%)	90% (87% to 93%)	37% (33% to 42%)	99% (99% to 100%)	
Mirkhani—model 1	33% (29% to 38%)	97% (95% to 99%)	44% (40% to 49%)	95% (93% to 97%)	
Mirkhani—model 2	30% (26% to 34%)	99% (98% to 100%)	61% (56% to 66%)	95% (93% to 97%)	
Mwaniki	95% (93% to 97%)	14% (11% to 17%)	7% (5% to 10%)	97% (96 %to 99%)	
Nigrovic	94% (91% to 96%)	51% (46% to 56%)	12% (9% to 15%)	99% (98% to 100%)	
Oostenbrink combined*	74% (69% to 80%)	44% (40% to 50%)	9% (6% to 12%)	96% (94% to 98%)	
Spanos CSF predictors	54% (49% to 59%)	92% (89% to 94%)	32% (27% to 36%)	97% (95% to 98%)	
Tokuda	58% (52% to 64%)	73% (69% to 78%)	14% (10% to 18%)	96% (94% to 98%)	
Wang	80% (76% to 83%)	60% (56% to 65%)	12% (9% to 16%)	98% (96% to 99%)	
* Adjusted cut-off.

CSF, cerebrospinal fluid NPV, negative predictive value; PPV, positive predictive value

Median specificity was 67% (IQR 49%–91%) overall. Highest specificity was reached by the model of Deivanayagam (100%, 95% CI 99% to 100%) and the second model of Mirkhani (99%, 95% CI 98% to 100%) overall. Sensitivity of these models was only 24% (95% CI 20% to 28%) and 30% (95% CI 26% to 34%), respectively. Performance of models that were originally developed in children or neonates (n=13) differed from models developed in adults (n=3) or both children and adults (n=4), with median sensitivity in child models of 83% (IQR 80%–92%) compared with 58% (IQR 44%–74%) in (child and) adult models, and median specificity of 60% (IQR 44%–80%) in child models compared with 91% (IQR 75%–95%) in models developed in adults or both. The combination of sensitivity and specificity was best in the models of de Cauwer (resp. 97%, 50%) and Nigrovic (resp. 94%, 51%), and both showed an NPV of ≥99%.

Neonates

Discrimination was excellent in 2 models in neonates and good in 4 out of 13 models. The AUCs ranged from 0.58 to 0.91 (median 0.79, IQR 0.75–0.82, online supplemental table S6). Calibration in the large showed overestimation or underestimation in all of the models validated in the neonate cohort. Median sensitivity of the 20 models was 78% (IQR 59%–90%, online supplemental table S7). None of the models showed a sensitivity of 100%. Median specificity was 70% (IQR 45%–88%) in neonates. Only the model of Deivanayagam showed a specificity of 100% (95% CI 99% to 100%), with a sensitivity of 17% (95% CI 11% to 23%). Models developed in neonates (n=3, median sensitivity 88% (IQR 81%–88%), median specificity 86% (IQR 67%–88%)) overall performed better compared with models that were developed in children or adults or both (n=17, median sensitivity 73% (IQR 53%–91%), median specificity 52% (IQR 43%–88%)) in this cohort of neonates.

Children ≥28 days of age

Discrimination was excellent in six models and good in six models in children ≥28 days of age (online supplemental table S8). The AUCs ranged from 0.74 to 0.96 (median 0.89, IQR 0.82–0.92). Calibration in the large showed overestimation or underestimation in all of the models; however, the CSF model of Oostenbrink and the model of Boyer showed reasonable calibration with a slope of 1.1 (95% CI 0.6 to 1.5) and 1.0 (95% CI 0.5 to 1.4), respectively. Median sensitivity of the models was 82% (IQR 70%–93%). One model (De Cauwer) showed a 100% sensitivity, however with a specificity of 50% (IQR 44%–56%, online supplemental table S9). Moreover, nine models in this cohort showed a NPV of 99% or higher. Median specificity was 70% (IQR 48%–92%) in this cohort.

Models developed in children ≥28 days of age, adults or both (n=17, median sensitivity 85% (IQR 64–93), median specificity 65% (IQR 40–92)) overall showed higher sensitivity but lower specificity compared with models developed in neonates (n=3, median sensitivity 79% (IQR 75–84), median specificity 92% (IQR 74–92)) in this cohort of children aged ≥28 days of age.

Discussion

We validated 23 clinical and laboratory-based diagnostic prediction models for bacterial meningitis, identified in a systematic review, using a cohort of 450 children with a suspected CNS infection. Quality of the included studies varied widely regarding study design, statistical analyses and reporting on model-building procedures. Discrimination was excellent in two models and good in 6 out of 13 models (4 out of 13 in neonates). Calibration showed relevant overestimation or underestimation of BM by all models. However, a sensitivity of 100% is required for implementation in clinical care due to the devastating consequences of missing this disease. Therefore, the results of these prediction models should always be incorporated with all information from the patient’s history, physical examination and ancillary investigations, and although some of these models provide valuable diagnostic information, they should not be used as a stand-alone test.

Children models performed worse in our children cohort compared with the adult cohort in which they were validated previously.8 Moreover, all models validated in this study performed worse than in their original publication. This is likely largely due to differences in case-mix between our validation cohort and the original derivation cohorts. Patient’s age differed significantly between the derivation cohorts, ranging from models derived in preterm neonates only to patients of all ages or adults, whereas our validation cohort consisted of children aged 0–18 years old. Even though discrimination overall was good to excellent, we show calibration was poor, which can be expected and can have many causes. Often, patient characteristics change over time and disease incidence or prevalence rates can vary between different populations, healthcare settings and countries. When an algorithm is developed in a cohort with a high incidence of BM, it may systematically give overestimated risk estimates (poor calibration in the large) when used in a setting with a lower BM incidence, as is the case in this study. Moreover, BM symptoms can vary greatly between different age groups. For example, neonates often show very non-specific signs and symptoms in meningitis, resulting in a different effect of clinical predictors and therefore poorer calibration slopes when validating a clinical characteristics-based models in this population. Other common causes of reduced discrimination and calibration are related to methodological problems regarding the algorithm itself, such as statistical overfitting. When using these models in new populations, it is important to take into account that the incidence of the outcome in the new population, as well as the different effect of predictors in this new population, will result in varying calibration, depending on the population in which the model is used. Before applying a model to a new population, recalibration of the model will sometimes be needed to adjust for these differences.

We chose to validate the prediction models in a broad cohort of patients with a suspected CNS infection because this is the population in which these models will be used in daily practice Therefore, validation in such a cohort provides a realistic view. Moreover, it shows that these models cannot be directly applied to all new populations without validating and adjusting the model to the specific characteristics of the population accordingly.

The question also remains if machine-learning-based algorithms outperform clinical judgement. A systematic review on comparing diagnostic prediction models with clinical judgement for various medical conditions found that prediction models reduced the proportion of missed diagnoses in only 2 out of 46 publications.68 This was offset by a larger amount of false positives as well. Comparing the combination of clinical judgement assisted by prediction rules to clinical judgement alone would provide the most valuable information on the added value of prediction models on patient outcome, but studies on this topic are lacking thus far.

To date, a large amount of prediction models for BM have been developed but none showed excellent discrimination and calibration when validated in a broader population of all patients suspected of a CNS infection.

However, comparing the discriminative performance of prediction models for BM to an ideal diagnostic test with excellent discrimination might not be fair, since a diagnostic test for paediatric BM with 100% sensitivity and 100% specificity does not exist in clinical practice. All current tests show limitations that should be taken into account when assessing the results of an individual patient. Diagnostic prediction models could aid in addition to other diagnostic investigations, however should not be used on their own.

Future research should also focus on different ways of improving diagnosis in paediatric BM. Better biomarker-based point-of-care tests that can accurately exclude and include BM in children are needed, especially in complex cases in which definite diagnosis is still unclear after conventional CSF examination.

Our research has some limitations. First, some models included variables that were not available in our dataset. These models were validated without those variables, which could lead to difference in performance. Second, a substantial proportion of data were missing and had to be imputed. Although 30 different imputation sets were used, this could have led to some distortion of the performance measurements. Third, the number of patients with BM in our validation cohort was limited. Our validation cohort included 450 patients, including 30 patients with BM (7%), possibly leading to less reliable and generalisable results. When including a small number of cases, the findings are more susceptible to random variation. Because CIs were broad, performance in larger cohorts could find better performance. Last, 18 of the total of 468 (4%) episodes could not be included due to lack of data on predictor or outcome variables. Lack of data in the paper files of these episodes could be a result of an acute setting with an ill patient resulting in less time for data documentation, possibly influencing the amount of severely ill (BM) patients.

In conclusion, this review analysed 40 articles on diagnostic prediction models for BM in children and validated 23 prediction models in a multicentre prospective cohort of 450 children suspected of a CNS infection. The models showed good to excellent diagnostic accuracy with poor calibration in all models. Diagnostic prediction models could be of help in the diagnostic workup of paediatric BM but are not recommended to use on their own in routine individual patient care. Future research should focus on the added value of prediction models in clinical practice.

supplementary material

10.1136/bmjopen-2023-081172 online supplemental file 1

Acknowledgements

This work was previously presented at the European Congress of Clinical Microbiology & Infectious Diseases (Copenhagen 2023). We thank all participating physicians in the Netherlands for their cooperation.

Data availability statement

Data are available upon reasonable request.

Review Process File
7 8 2024

Funding: This work was supported by the Netherlands Organization for Health Research and Development (ZonMw; NWO-Vidi Grant [grant number 917.17.308 to MCB], NWO-Vici-Grant [grant number 918.19.627 to DvdB]); the Academic Medical Centre (AMC Fellowship to DvdB); the European Research Council (ERC Consolidator grant to MCB, 101001237, ERC Starting grant to D.B.) and Stichting de Merel (seed grant to MCB).

Prepublication history and additional supplemental material for this paper are available online. To view these files, please visit the journal online (https://doi.org/10.1136/bmjopen-2023-081172).

Provenance and peer review: Not commissioned; externally peer reviewed.

Patient consent for publication: Not applicable.

Ethics approval: This research involves human participants and was approved by the Medical Ethical Review Committee of the Amsterdam University Medical Centres. All participants or participants’ caregivers provided written informed consent prior to the start of this study. Participants gave informed consent to participate in the study before taking part.

Data availability free text: Requests to share the review protocol, template data collection forms, data extracted from included studies, data used for analysis, analytical code or anonymous individual patient data for scientific research can be made to the corresponding author.

Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
==== Refs
References

1 Baraff LJ Lee SI Schriger DL Outcomes of bacterial meningitis in children: a meta-analysis Pediatr Infect Dis J 1993 12 389 94 10.1097/00006454-199305000-00008 8327300
2 Rahimi BA Ishaq N Mudaser GM et al Outcome of acute bacterial meningitis among children in Kandahar, Afghanistan: A prospective observational cohort study PLoS One 2022 17 e0265487 10.1371/journal.pone.0265487 35404980
3 Chang C-J Chang W-N Huang L-T et al Bacterial meningitis in infants: the epidemiology, clinical features, and prognostic factors Brain Dev 2004 26 168 75 10.1016/S0387-7604(03)00122-0 15030905
4 van de Beek D Brouwer MC Koedel U et al Community-acquired bacterial meningitis Lancet 2021 398 1171 83 10.1016/S0140-6736(21)00883-7 34303412
5 Willems J Hermans E Schelstraete P et al Optimizing the use of antibiotic agents in the pediatric intensive care unit: A narrative review Paediatr Drugs 2021 23 39 53 10.1007/s40272-020-00426-y 33174101
6 van de Beek D Cabellos C Dzupova O et al ESCMID guideline: diagnosis and treatment of acute bacterial meningitis Clin Microbiol Infect 2016 22 S37 62 10.1016/j.cmi.2016.01.007 27062097
7 Runde TJ Anjum F Hafner JW Bacterial meningitis Treasure Island, FL StatPearls Publishing 2022
8 van Zeggeren IE Bijlsma MW Tanck MW et al Systematic review and validation of diagnostic prediction models in patients suspected of meningitis J Infect 2020 80 143 51 10.1016/j.jinf.2019.11.012 31794775
9 Bossuyt PM Reitsma JB Bruns DE et al STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies BMJ 2015 351 h5527 10.1136/bmj.h5527 26511519
10 Snell KIE Levis B Damen JAA et al Transparent reporting of multivariable prediction models for individual prognosis or diagnosis: checklist for systematic reviews and meta-analyses (TRIPOD-SRMA) BMJ 2023 381 e073538 10.1136/bmj-2022-073538 37137496
11 Geersing G-J Bouwmeester W Zuithoff P et al Search filters for finding prognostic and diagnostic prediction studies in medline to enhance systematic reviews PLoS One 2012 7 e32844 10.1371/journal.pone.0032844 22393453
12 Moons KGM Altman DG Reitsma JB et al New guideline for the reporting of studies developing, validating, or updating a multivariable clinical prediction model: the TRIPOD statement Adv Anat Pathol 2015 22 303 5 10.1097/PAP.0000000000000072 26262512
13 de Blauw D Bruning AHL Wolthers KC et al Incidence of childhood meningoencephalitis in children with a suspected meningoencephalitis in the Netherlands Pediatr Infect Dis J 2022 41 290 6 10.1097/INF.0000000000003441 34966139
14 van de Beek D de Gans J Spanjaard L et al Clinical features and prognostic factors in adults with bacterial meningitis N Engl J Med 2004 351 1849 59 10.1056/NEJMoa040845 15509818
15 Khatib U van de Beek D Lees JA et al Adults with suspected central nervous system infection: A prospective study of diagnostic accuracy J Infect 2017 74 1 9 10.1016/j.jinf.2016.09.007 27746160
16 Kestenbaum LA Ebberson J Zorc JJ et al Defining cerebrospinal fluid white blood cell count reference values in neonates and young infants Pediatrics 2010 125 257 64 10.1542/peds.2009-1181 20064869
17 Thomson J Sucharew H Cruz AT et al Cerebrospinal fluid reference values for young infants undergoing lumbar puncture Pediatrics 2018 141 e20173405 10.1542/peds.2017-3405 29437883
18 Zimmermann P Curtis N Bacterial meningitis in the absence of pleocytosis in children: A systematic review Pediatr Infect Dis J 2021 40 582 7 10.1097/INF.0000000000003085 33591075
19 Toll DB Janssen KJM Vergouwe Y et al Validation, updating and impact of clinical prediction rules: a review J Clin Epidemiol 2008 61 1085 94 10.1016/j.jclinepi.2008.04.008 19208371
20 Steyerberg EW Vickers AJ Cook NR et al Assessing the performance of prediction models: a framework for traditional and novel measures Epidemiology 2010 21 128 38 10.1097/EDE.0b013e3181c30fb2 20010215
21 Debray TPA Vergouwe Y Koffijberg H et al A new framework to enhance the interpretation of external validation studies of clinical prediction models J Clin Epidemiol 2015 68 279 89 10.1016/j.jclinepi.2014.06.018 25179855
22 Muller MP Tomlinson G Marrie TJ et al Can routine laboratory tests discriminate between severe acute respiratory syndrome and other causes of community-acquired pneumonia? Clin Infect Dis 2005 40 1079 86 10.1086/428577 15791504
23 Ku LC Boggess KA Cohen-Wolkowiez M Bacterial meningitis in infants Clin Perinatol 2015 42 29 45 10.1016/j.clp.2014.10.004 25677995
24 van Buuren S Multiple imputation of discrete and continuous data by fully conditional specification Stat Methods Med Res 2007 16 219 42 10.1177/0962280206074463 17621469
25 Robin X Turck N Hainard A et al PROC: an open-source package for R and S+ to analyze and compare ROC curves BMC Bioinformatics 2011 12 77 10.1186/1471-2105-12-77 21414208
26 Kundu S Aulchenko YS van Duijn CM et al PredictABEL: an R package for the assessment of risk prediction models Eur J Epidemiol 2011 26 261 4 10.1007/s10654-011-9567-4 21431839
27 How many imputations? stefvanbuuren Available https://stefvanbuuren.name/fimd/sec-howmany.html
28 Abdelrahim NA Fadl-Elmula IM Ali HM Bacterial meningitis in sudanese children; critical evaluation of the clinical decision using clinical prediction rules BMC Pediatr 2019 19 319 10.1186/s12887-019-1684-3 31492124
29 Bonsu BK Harper MB Differentiating acute bacterial meningitis from acute viral meningitis among children with cerebrospinal fluid pleocytosis: a multivariable regression model Pediatr Infect Dis J 2004 23 511 7 10.1097/01.inf.0000129689.58211.9e 15194831
30 Bonsu BK Ortega HW Marcon MJ et al A decision rule for predicting bacterial meningitis in children with cerebrospinal fluid pleocytosis when gram stain is negative or unavailable Acad Emerg Med 2008 15 437 44 10.1111/j.1553-2712.2008.00099.x 18439199
31 Boum Y Nyehangane D Llosa AE et al Utility of clinical and laboratory decision rules in identifying bacterial meningitis among children with suspicion of central nervous system infections in a malaria-endemic area, mbarara, uganda Pediatr Infect Dis J 2019 38 e131 3 10.1097/INF.0000000000002185 30199480
32 Brivet FG Ducuing S Jacobs F et al Accuracy of clinical presentation for differentiating bacterial from viral meningitis in adults: a multivariate approach Intensive Care Med 2005 31 1654 60 10.1007/s00134-005-2811-1 16244879
33 Chavanet P Schaller C Levy C et al Performance of a predictive rule to distinguish bacterial and viral meningitis J Infect 2007 54 328 36 10.1016/j.jinf.2006.06.009 16887191
34 Chen Y Yin Z Gong X et al A sequential guide to identify neonates with low bacterial meningitis risk: A multicenter study Ann Clin Transl Neurol 2021 8 1132 40 10.1002/acn3.51356 33836125
35 De Cauwer HG Eykens L Hellinckx J et al Differential diagnosis between viral and bacterial meningitis in children Eur J Emerg Med 2007 14 343 7 10.1097/MEJ.0b013e328270366b 17968200
36 Deivanayagam N Ashok TP Nedunchelian K et al Evaluation of CSF variables as a diagnostic test for bacterial meningitis J Trop Pediatr 1993 39 284 7 10.1093/tropej/39.5.284 8271335
37 Delannoy Q Pean-de-Ponfilly G Mesnil C et al Validation of the bacterial meningitis score in adults consulting at an emergency department: a retrospective multicentric study Eur J Emerg Med 2020 27 447 53 10.1097/MEJ.0000000000000721 32496282
38 Dubos F Moulin F Raymond J et al Distinction between bacterial and aseptic meningitis in children: refinement of a clinical decision rule Arch Pediatr 2007 14 434 8 10.1016/j.arcped.2006.12.009 17258439
39 Freedman SB Marrocco A Pirie J et al Predictors of bacterial meningitis in the era after haemophilus influenzae Arch Pediatr Adolesc Med 2001 155 1301 6 10.1001/archpedi.155.12.1301 11732947
40 Hoen B Viel JF Paquot C et al Multivariate approach to differential diagnosis of acute meningitis Eur J Clin Microbiol Infect Dis 1995 14 267 74 10.1007/BF02116518 7649188
41 Huang H Tan J Gong X et al Comparing single vs. combined cerebrospinal fluid parameters for diagnosing full-term neonatal bacterial meningitis Front Neurol 2019 10 12 10.3389/fneur.2019.00012 30728800
42 Li H Xiao R Javed R et al Evaluation of cerebrospinal fluid and blood parameters finding in early diagnosis and drug therapy of suspected bacterial meningitis in neonates J Res Med Sci 2020 25 77 10.4103/jrms.JRMS_470_19 33088314
43 Mintegi S García S Martín MJ et al Clinical prediction rule for distinguishing bacterial from aseptic meningitis Pediatrics 2020 146 e20201126 10.1542/peds.2020-1126 32843440
44 Mirkhani A Nabavi M Roshan Poor A et al Using predictive model for screening bacterial meningitis in national surveillance system in Iran Stud Health Technol Inform 2018 251 141 4 29968622
45 Mwaniki MK Talbert AW Njuguna P et al Clinical indicators of bacterial meningitis among neonates and young infants in rural Kenya BMC Infect Dis 2011 11 301 10.1186/1471-2334-11-301 22044635
46 Nigrovic LE Kuppermann N Malley R Development and validation of a multivariable predictive model to distinguish bacterial from aseptic meningitis in children in the post-haemophilus influenzae era Pediatrics 2002 110 712 9 10.1542/peds.110.4.712 12359784
47 Obiero CW Mturi N Mwarumba S et al Clinical features to distinguish meningitis among young infants at a rural kenyan hospital Arch Dis Child 2021 106 130 6 10.1136/archdischild-2020-318913 32819909
48 Oostenbrink R Moons KGM Derksen-Lubsen AG et al A diagnostic decision rule for management of children with meningeal signs Eur J Epidemiol 2004 19 109 16 10.1023/b:ejep.0000017828.13995.76 15074565
49 Pelkonen T Urtti S Cardoso O et al Accuracy of clinical and cerebrospinal fluid indicators in the diagnosis of bacterial meningitis in infants <90 days of age in Luanda, Angola Pediatr Infect Dis J 2021 40 e462 5 10.1097/INF.0000000000003305 34561386
50 Spanos A Harrell FE Durack DT Differential diagnosis of acute meningitis. an analysis of the predictive value of initial observations JAMA 1989 262 2700 7 2810603
51 Thome J Bovier-Lapierre M Vercherat M et al Bacterial or viral meningitis? study of a numerical score permitting an early etiologic orientation in meningitis difficult to diagnose Pediatrie (Bucur) 1980 35 225 36 7393689
52 Tokuda Y Koizumi M Stein GH et al Identifying low-risk patients for bacterial meningitis in adult patients with acute meningitis Intern Med 2009 48 537 43 10.2169/internalmedicine.48.1832 19336955
53 Wang Y Lei X Zhao Y et al An improved clinical prediction rule for identifying neonatal bacterial meningitis: a multicenter cohort study Transl Pediatr 2021 10 64 72 10.21037/tp-20-255 33633938
54 Baty V Viel JF Schuhmacher H et al Prospective validation of a diagnosis model as an aid to therapeutic decision-making in acute meningitis Eur J Clin Microbiol Infect Dis 2000 19 422 6 10.1007/s100960000287 10947216
55 Bonsu BK Harper MB Explanation of mathematical model Pediatr Infect Dis J 2004 23 893 10.1097/01.inf.0000137586.42248.00 15361741
56 Dubos F Korczowski B Aygun DA et al Distinguishing between bacterial and aseptic meningitis in children: european comparison of two clinical decision rules Arch Dis Child 2010 95 963 7 10.1136/adc.2010.186056 20660523
57 Dubos F Lamotte B Bibi-Triki F et al Clinical decision rules to distinguish between bacterial and aseptic meningitis Arch Dis Child 2006 91 647 50 10.1136/adc.2005.085704 16595647
58 Leblebicioğlu H Esen S Bedir A et al The validity of spanos’ and hoen’s models for differential diagnosis of meningitis Eur J Clin Microbiol Infect Dis 1996 15 252 4 10.1007/BF01591365 8740864
59 McKinney WP Heudebert GR Harper SA et al Validation of a clinical prediction rule for the differential diagnosis of acute meningitis J Gen Intern Med 1994 9 8 12 10.1007/BF02599135 8133355
60 Nigrovic LE Malley R Kuppermann N Meta-analysis of bacterial meningitis score validation studies Arch Dis Child 2012 97 799 805 10.1136/archdischild-2012-301798 22764093
61 Oostenbrink R Moons KG Donders AR et al Prediction of bacterial meningitis in children with meningeal signs: reduction of lumbar punctures Acta Paediatr 2001 90 611 7 11440091
62 Oostenbrink R Moons KGM Twijnstra MJ et al Children with meningeal signs: predicting who needs empiric antibiotic treatment Arch Pediatr Adolesc Med 2002 156 1189 94 10.1001/archpedi.156.12.1189 12444828
63 Mentis A-FA Garcia I Jiménez J et al Artificial intelligence in differential diagnostics of meningitis: A nationwide study Diagnostics (Basel) 2021 11 602 10.3390/diagnostics11040602 33800653
64 Cheng X Zhang Q Fu Z et al Establishment of a predictive model for purulent meningitis in preterm infants Transl Pediatr 2022 11 1018 27 10.21037/tp-22-236 35800267
65 Dalai R Dutta S Pal A et al Is lumbar puncture avoidable in low-risk neonates with suspected sepsis? Am J Perinatol 2022 39 99 105 10.1055/s-0040-1714397 32693413
66 Jaeger F Leroy J Duchêne F et al Validation of a diagnosis model for differentiating bacterial from viral meningitis in infants and children under 3.5 years of age Eur J Clin Microbiol Infect Dis 2000 19 418 21 10.1007/s100960000292 10947215
67 Mirkhani A Roshanpoor A Pournik O et al Developing a prognostic model to predict mortality in patients with acute bacterial meningitis Stud Health Technol Inform 2021 281 774 8 10.3233/SHTI210280 34042683
68 Sanders S Doust J Glasziou P A systematic review of studies comparing diagnostic clinical prediction rules with clinical judgment PLoS One 2015 10 e0128233 10.1371/journal.pone.0128233 26039538
