
==== Front
JMIR Bioinform Biotech
JMIR Bioinform Biotech
JBB
JMIR Bioinformatics and Biotechnology
2563-3570
JMIR Publications Toronto, Canada

v3i1e38845
10.2196/38845
Original Paper
Original Paper
Reducing Crowding in Emergency Departments With Early Prediction of Hospital Admission of Adult Patients Using Biomarkers Collected at Triage: Retrospective Cohort Study
Mavragani Amaryllis
Ma Xiaomeng
Nagavally Sneha
Mortey Kingsley
Musawir Haseeb
Monahan Ann Corneille MSHI, PhD https://orcid.org/0000-0002-2536-2230
1University of Alabama at Birmingham 1720 University Blvd Birmingham, AL, 35294 United States 1 2056174780 monahanannc@gmail.com

Feldman Sue S MEd, RN, PhD 2https://orcid.org/0000-0002-1173-3993

Fitzgerald Tony P ScD 34https://orcid.org/0000-0001-6437-5686

1 University of Alabama at Birmingham Birmingham, AL United States
2 Department of Health Services Administration University of Alabama at Birmingham Birmingham, AL United States
3 School of Mathematical Sciences University College Cork Cork Ireland
4 School of Public Health University College Cork Cork Ireland
Corresponding Author: Ann Corneille Monahan monahanannc@gmail.com
Jan-Dec 2022
13 9 2022
3 1 e3884521 4 2022
31 5 2022
5 7 2022
17 7 2022
©Ann Corneille Monahan, Sue S Feldman, Tony P Fitzgerald. Originally published in JMIR Bioinformatics and Biotechnology (https://bioinform.jmir.org), 13.09.2022.
2022
https://creativecommons.org/licenses/by/4.0/ This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Bioinformatics and Biotechnology, is properly cited. The complete bibliographic information, a link to the original publication on https://bioinform.jmir.org/, as well as this copyright and license information must be included.

Background

Emergency department crowding continues to threaten patient safety and cause poor patient outcomes. Prior models designed to predict hospital admission have had biases. Predictive models that successfully estimate the probability of patient hospital admission would be useful in reducing or preventing emergency department “boarding” and hospital “exit block” and would reduce emergency department crowding by initiating earlier hospital admission and avoiding protracted bed procurement processes.

Objective

To develop a model to predict imminent adult patient hospital admission from the emergency department early in the patient visit by utilizing existing clinical descriptors (ie, patient biomarkers) that are routinely collected at triage and captured in the hospital’s electronic medical records. Biomarkers are advantageous for modeling due to their early and routine collection at triage; instantaneous availability; standardized definition, measurement, and interpretation; and their freedom from the confines of patient histories (ie, they are not affected by inaccurate patient reports on medical history, unavailable reports, or delayed report retrieval).

Methods

This retrospective cohort study evaluated 1 year of consecutive data events among adult patients admitted to the emergency department and developed an algorithm that predicted which patients would require imminent hospital admission. Eight predictor variables were evaluated for their roles in the outcome of the patient emergency department visit. Logistic regression was used to model the study data.

Results

The 8-predictor model included the following biomarkers: age, systolic blood pressure, diastolic blood pressure, heart rate, respiration rate, temperature, gender, and acuity level. The model used these biomarkers to identify emergency department patients who required hospital admission. Our model performed well, with good agreement between observed and predicted admissions, indicating a well-fitting and well-calibrated model that showed good ability to discriminate between patients who would and would not be admitted.

Conclusions

This prediction model based on primary data identified emergency department patients with an increased risk of hospital admission. This actionable information can be used to improve patient care and hospital operations, especially by reducing emergency department crowding by looking ahead to predict which patients are likely to be admitted following triage, thereby providing needed information to initiate the complex admission and bed assignment processes much earlier in the care continuum.

emergency care
prehospital
emergency
information system
crowding
boarding
exit block
medical informatics
application
health service research
personalized medicine
predictive medicine
model
probabilistic
polynomial model
decision support technique
decision support
evidence-based health care
management information systems
algorithm
machine learning
model
predict
risk
==== Body
pmcIntroduction

Overview

The problem of emergency department (ED) crowding is well known in health care as a complex, multi-dimensional problem that threatens patient safety and care quality and has remained largely unresolved for over 20 years. Despite ED efficiency interventions [1,2] and government policy [3] aimed at reducing crowding, it continues to threaten patient safety and contribute to poor patient outcomes [4-6]. ED crowding occurs when ED demand exceeds the staff’s ability to provide quality care in a reasonable time frame [7,8]. The main causes of crowding are ED “boarding” [9-12] (eg, when an ED bed is occupied by a patient due to be admitted to the hospital, but the patient remains in the ED because no inpatient bed has been assigned) and hospital “exit block” [6] (eg, when patients are delayed or blocked from transitioning out of the ED and into the hospital in a reasonable time frame).

Although some recent literature has attributed ED boarding to insufficient hospital bed capacity [13-17], this description of the situation belies boarding’s complex roots and suggests that hospitals simply do not have inpatient beds available because they are all occupied by patients. In fact, this is rarely the case, as occupancy rates in most US hospitals average 40% for rural hospitals and 65% for urban hospitals [18-20]; these rates have been slowly declining for decades [18-21]. Instead, insufficient bed capacity in most hospitals refers to a shortage of available beds for ED admission. Reasons for this “shortage” include existing bed reservations, which can be for elective surgery patients who might require admission [9,10], for transfer patients from other hospitals, and for geographic bed plans that assign beds to specialties (eg, orthopedics) to keep relevant patients and providers close together [14]. There are positive logistical and care-quality reasons for these bed reservations, but there are also financial reasons that may benefit the hospital yet contribute to ED boarding. For example, reserving beds for highly reimbursable elective procedures that might not be utilized [9], instead of opening the beds for the immediate needs of ED admissions, increases boarding.

Securing hospital beds for ED patients is a time-intensive, interdepartmental negotiation requiring multiple approvals before an ED patient can be transitioned into an inpatient hospital bed. Larger hospitals have bed managers dedicated to effectively utilizing each hospital bed and the patient support services each requires. Much like air traffic controllers, bed managers are the conductors of a complex series of interdependent processes and activities. Bed management involves assessing bed availability throughout the hospital, assessing whether unit resources are in place to enable a particular bed to be filled by a particular patient, identifying additional unit resources that are required to fill a particular bed, determining whether sufficient resources are available to care for specialty patients (eg, cardiac patients) in a general medicine unit, identifying the required resources and available staff (eg, who is at the hospital and who is on call), identifying which beds are reserved for urgent postoperation surgical cases and which are reserved for elective surgeries, and possessing knowledge of matters spanning multiple departments with a multitude of players. This complex and important process may be unnecessarily convoluted in hospitals that have grown in size and responsibility and have become incongruent with effective organizational management. Examinations of clinical workflows for admitting hospital patients from the ED have revealed processes layered with cultural and organizational factors that exacerbate an already inefficient process. There can be 50 to 75 steps between a bed request and time to admission orders, and staff have reported they believe the process is excessively complex, redundant, and in some respects, unsafe [22]. The earlier bed managers have information about a patient who will likely be admitted, the earlier they can begin the bed assignment process, and the earlier the patient can move out of the ED to a hospital bed. This often dysfunctional and protracted hospital transition and bed assignment process blocks patients from transitioning out of the ED to inpatient hospital care (ie, exit block), resulting in the patient waiting long periods of time in the ED for an inpatient hospital bed assignment (ie, boarding).

Boarding negatively affects hospital operations, causing resource strain due to boarded patients’ continued consumption of nurse and physician resources. It precludes the ability to see more patients, because the boarder is occupying an ED bed when an ED level of care is likely not needed [23]. This strain results in a ripple effect throughout the ED that limits all patients’ access to timely emergency care [24] and further impacts the emergency medical services system by increasing ambulance diversion and patient offload time (the time paramedics spend waiting for an ED bed to become available, after which they are able to return to service) [25]. Reduction or removal of the exit block that causes boarding would dramatically reduce the duration of patient boarding. Thus, removal of the 2 main causes of ED crowding (ie, exit block and boarding) would greatly reduce crowding and increase access to care.

Easing ED crowding requires a multifaceted approach. That is to say that there is not one single solution, but rather multiple solutions applied at various points of care that hold promise in easing ED crowding. This paper will report on the use of biomarkers, measured very early in the continuum of care, as a mechanism to predict admission, thereby enabling hospitals to initiate the complex and time-consuming bed management process early and reduce ED crowding.

Background and Rationale

Prior Interventions to Address Boarding and Crowding

Hospitals have implemented a variety of hospital-level and ED-level interventions to reduce boarding and crowding. In terms of hospital-level interventions, some hospitals have reportedly reduced boarding by improving inpatient bed availability, for example by shortening patient stays through better management of hospital services that are not continuously available (eg, catheterizations [26,27]), moving discharged patients awaiting transportation or nonacute care services to “discharge lounges” [28,29], managing discharges in a more timely manner and expediting discharges [30-32], and managing bed cleaning turnaround more efficiently [33-35]. Because ED crowding does not have a singular cause, it also does not have a singular solution. As such, these measures may only address part of the issue; they do not address boarding’s root cause—exit block. Rather, these efficiency measures contribute to process improvement and operational efficiency, because they successfully and sensibly increase hospital bed availability and streamline and simplify processes for providers and administrators, saving them time and improving flow. Thus, they have the potential to reduce boarding and consequent crowding. Hospitals could benefit from early intelligence about demand for beds, and the need to ensure that the right types of beds are available for ED patients.

ED-level interventions have primarily focused on reducing crowding through improvements in ED flow and throughput, such as by using fast tracking [36-38], split-flow processing [39-41], rapid assessment zones [42-44], team triage [45], triage nurse ordering [46], triage standing orders [47], bedside registration [48], physician scribes [49-51], ED flow coordinators [52], point-of-care testing [53-56], and physical expansion of the ED [57]. While some of these measures may make positive contributions to ED efficiency and flow, they are not unlike the hospital-level interventions, which contribute to solving parts of the problem but do not address exit block [58]. Instead, these measures primarily promote efficiency in subsections of the ED care continuum and move patients more quickly toward upstream bottlenecks in the ED process. Even a dedicated ED flow coordinator who successfully increases flow throughout the ED will see much of that benefit lost if improvements are not also implemented outside the ED [52].

Interdepartmental Interventions to Address Exit Block

Interventions that have included interdepartmental collaboration with hospital management support have made positive steps toward reducing exit block. Such interventions have resulted in a 68% reduction in the time from inpatient bed requests to receipt of inpatient admission orders (210 minutes to 75 minutes; this does not mean a bed has been assigned, only that the order has been created) and a 25% reduction in the time from inpatient bed requests to patient departure from the ED (360 minutes to 270 minutes) [22]. These interventions are primarily viewed through a process improvement lens, in terms of bed management strategy. For example, a study by Barrett et al [59] reported a 52% reduction in “hold time” (the time from admission decision to departure from the ED) when full-time bed managers were able to identify and assign patients to beds within 15 minutes of a bed request. Another such study, by Howell et al [60], reported a 90-minute reduction between ED patient registration and patient physical departure from the ED for admitted patients when a dedicated “bed traffic controller” was used. The difference between the Barrett et al [59] study and the Howell et al [60] study is that the former employed resources at the micro or patient level, whereas the latter employed resources at the macro or process level, presumably with a top-down view of “traffic.”

The process improvements and bed management strategies reported by Barrett et al [59] and Howell et al [60] also demonstrated how real-time ED data on congestion, flow, and patient admissions can be used by hospital staff outside of the ED, such as the hospital bed manager, to prepare for and manage admissions and bed demand. With reliable information to predict the likeliness of being admitted, effective bed management strategies could be deployed earlier in the admission continuum cycle.

Predictive Modeling in Health Care

A variety of models have been used to estimate the risk of hospital admission from the ED, including logistic regression and machine learning, and a variety of predictor variables have been used, including the primary complaint, prior ED visits, referral source, medical history, and mode of arrival. However, model reliance on information that is not readily available (such as patient records) or is inaccurate (such as patient reports of medical history) can be problematic for the application and operation of hospital-admission prediction models. Immediately available point-of-care information from patient biomarkers, such as age, gender, vital signs, and acuity level, offer an advantage over previously collected information [61].

Currently, there is no benchmark to compare hospital admission prediction models. This was evidenced by the authors’ previous systematic review and critical assessment study of models predicting hospital admission, which found that all had potential biases [61].

Biomarker Indicators of Admission

The word “biomarker,” short for “biological marker,” refers to a broad category of objective indicators of medical state that can be measured accurately and reproducibly [62]. Examples of biomarkers are age, x-ray images, vital signs, genes, alleles, gender, cognitive state, and acuity level. Vital signs are the most essential biomarker for monitoring hospitalized patients and are the simplest, least expensive, most readily available, and probably the most important information gathered on patients [63]. They are especially useful in the ED environment, which is populated by patients with a variety of symptoms and conditions, challenging care providers to assess patients quickly. Failure to recognize patient severity or acuity can be detrimental or fatal in the ED. Vital signs that are assessed in real time provide an opportunity to avert this risk to patients, because changes in vital signs have been shown to occur several hours before serious adverse events [64-68]. As such, vital signs can be used to identify ED patients at risk of deterioration [67-72].

The purpose of this study was to report on the development of a model that used patient biomarkers collected at triage (the 5 vital signs and age, gender, and acuity level) for the early prediction of the risk of imminent hospital admission or transfer from the ED for adult patients.

Methods

Study Design

This retrospective cohort study evaluated 1 year of consecutive data events for adult patients admitted to the ED and developed an algorithm to predict which patients would require imminent hospital admission. Eight variables collected at triage were evaluated for their role in the outcome of the patient ED visit. Logistic regression was used to model the study data.

Study Setting, Data Source, and Population

The sample population of deidentified data was drawn from 1 year (January 1, 2019, through December 31, 2019) of consecutive ED admissions to an academic medical center and were queried from its Informatics for Integrating Biology to the Bedside (i2b2) database [73], part of the National Institutes of Health–funded National Centers for Biomedical Computing. Transfer patients (ie, ED patients requiring inpatient hospital admission who were transferred to other hospitals for clinical reasons, such as to receive specialty care) were grouped with admitted patients, because their clinical presentation and reasons for transfer to other facilities for inpatient admission were clinically identical to those of admitted patients [74]. The academic medical center ED is a 48-bed, level-1 adult trauma center with an average daily ED census of 300 patients. The hospital has 1157 beds enterprise-wide.

All clinical data were collected at triage by nurses and entered into the electronic medical record at the point of care. Standardized methods were used to collect vital signs and acuity level.

Inclusion and Exclusion Criteria

All adult (ie, age ≥18 years), nonpsychiatric, nonobstetric, fully triaged patients admitted to the ED (including those transferred to other hospitals for admission) and subsequently admitted to the hospital or discharged from the ED were included in this study. Psychiatric, pediatric (ie, age <18 years), and obstetric patients were excluded. Psychiatric and obstetric patients were excluded because these populations’ symptomology, and the clinical variables that are evaluated to determine their course of treatment, are significantly clinically different from the general-medicine population [75,76]. Pediatric patients were excluded because the threshold for admission for these patients is lower than for adults [77], and their inclusion would result in overly sensitive inclusion criteria for adults.

Selection of Variables for Measurement

The choice of variables evaluated (Table 1) for model development was derived from a systematic review of studies evaluating models designed to predict hospital admission, which suggested variables that were most valuable for patient admission or discharge [61]. Predictors were selected a priori by expert knowledge. Although the literature suggests that SpO2 [78], level of consciousness [79], and mode of arrival [78,80-85] are important variables for consideration [61], the available data were too inconsistent and, therefore, were not included in this model.

Table 1 Eight variables were analyzed for their utility in predicting hospital admission and discharge.

Variables	Means of collection	
Predictor variables	

	Age	Provided by patient (or family or friend if patient was unable to report)	

	Acuity	The standardized 5-level Emergency Severity Index [86] was used by the triage nurse to categorize patient acuity from most urgent (level 1) to least urgent (level 5)	

	Systolic blood pressure; diastolic blood pressure; heart rate, respiration rate; temperature	Typical, standardized methods were used to collect vital signs; blood pressure was the only variable collected by 2 methods: manual (the primary method) and automated	

	Gender	Provided by patient (or family or friend if patient was unable to report); if the patient was unaccompanied, clinicians determined gender by visual inspection	
Outcome variable	

	Admitted or discharged	Determined by physician	

Study Protocol and Data Management

The data were exported from i2b2, imported into an Excel table for review and cleaning, then exported to the Stata statistical package (version 14.1; StataCorp) for analysis. The data were evaluated for missing values.

Data Analysis and Model Development

Data distribution was investigated with summary statistics and histograms. We examined the univariate associations of age, systolic blood pressure (BP), diastolic BP, heart rate, acuity, and gender with the probability of admission using a logistic regression. Traditional logistic regression assumes that the association between continuous risk factors and the probability of admission is linear on a log-odds scale. We considered a more flexible model in which continuous risk factors were included as fractional polynomials, a model-building technique that allows for nonlinear associations [87], and temperature, respiration, and acuity were included as categorical risk factors.

We performed a multivariable fractional polynomial (MFP) analysis that included all risk factors. Temperature values were tightly clustered, with 97% of values between 36.1 °C and 37.8 °C, with a wide spread in values above and below these values. For the purposes of modeling, we created a modified temperature variable where values less than 36.1 °C or greater than 37.8 °C were truncated. Fractional polynomials of the modified temperature were combined with dummy variables indicating high (>37.8 °C) and low (<36.1 °C) temperatures. Respiration was included as a categorical variable due to difficulty in modeling the association between respiration as a continuous variable and the probability of admission. The MFP analysis included nonlinear relationships if they were sufficiently supported by the data. The fit of a second order fractional polynomial was compared to that of the null model, the linear model, and finally to the optimal first-order polynomial. Convergence was achieved when the functional forms did not change. The significance level for the comparison of fractional polynomial models was set equal to 0.01. As some subjects visited the ED more than once, we considered a robust MFP analysis that allowed for correlation between repeat observations of the same subject. Descriptive statistics considered each patient visit as unique. Patient numbers refer to the number of ED encounters.

Model performance was assessed by discrimination and calibration. Discrimination, the model’s ability to accurately distinguish between admission and nonadmission [88], was measured with the area under the receiver operating characteristics curve (AUROC) [89]. To assess potential overoptimism, we also calculated a 10-fold cross-validated AUROC. Calibration, the extent to which the model-predicted probabilities agree with observed binary outcomes [90], is a more appropriate gauge of model performance [91] and was measured by Hosmer-Lemeshow goodness of fit and evaluated graphically using a “calibration belt” [92] for internal validation. The calibration belt methodology formulated the relationship between the predictions and the true probabilities of admission with a second logit regression model based on a polynomial transformation of the predictions. The degree of the polynomial was forwardly selected, beginning with the second order on the basis of a sequence of likelihood-ratio tests [91].

The model was designed to be hospital-specific with application to a particular ED population. As such, we did not measure external validity. Risk factors were evaluated for extreme values, resulting in the loss of less than 2% of patient events overall.

Ethical Considerations

Ethical approval to conduct the investigation was obtained from The University of Alabama at Birmingham Institutional Review Board (IRB-300007437). This study was conducted on a data set that was void of any protected health information.

Results

Descriptive Data

The population consisted of 93,847 adults (age ≥18 years) who were fully triaged, general-medicine (ie, nonpsychiatric and nonobstetric) patients admitted to the ED from January 1, 2019, through December 31, 2019, and subsequently discharged from the ED or admitted to the hospital. The mean age of the 93,847 patients was 46.3 years; 55.6% (52,147) were female; 56.4% (52,974) had acuity level 3; mean systolic BP was 139 mmHg; mean diastolic BP was 84 mmHg; mean heart rate was 87.2 beats/minute; mean respiration rate was 17.7 breaths/minute; and mean temperature was 36.8 °C (Table 2). Temperature and respiration rate had long-tailed, tightly clustered distributions. Temperature ranged from 27 °C to 40.3 °C with only 1% (938) of values less than 36 °C and 1% (938) greater than 38.4 °C. Respiration rate was recorded as a whole number and was clustered at even numbers, with 26% (24,400), 40% (37,539), and 12% (11,262) of subjects having respiration rates of 16, 18, and 20 breaths per minute, respectively, ranging from 10 to 40, with 1% (938) of values less than 14 and 1% (938) greater than 26. Compared to those not admitted, admitted patients were more likely to be male; be older; have lower systolic BP and lower diastolic BP; have higher heart rate, respiration rate, and temperature; and be more acute, as indicated by a lower Emergency Severity Index (ESI) [86] level. This index has a scale of 1 to 5, with 1 being most urgent and 5 being least urgent. Of those admitted, 45% (5779/12,711) had acuity scores less than 3, compared to only 8% (6426/81,136) of those not admitted.

Table 2 Values for predictor variables by admission status.

Variables	Not admitted (N=81,136)	Admitted (N=12,711)	Total (N=93,847)	
Age (years), mean (SD)	44.8 (17.4)	55.8 (16.9)	46.3 (17.3)	
Gender, n (%)	

	Male	35,169 (43.3)	6531 (51.4)	41,700 (44.4)	

	Female	45,967 (56.7)	6180 (48.6)	52,147 (55.6)	
Systolic blood pressure (mm Hg), mean (SD)	139.1 (23.2)	137.9 (28.5)	139.0 (50)	
Diastolic blood pressure (mm Hg), mean (SD)	84.4 (13.5)	81.2 (16)	84. (13.9)	
Heart rate (beats/minute), mean (SD)	86.3 (15.5)	92.9 (18.6)	87.2 (16.1)	
Respiration rate (breaths/minute), mean (SD)	17.6 (1.8)	18.6 (3.1)	17.7 (2.1)	
Temperature (°C), mean (SD)	36.8 (0.4)	36.8 (0.6)	36.8 (0.4)	
Emergency Severity Index level, n (%)a	

	1	23 (0)	571 (4.5)	594 (0.6)	

	2	6403 (7.9)	5208 (41)	11,611 (12.4)	

	3	46,454 (57.3)	6520 (51.3)	52,974 (56.4)	

	4	26,128 (32.2)	385 (3)	26,513 (28.3)	

	5	2128 (2.6)	27 (0.2)	2155 (2.3)	
aRanges from most urgent (1) to least urgent (5).

Probability of Admission

Male admission rates were higher, at 15.7% (6531/41,700) compared to 11.9% (6180/52,147) for women. There was a strong association between acuity and admission, with the probability of admission falling sharply from 96.1% (571/594) for the most urgent patients (ESI 1) to 1.1% (23/2155) for the least urgent patients (ESI 5) (Table 3). Figure 1 shows variable distributions and their associations with the probability of admission, shown on a logit scale. Results are based on second order fractional polynomials. There was a clear, nonlinear association for all continuous variables except age. The probability of admission increased until age 80 and then leveled off. For systolic BP, diastolic BP, heart rate (Figure 1), respiration rate, and temperature (Figure 2), the association was nonlinear, with the probability of admission lowest at the center of the distribution and higher at the extremes. For example, for systolic BP, the probability of admission was lowest, at 15%, for values between 120 mm Hg and 150 mm Hg, and the probability of admission continued to rise at values outside this range, reaching 95% for values <75 mm Hg and a probability of 40% at values >235 mm Hg. Similarly, at a temperature of 36.7 °C, the probability of admission was lowest, at 17%, increasing to 50% at 39.4 °C and to 80% at 35 °C.

The MFP logistic regression showed significant associations with acuity, respiration, and gender, first-order fractional polynomials for age, and second-order fractional polynomials for systolic BP, diastolic BP, temperature, and heart rate.

Table 3 Probability of hospital admission by acuity level. The most acute patients are at Emergency Severity Index levels 1 and 2, with the least urgent at level 5. A total of 12,711 of 93,847 (13.5%) patients were admitted.

Emergency Severity Index level	Total patients, n	Admitted patients, n	Probability of admission, %	
1	594	571	96.1	
2	11,611	5208	44.9	
3	52,974	6520	12.3	
4	26,513	385	1.5	
5	2155	27	1.1	

Figure 1 Histograms of variable distributions overlaid with graphs showing the probability of admission on a log-odds scale. The shaded areas around the curves represent the 95% CI. BP: blood pressure.

Figure 2 Histograms of variable distributions overlaid with graphs showing the observed admission rates.

Fit and Calibration

In Figure 3, the observed admission rate is plotted against the predicted admission rate based on the results of our final logistic regression model. There was excellent agreement between observed probabilities and predicted probabilities based on the model. The 95% CI fell below the identity line at the high end, which indicated that the model slightly overpredicted risk for patients who had a probability of admission over 0.59. This difference in probabilities was less than 0.04. For admission probabilities less than 0.59, the bias was less than 0.01.

Figure 3 Calibration belt of observed versus predicted admission probabilities. The bisector is the line of perfect calibration. The calibration belt (shown in gray) represents the 95% confidence level calibration of the model.

Model Discrimination

Model discrimination was measured with the AUROC (Figure 4). The AUROC was 0.841, indicating that the model had good ability to discriminate between patients who would and would not be admitted [93]. The 10-fold cross-validated AUROCs ranged from 0.839 to 0.842.

Figure 4 Area under the receiver operating characteristics curve for predicting admission. AUROC: Area under the receiver operating characteristics curve.

Missing Values

Age and gender values were complete for all 113,739 patients. Missing values for all other variables were very low, ranging from 1.5% to 1.9% (1650 to 2143), except for temperature and acuity, which were missing for 6.8% (7685) and 9.2% (10,419) of patients, respectively. As 83% (8,647/10,419) of cases missing acuity had been admitted, this group was identified as “missing not at random.” We found no evidence to suggest that temperature was not “missing at random,” and the degree of missingness was low enough that we did not expect it to bias results. Cases with missing values were excluded.

Discussion

Principal Findings

We have illustrated the application of sophisticated fractional polynomials to identify and model nonlinear associations between risk factors and the probability of admission using immediately available patient biomarkers (age, systolic BP, diastolic BP, heart rate, respiration rate, temperature, gender, and acuity level) collected at triage. The resulting prediction model exhibited excellent calibration with good agreement between observed and predicted admissions at all risk-of-admission levels. The model showed good ability to discriminate between patients who would and would not be admitted. Methodological techniques promoted internal validity and mitigated against overfitting and endogeneity, which can arise when predictor variables are correlated with the outcome due to their relationship with variables not in the model [94]. Given our large sample size, a priori inclusion of risk factors, and predictor selection based on topic knowledge, the risk of variable omissions was reduced, and the risk of overfitting due to the use of optimal fractional polynomials was not a concern. A 10-fold cross-validation yielded almost identical AUROC values.

Categorization of respiration rate and truncation of temperature could cause loss of relevant information [95]. However, our transformations were informed by the data: our cut-off points for respiration rate reflected how it was recorded (ie, responses were usually 1 of 3 values and showed a preference for even numbers), and although temperature was truncated, less than 4% (3753/93,847) of observations were affected, indicator variables for low and high temperature were included, and temperature was included as a continuous variable.

Anecdotal information suggests that in the practice of a busy ED, failure to record information deemed nonessential can occur when a patient is already scheduled for hospital admission, and this is most likely to occur when the patient is receiving life-saving care. These cases are very acute and are likely to be admitted. The patients with a missing acuity level tended to be very acute (ie, most cases with missing acuity were admitted) and we were comfortable excluding them, because they were not the patient group that the model aimed to identify as requiring admission; these patients likely had already been identified as urgently needing care and were already likely to be admitted. This is a site-specific model designed to operate in a test environment and show proof of concept; generalizability is not assumed. Because this model uses standardized biomarker data and not data specific to the study environment, it is possible that with recalibration, this model could be useful outside the study environment. It is worth noting, however, that in addition to the real-time availability of electronic patient data, a requirement of model implementation is an application to retrieve the data and apply it to the model algorithm to produce a patient’s likelihood of admission, then provide the information to bed managers to begin securing patient beds early.

This model, as proposed, has real-world utility for those involved in the patient admission continuum, because it allows patients to be moved out of the ED sooner, thereby easing exit block and benefiting patient care and hospital operations [59,60]. The model’s reliance on biomarkers that are routinely collected at the initial point of care (ie, ED triage) and have standardized definitions, measurements, and interpretations [62] is advantageous for a model that can be used very early in the patient care continuum. That, however, does not imply that model development and implementation within a setting is easy. Rather, the data coming from electronic medical records may require labor-intensive preparation to make it suitable for model development and implementation.

This model also showed that a hospital can develop a system for identification of patients at high risk of admission for use in resolving problems such as exit block. The model can be adapted to other ED environments using each ED’s individual data.

Comparison With Prior Work

Addressing ED overcrowding and exit block cannot be accomplished by applying a one-size-fits-all solution. The most recent prior work in this area centers around different methods and models for addressing the same problem—ED crowding. For example, Acuna et al [96] optimized ED crowding by creating an ambulance allocation model that led to a 31% improvement in ED crowding, and Isfahani et al [97] used a computer simulation model to assess the effect of ED discharge lounges, finding there was a 5% reduction in admission waiting times. In terms of applying algorithms, Brink et al [98] developed an 8-variable model to predict hospital admission for elderly patients and Marcusson et al [99] developed a 38-variable model to predict hospital admission for elderly patients; both models aimed at helping patients receive care sooner.

Limitations

The applicability of this study should be understood in the context of its limitations. As mentioned earlier, this study was performed as a proof of concept in a large academic medical center and may lack generalizability to other environments. If this model were applied in a setting where complex processes to secure inpatient beds are not undertaken by the hospital (involving, for example, providers, equipment, and other specialized resources for different patient conditions), or where securing beds does not require a large amount of time, then the time-saving advantages of this model would not be realized by bed managers. Additionally, there may have been confounding factors that were mediating or moderating factors in our model and outside the scope of this study. Lastly, while exit block and ED boarding have been reported internationally, this study was conducted in a US hospital, and is not internationally generalizable. Perhaps a similar, recalibrated version of our derived model would have applications in divergent ED settings. However, we recommend that other hospitals develop hospital-specific models using the MFP modeling techniques presented here.

Conclusion

This primary data study illustrates the application of a site-specific risk prediction model to reduce ED crowding due to exit block. MFPs were used to predict the probability of admission based on 8 biomarkers (5 vital signs and age, gender, and acuity level) and to generate variables utilized by the logistic regression model to produce a site-specific formula to analyze future input data. This intervention can reduce ED exit block, the known source of ED boarding and crowding, by enabling the hospital to seek and requisition hospital beds earlier and transition ED patients into those beds earlier. This intervention requires interdepartmental collaboration with the support of hospital management to be successfully implemented into hospital structures and processes. Compared to other interventions in the hospital admission and bed assignment process that have successfully reduced crowding [22,59,60], this model goes a step further by looking ahead to predict which patients will be admitted, thereby providing the needed information to initiate admission and bed assignment processes much earlier in the care continuum. The model’s prediction of patient admissions combined with the utility of real-time hospital data to improve congestion, flow, and patient admissions [59,60] results in a powerful tool to impact the ED crowding crisis.

Abbreviations

AUROC area under the receiver operating characteristics curve

BP blood pressure

ED emergency department

ESI Emergency Severity Index

MFP multivariable fractional polynomial

Conflicts of Interest: None declared.
==== Refs
1 Stead LG Decker WW The International Journal of Emergency Medicine: successes of the first year Int J Emerg Med 2009 04 15 2 1 1 2 10.1007/s12245-009-0102-2
2 Kauppila T Seppänen Katri Mattila J Kaartinen J The effect on the patient flow in a local health care after implementing reverse triage in a primary care emergency department: a longitudinal follow-up study Scand J Prim Health Care 2017 06 08 35 2 214 220 10.1080/02813432.2017.1333320 28593802 28593802
3 Hospital Compare Centers for Medicare and Medicaid Services 2022-02-05 https://www.cms.gov/Medicare/Quality-Initiatives -Patient-Assessment-Instruments/HospitalQualityInits/HospitalCompare
4 Carter EJ Pouch SM Larson EL The relationship between emergency department crowding and patient outcomes: a systematic review J Nurs Scholarsh 2014 03 46 2 106 15 10.1111/jnu.12055 24354886 24354886
5 Reznek MA Murray E Youngren MN Durham NT Michael SS Door-to-Imaging Time for Acute Stroke Patients Is Adversely Affected by Emergency Department Crowding Stroke 2017 01 48 1 49 54 10.1161/STROKEAHA.116.015131 27856953 STROKEAHA.116.015131 27856953
6 Richards JR van der Linden MC Derlet RW Providing Care in Emergency Department Hallways: Demands, Dangers, and Deaths Advances in Emergency Medicine 2014 12 25 2014 1 7 10.1155/2014/495219 10.1155/2014/495219
7 Sinclair D Emergency department overcrowding - implications for paediatric emergency medicine Paediatr Child Health 2007 07 12 6 491 494 10.1093/pch/12.6.491 19030415 19030415
8 American College of Emergency Physicians (ACEP) Crowding. Policy statement Ann Emerg Med 2013 06 61 6 726 7 10.1016/j.annemergmed.2013.03.037 23684339 S0196-0644(13)00321-1
9 Hospital emergency departments: Crowding continues to occur, and some patients wait longer than recommended time frames Government Accountability Office 2009 2022-07-18 https://www.gao.gov/products/gao-09-347
10 Institute of Medicine Hospital-Based Emergency Care: At the Breaking Point 2007 Washington, DC The National Academies Press
11 Higginson I Emergency department crowding Emerg Med J 2012 06 29 6 437 43 10.1136/emermed-2011-200532 22223713 emermed-2011-200532 22223713
12 Mason S Knowles E Boyle A Exit block in emergency departments: a rapid evidence review Emerg Med J 2017 01 34 1 46 51 10.1136/emermed-2015-205201 27789568 emermed-2015-205201 27789568
13 Cowan RM Trzeciak S Clinical review: Emergency department overcrowding and the potential impact on the critically ill Crit Care 2005 06 9 3 291 5 10.1186/cc2981 15987383 cc2981 15987383
14 Rabin E Kocher K McClelland M Pines J Hwang U Rathlev N Asplin B Trueger NS Weber E Solutions to emergency department 'boarding' and crowding are underused and may need to be legislated Health Aff (Millwood) 2012 08 31 8 1757 66 10.1377/hlthaff.2011.0786 22869654 31/8/1757 22869654
15 Derlet RW Richards JR Ten solutions for emergency department crowding West J Emerg Med 2008 01 9 1 24 7 19561699 19561699
16 Handel DA Hilton JA Ward MJ Rabin E Zwemer FL Pines JM Emergency department throughput, crowding, and financial outcomes for hospitals Acad Emerg Med 2010 08 17 8 840 7 10.1111/j.1553-2712.2010.00814.x 10.1111/j.1553-2712.2010.00814.x 20670321 ACEM814 20670321
17 Rutherford PA Kotagal U Luther K Provost L Ryckman F Taylor J Achieving Hospital-wide Patient Flow 2020 Boston, MA Institute for Healthcare Improvement
18 AHA's Annual Survey of Hospitals 2016 American Hospital Association 2016 2022-02-05 https://www.ahadata.com/aha- annual-survey-database
19 Protect Public Data Hub: Inpatient Bed Dashboard U.S. Department of Health & Human Services 2022 2022-02-05 https://public-data-hub-dhhs.hub.arcgis.com/pages/Hospital%20Utilization
20 March 2019 Report to the Congress: Medicare Payment Policy The Medicare Payment Advisory Commission 2019 2022-02-05 https://www.medpac.gov/document/march-2019-report-to-the-congress-medicare-payment-policy/
21 June 2014 Report to the Congress: Medicare and the Health Care Delivery System The Medicare Payment Advisory Commission 2014 2021-02-05 https://www.medpac.gov/document/http-www-medpac-gov-docs-default-source-reports-jun14 _entirereport-pdf/
22 Amarasingham R Swanson TS Treichler DB Amarasingham SN Reed WG A rapid admission protocol to reduce emergency department boarding times Qual Saf Health Care 2010 06 19 3 200 4 10.1136/qshc.2008.031641 20142408 qshc.2008.031641 20142408
23 Bukata R Why is ED holding still an issue? Emergency Physicians Monthly 2017 2019-05-01 http://epmonthly.com/article/why -is-ed-holding-still-an-issue/
24 Nicks BA Manthey DM The impact of psychiatric patient boarding in emergency departments Emerg Med Int 2012 2012 360308 5 10.1155/2012/360308 10.1155/2012/360308 22888437 22888437
25 Geiderman JM Marco CA Moskop JC Adams J Derse AR Ethics of ambulance diversion Am J Emerg Med 2015 06 33 6 822 7 10.1016/j.ajem.2014.12.002 25616586 S0735-6757(14)00895-X 25616586
26 Levin SR Dittus R Aronsky D Weinger MB Han J Boord J France D Optimizing cardiology capacity to reduce emergency department boarding: a systems engineering approach Am Heart J 2008 12 156 6 1202 9 10.1016/j.ahj.2008.07.007 19033021 S0002-8703(08)00617-0 19033021
27 Levin S Dittus R Aronsky D Weinger M France D Evaluating the effects of increasing surgical volume on emergency department patient access BMJ Qual Saf 2011 02 20 2 146 52 10.1136/bmjqs.2008.030007 21209127 bmjqs.2008.030007
28 Smith K Implementation of the Discharge Hospitality Center to Reduce Emergency Department Boarding: A Quality Improvement Project The Free Library 2018 2022-07-18 https://www.thefreelibrary.com/Implementation+of+the+Discharge +Hospitality+Center+to+Reduce...-a0568974204
29 Franklin BJ Vakili S Huckman RS Hosein S Falk N Cheng K Murray M Harris S Morris CA Goralnick E The Inpatient Discharge Lounge as a Potential Mechanism to Mitigate Emergency Department Boarding and Crowding Ann Emerg Med 2020 06 75 6 704 714 10.1016/j.annemergmed.2019.12.002 31983501 S0196-0644(19)31433-7 31983501
30 Johnson M Sensei Lean Capasso Vin Improving patient flow through a better discharge process J Healthc Manag 2012 57 2 89 93 22530290 22530290
31 Powell ES Khare RK Venkatesh AK Van Roo BD Adams JG Reinhardt G The relationship between inpatient discharge timing and emergency department boarding J Emerg Med 2012 02 42 2 186 96 10.1016/j.jemermed.2010.06.028 20888163 S0736-4679(10)00653-0 20888163
32 El-Eid GR Kaddoum R Tamim H Hitti EA Improving hospital discharge time: a successful implementation of Six Sigma methodology Medicine (Baltimore) 2015 03 94 12 e633 10.1097/MD.0000000000000633 10.1097/MD.0000000000000633 25816029 00005792-201503040-00004 25816029
33 Kehoe B From good to great: 2010 ES department of the year: Doylestown Hospital Health Facilities Management 2010 2022-08-18 https://www.hfmmagazine.com/articles/1147-from-good-to-great
34 Tortorella F Ukanowicz D Douglas-Ntagha P Ray R Triller M Improving bed turnover time with a bed management system J Nurs Adm 2013 01 43 1 37 43 10.1097/NNA.0b013e3182785fe7 23232178 00005110-201301000-00008 23232178
35 VA National Bed Control System Department of Veterans Affairs 2021 2022-01-29 https://catalog.data.gov/dataset/va-national -bed-control-system
36 Cheney C Adopt this 5-part process to reduce ER length of stay Health Leaders 2019 2019-05-05 https://www.healthleaders media.com/clinical-care/adopt-5-part-process-reduce-er-length-stay
37 Celona C Amaranto A Ferrer R Wieland M Abrams S Obusan F LoPuzzo Sharon Joy Valsamma Interdisciplinary Design to Improve Fast Track in the Emergency Department Adv Emerg Nurs J 2018 40 3 198 203 10.1097/TME.0000000000000199 30059375 01261775-201807000-00009 30059375
38 Chrusciel J Fontaine X Devillard A Cordonnier A Kanagaratnam L Laplanche D Sanchez S Impact of the implementation of a fast-track on emergency department length of stay and quality of care indicators in the Champagne-Ardenne region: a before-after study BMJ Open 2019 06 19 9 6 e026200 10.1136/bmjopen-2018-026200 31221873 bmjopen-2018-026200
39 Bish P McCormick M Otegbeye M Ready-JET-Go: Split Flow Accelerates ED Throughput J Emerg Nurs 2016 03 42 2 114 9 10.1016/j.jen.2015.06.003 26264788 S0099-1767(15)00284-6 26264788
40 Garrett JS Berry C Wong H Qin H Kline JA The effect of vertical split-flow patient management on emergency department throughput and efficiency Am J Emerg Med 2018 09 36 9 1581 1584 10.1016/j.ajem.2018.01.035 29352674 S0735-6757(18)30035-4 29352674
41 Wallingford G Joshi N Callagy P Stone J Brown I Shen S Introduction of a Horizontal and Vertical Split Flow Model of Emergency Department Patients as a Response to Overcrowding J Emerg Nurs 2018 07 44 4 345 352 10.1016/j.jen.2017.10.017 29169818 S0099-1767(17)30435-X 29169818
42 Bullard MJ Villa-Roel C Guo X Holroyd BR Innes G Schull MJ Vandermeer B Ospina M Rowe BH The role of a rapid assessment zone/pod on reducing overcrowding in emergency departments: a systematic review Emerg Med J 2012 05 29 5 372 8 10.1136/emj.2010.103598 21515880 emj.2010.103598 21515880
43 Anderson J Burke R Augusto K Beagan B Rodrigues-Belong M Frazer L Stack Colin Shukla Anil Pope Jen V The Effect of a Rapid Assessment Zone on Emergency Department Operations and Throughput Ann Emerg Med 2020 02 75 2 236 245 10.1016/j.annemergmed.2019.07.047 31668573 S0196-0644(19)30664-X 31668573
44 Chartier L Josephson T Bates K Kuipers M Improving emergency department flow through Rapid Medical Evaluation unit BMJ Qual Improv Rep 2015 4 1 u206156.w2663 10.1136/bmjquality.u206156.w2663 10.1136/bmjquality.u206156.w2663 26734447 bmjquality_uu206156.w2663
45 Ming T Lai A Lau P Can Team Triage Improve Patient Flow in the Emergency Department? A Systematic Review and Meta-Analysis Adv Emerg Nurs J 2016 38 3 233 50 10.1097/TME.0000000000000113 27482995 01261775-201607000-00009 27482995
46 Rowe BH Guo X Villa-Roel C Schull M Holroyd B Bullard M Vandermeer B Ospina M Innes G The role of triage liaison physicians on mitigating overcrowding in emergency departments: a systematic review Acad Emerg Med 2011 02 18 2 111 20 10.1111/j.1553-2712.2010.00984.x 10.1111/j.1553-2712.2010.00984.x 21314769 21314769
47 Hwang CW Payton T Weeks E Plourde M Implementing Triage Standing Orders in the Emergency Department Leads to Reduced Physician-to-Disposition Times Advances in Emergency Medicine 2016 06 15 2016 1 6 10.1155/2016/7213625
48 McHugh M Van Dyke KJ Howell E Adams F Moss D Yonek J Changes in patient flow among five hospitals participating in a learning collaborative J Healthc Qual 2013 35 1 21 9 10.1111/j.1945-1474.2011.00163.x 22092988 22092988
49 Heaton H Nestler D Lohse C Sadosty A Impact of scribes on emergency department patient throughput one year after implementation Am J Emerg Med 2017 02 35 2 311 314 10.1016/j.ajem.2016.11.017 27856140 S0735-6757(16)30826-9 27856140
50 Walker K Ben-Meir M Dunlop W Rosler R West A O'Connor Gabrielle Chan T Badcock D Putland M Hansen K Crock C Liew D Taylor D Staples M Impact of scribes on emergency medicine doctors' productivity and patient throughput: multicentre randomised trial BMJ 2019 01 30 364 l121 10.1136/bmj.l121 30700408 30700408
51 Heaton HA Schwartz EJ Gifford WJ Koch KA Lohse CM Monroe RJ Thompson KM Walker LE Hellmich TR Impact of scribes on throughput metrics and billing during an electronic medical record transition Am J Emerg Med 2020 08 38 8 1594 1598 10.1016/j.ajem.2019.158433 31522929 S0735-6757(19)30582-0 31522929
52 Murphy S Barth B Carlton E Gleason M Cannon C Does an ED flow coordinator improve patient throughput? J Emerg Nurs 2014 11 40 6 605 12 10.1016/j.jen.2014.03.007 24974359 S0099-1767(14)00108-1 24974359
53 Jarvis P Davies M Mitchell K Taylor I Baker M Can the Introduction of Point-of-Care Testing for Renal Function in the Emergency Department Reduce Overcrowding? Point Care 2015 14 42 4 10.1097/POC.0000000000000048
54 Kankaanpää Meri Raitakari M Muukkonen L Gustafsson S Heitto M Palomäki Ari Suojanen Kimmo Harjola Veli-Pekka Use of point-of-care testing and early assessment model reduces length of stay for ambulatory patients in an emergency department Scand J Trauma Resusc Emerg Med 2016 10 18 24 1 125 10.1186/s13049-016-0319-z 27756354 10.1186/s13049-016-0319-z 27756354
55 Li L McCaughey E Iles-Mann J Sargeant A Westbrook J Georgiou A Does Point-of-Care Testing Impact Length of Stay in Emergency Departments (EDs)?: A Before and After Study of 26 Rural and Remote EDs Stud Health Technol Inform 2018 252 99 104 30040690 30040690
56 Singer AJ Taylor M LeBlanc D Meyers K Perez K Thode HC Pines JM Early Point-of-Care Testing at Triage Reduces Care Time in Stable Adult Emergency Department Patients J Emerg Med 2018 08 55 2 172 178 10.1016/j.jemermed.2018.04.061 29887410 S0736-4679(18)30489-X 29887410
57 Mumma BE McCue JY Li C Holmes JF Effects of emergency department expansion on emergency department patient flow Acad Emerg Med 2014 05 19 21 5 504 9 10.1111/acem.12366 10.1111/acem.12366 24842500 24842500
58 Henderson K Boyle A Exit block in the emergency department: recognition and consequences Br J Hosp Med (Lond) 2014 11 02 75 11 623 6 10.12968/hmed.2014.75.11.623 25383431 25383431
59 Barrett L Ford S Ward-Smith P A bed management strategy for overcrowding in the emergency department Nurs Econ 2012 30 2 82 5, 116 22558725 22558725
60 Howell E Bessman E Kravet S Kolodner K Marshall R Wright S Active bed management by hospitalists and emergency department throughput Ann Intern Med 2008 12 02 149 11 804 11 10.7326/0003-4819-149-11-200812020-00006 19047027 149/11/804 19047027
61 Monahan AC Feldman SS Models Predicting Hospital Admission of Adult Patients Utilizing Prehospital Data: Systematic Review Using PROBAST and CHARMS JMIR Med Inform 2021 09 16 9 9 e30022 10.2196/30022 34528893 v9i9e30022 34528893
62 Strimbu K Tavel JA What are biomarkers? Current Opinion in HIV and AIDS 2010 5 6 463 466 10.1097/coh.0b013e32833ed177 20978388
63 Kellett J Sebat F Make vital signs great again - A call for action Eur J Intern Med 2017 11 45 13 19 10.1016/j.ejim.2017.09.018 28941841 S0953-6205(17)30367-9 28941841
64 Kause J Smith G Prytherch D Parr M Flabouris A Hillman K Intensive Care Society (UK)AustralianNew Zealand Intensive Care Society Clinical Trials Group A comparison of antecedents to cardiac arrests, deaths and emergency intensive care admissions in Australia and New Zealand, and the United Kingdom--the ACADEMIA study Resuscitation 2004 09 62 3 275 82 10.1016/j.resuscitation.2004.05.016 15325446 S0300957204002473 15325446
65 Buist M Bernard S Nguyen TV Moore G Anderson J Association between clinically abnormal observations and subsequent in-hospital mortality: a prospective study Resuscitation 2004 08 62 2 137 41 10.1016/j.resuscitation.2004.03.005 15294398 S0300957204001236 15294398
66 Hillman KM Bristow PJ Chey T Daffurn K Jacques T Norman SL Bishop GF Simmons G Antecedents to hospital deaths Intern Med J 2001 08 31 6 343 8 10.1046/j.1445-5994.2001.00077.x 11529588 11529588
67 Henriksen DP Brabrand M Lassen AT Prognosis and risk factors for deterioration in patients admitted to a medical emergency department PLoS One 2014 9 4 e94649 10.1371/journal.pone.0094649 24718637 PONE-D-14-01342 24718637
68 Barfod C Lauritzen MMP Danker JK Sölétormos György Forberg JL Berlac PA Lippert F Lundstrøm Lars Hyldborg Antonsen K Lange KHW Abnormal vital signs are strong predictors for intensive care unit admission and in-hospital mortality in adults triaged in the emergency department - a prospective cohort study Scand J Trauma Resusc Emerg Med 2012 04 10 20 28 10.1186/1757-7241-20-28 22490208 1757-7241-20-28 22490208
69 Ljunggren M Castrén Maaret Nordberg M Kurland L The association between vital signs and mortality in a retrospective cohort study of an unselected emergency department population Scand J Trauma Resusc Emerg Med 2016 03 03 24 21 10.1186/s13049-016-0213-8 26940235 10.1186/s13049-016-0213-8 26940235
70 Farrohknia N Castrén Maaret Ehrenberg A Lind L Oredsson S Jonsson H Asplund K Göransson Katarina E Emergency department triage scales and their components: a systematic review of the scientific evidence Scand J Trauma Resusc Emerg Med 2011 06 30 19 42 10.1186/1757-7241-19-42 21718476 1757-7241-19-42 21718476
71 Pedersen NE Oestergaard D Lippert A End points for validating early warning scores in the context of rapid response systems: a Delphi consensus study Acta Anaesthesiol Scand 2016 05 60 5 616 22 10.1111/aas.12668 26708159 26708159
72 Cardona-Morrell M Prgomet M Lake R Nicholson M Harrison R Long J Westbrook J Braithwaite J Hillman K Vital signs monitoring and nurse-patient interaction: A qualitative observational study of hospital practice Int J Nurs Stud 2016 04 56 9 16 10.1016/j.ijnurstu.2015.12.007 26775214 S0020-7489(15)00383-1 26775214
73 i2b2: Informatics for Integrating Biology and the Bedside i2b2 tranSMARTFoundation 2021-12-20 https://www.i2b2.org/about/index.html
74 EMTALA Fact Sheet American College of Emergency Physicians 2022-01-20 https://www.acep.org/life-as-a-physician/ethics --legal/emtala/emtala-fact-sheet/
75 Care of the Psychiatric Patient in the Emergency Department – A Review of the Literature American College of Emergency Physicians Emergency Medicine Practice Committee 2014 2022-07-18 https://tinyurl.com/y7zpxc4a
76 Triage of the Obstetrics Patient in the Emergency Department: Is There Only One Patient? Pennsylvania Patient Safety Advisor 2008 05 2022-08-18 http://patientsafety.pa.gov/ADVISORIES/Pages/200809_85.aspx
77 Paediatric emergency triage, assessment and treatment World Health Organization 2016 2022-02-05 https://apps.who.int/iris/bitstream/handle/10665/204463/9789241510219_eng.pdf
78 Lucke JA de Gelder J Clarijs F Heringhaus C de Craen AJM Fogteloo AJ Blauw GJ Groot BD Mooijaart SP Early prediction of hospital admission for emergency department patients: a comparison between patients younger or older than 70 years Emerg Med J 2018 01 16 35 1 18 27 10.1136/emermed-2016-205846 28814479 emermed-2016-205846 28814479
79 Burch VC Tarr G Morroni C Modified early warning score predicts the need for hospital admission and inhospital mortality Emerg Med J 2008 10 01 25 10 674 8 10.1136/emj.2007.057661 18843068 25/10/674 18843068
80 Parker CA Liu N Wu SX Shen Y Lam SSW Ong MEH Predicting hospital admission at the emergency department triage: A novel prediction model Am J Emerg Med 2019 08 37 8 1498 1504 10.1016/j.ajem.2018.10.060 30413365 S0735-6757(18)30891-X 30413365
81 Peck J Benneyan J Nightingale D Gaehde S Predicting emergency department inpatient admissions to improve same-day patient flow Acad Emerg Med 2012 09 19 9 E1045 54 10.1111/j.1553-2712.2012.01435.x 10.1111/j.1553-2712.2012.01435.x 22978731 22978731
82 Kraaijvanger N Rijpsma D Roovers L van Leeuwen H Kaasjager K van den Brand L Horstink L Edwards M Development and validation of an admission prediction tool for emergency departments in the Netherlands Emerg Med J 2018 08 07 35 8 464 470 10.1136/emermed-2017-206673 29627769 emermed-2017-206673 29627769
83 Cameron A Rodgers K Ireland A Jamdar R McKay GA A simple tool to predict admission at the time of triage Emerg Med J 2015 03 13 32 3 174 9 10.1136/emermed-2013-203200 24421344 emermed-2013-203200 24421344
84 Cagle R Darling E Kim B Identifying healthcare activities using a real-time location system J Med Pract Manage 2014 30 2 128 30 25807605 25807605
85 Fihn SD Francis J Clancy C Nielson C Nelson K Rumsfeld J Cullen T Bates J Graham GL Insights from advanced analytics at the Veterans Health Administration Health Aff (Millwood) 2014 07 33 7 1203 11 10.1377/hlthaff.2014.0054 25006147 33/7/1203 25006147
86 Emergency Severity Index (ESI): A Triage Tool for Emergency Departments Agency for Healthcare Research Quality 2020 2021-10-12 https://www.ahrq.gov/patient-safety/settings/emergency-dept/esi.html
87 Royston P Ambler G Sauerbrei W The use of fractional polynomials to model continuous risk variables in epidemiology Int J Epidemiol 1999 10 01 28 5 964 74 10.1093/ije/28.5.964 10597998 10597998
88 Steyerberg EW Vickers AJ Cook NR Gerds T Gonen M Obuchowski N Pencina MJ Kattan MW Assessing the performance of prediction models: a framework for traditional and novel measures Epidemiology 2010 01 21 1 128 38 10.1097/EDE.0b013e3181c30fb2 20010215 00001648-201001000-00022 20010215
89 Heagerty PJ Zheng Y Survival model predictive accuracy and ROC curves Biometrics 2005 03 61 1 92 105 10.1111/j.0006-341X.2005.030814.x 15737082 BIOM030814 15737082
90 Waljee A Higgins P Singal A A primer on predictive models Clin Transl Gastroenterol 2014 01 02 5 e44 10.1038/ctg.2013.19 24384866 ctg201319 24384866
91 Moran JL Santamaria JD Duke GJ Modelling hospital outcome: problems with endogeneity BMC Med Res Methodol 2021 06 21 21 1 124 10.1186/s12874-021-01251-8 34154530 10.1186/s12874-021-01251-8 34154530
92 Nattino G Lemeshow S Phillips G Finazzi S Bertolini G Assessing the Calibration of Dichotomous Outcome Models with the Calibration Belt Stata J 2018 01 01 17 4 1003 1014 10.1177/1536867X1801700414
93 Tape TG Interpreting diagnostic tests: The area under an ROC curve University of Nebraska Medical Center 2020-02-10 http://gim.unmc.edu/dxtests/roc3.htm
94 Mennemeyer ST Can econometrics rescue epidemiology? Ann Epidemiol 1997 05 7 4 249 50 10.1016/s1047-2797(97)00021-5 9177106 S1047-2797(97)00021-5 9177106
95 Moons Karel G M de Groot Joris A H Bouwmeester Walter Vergouwe Yvonne Mallett Susan Altman Douglas G Reitsma Johannes B Collins Gary S Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist PLoS Med 2014 10 11 10 e1001744 10.1371/journal.pmed.1001744 25314315 PMEDICINE-D-14-00436 25314315
96 Acuna J Zayas-Castro J Charkhgard H Ambulance allocation optimization model for the overcrowding problem in US emergency departments: A case study in Florida Socio-Econ Plan Sci 2020 09 71 100747 10.1016/j.seps.2019.100747 10.1016/j.seps.2019.100747
97 Nasr Isfahani M Davari F Azizkhani R Rezvani M Decreased Emergency Department Overcrowding by Discharge Lounge: A Computer Simulation Study Int J Prev Med 2020 11 13 10.4103/ijpvm.IJPVM_582_18 32175053 IJPVM-11-13 32175053
98 Brink A Alsma J Brink HS de Gelder J Lucke JA Mooijaart SP Zietse R Schuit SCE Lingsma HF Prediction admission in the older population in the Emergency Department: the CLEARED tool Neth J Med 2020 12 78 6 357 367 33380533 33380533
99 Marcusson J Nord M Dong H Lyth J Clinically useful prediction of hospital admissions in an older population BMC Geriatr 2020 03 06 20 1 95 10.1186/s12877-020-1475-6 32143637 10.1186/s12877-020-1475-6 32143637
