
==== Front
Heliyon
Heliyon
Heliyon
2405-8440
Elsevier

S2405-8440(24)12178-0
10.1016/j.heliyon.2024.e36147
e36147
Research Article
Variation in nurses’ compliance with an Early Warning Score protocol: A retrospective cohort study
Leenen Jobbe PL. j.p.l.leenen@isala.nl
ab⁎
Mondria Chantal L. c
a Connected Care Centre, Isala, Zwolle, the Netherlands
b Research Group IT Innovations in Healthcare, Windesheim University of Applied Sciences, Zwolle, the Netherlands
c Department Healthcare and Wellbeing, Windesheim University of Applied Sciences, Zwolle, the Netherlands
⁎ Corresponding author. Dr. van Heesweg 2, 8025 AB, Zwolle, the Netherlands. j.p.l.leenen@isala.nl
13 8 2024
30 8 2024
13 8 2024
10 16 e3614721 3 2024
25 7 2024
9 8 2024
© 2024 The Authors
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
Introduction

Early Warning Score (EWS) protocols are based on intermittent vital sign measurements, and aim to detect clinical deterioration in a timely manner. Despite its predictive value, its effectiveness remains suboptimal. An important limitation appears to be poor compliance with the EWS protocol and its variation between general wards. The current research does not yet provide an understanding of EWS compliance and variation in different nursing wards.

Aim

To explore the variation in nurses' compliance with the EWS protocol among patients with and without complications and between different nursing wards.

Methods

In a retrospective single-center cohort study, all patient files from three nursing wards of a tertiary teaching hospital in the Netherlands were reviewed over a 1-month period. Compliance was divided into three categories:1) calculation accuracy, 2) monitoring frequency end 3) clinical response.

Results

The cohort of 210 patients contained 5864 measurements, of which 4125 (70.6 %) included EWS. Significant differences in the measured vital signs within incomplete measurements were found among nursing wards. Compliance to monitoring frequency was higher within EWSs of 0–1 (78.4 %) than within EWSs of ≥2 (26.1 %). The proportion of correct follow-up was significantly higher in patients with complications, as was the correct clinical response to an EWS of ≥3 (84.8 % vs. 55.0; p = .011).

Conclusion

Our results suggest suboptimal compliance with the EWS protocol, with large variations between patients with and without complications and between different general care wards. Nurses tended to be more compliant with the EWS protocol for patients with complications.

Highlights

• Compliance with Early Warning Score protocols in general wards remains suboptimal.

• Large variations in compliance were found between patients with and without complications.

• Nurses tended to be more compliant with the protocol for patients with complications.

• Large variations in compliance were observed among different general wards.

• General wards focus on the most important vital signs in relation to their areas of specialization.

Keywords

Physiologic monitoring [MeSH]
Vital signs [MeSH]
Early warning score (MeSH)
EWS
Protocol compliance
Rapid response system
Nursing assessment
==== Body
pmcAbbreviations

EWS Early Warning Score

FFHET Fisher-Freeman-Halton Exact Test

IM Internal Medicine

RN Registered Nurse

RRT Rapid Response Team

1 Introduction

Most adverse events occurring in hospital wards are preceded by a considerable period of changes in vital signs, which are important indicators of clinical deterioration [1]. Therefore, Early Warning Scores (EWS) are widely used in daily clinical practice to facilitate the early identification of deteriorating patients [2,3]. EWS includes objective parameters, such as vital signs and level of consciousness, and a subjective parameter, named nurses’ worry indicator. These parameters are measured at predetermined intervals. Cut-off points lead to an aggregated risk score. Based on this score, registered nurses can trigger responses according to protocol [4,5]. These responses include increasing the monitoring frequency or requesting assistance from physicians or Rapid Response Teams (RRTs).

Previous studies demonstrated that EWS can predict adverse patient outcomes, such as unexpected deaths or cardiopulmonary arrests requiring unplanned intensive care unit (ICU) admission [3,[6], [7], [8], [9]]. In addition, EWSs are user-friendly and provide a common language for healthcare providers across different specialties [7]. Despite these strengths of EWS, research has demonstrated that the identification of clinical deterioration is often late or entirely missed [[10], [11], [12]]. This might be a result of several limitations of the EWS. First, EWSs are limited by its intermittent nature [13]. Predefined intervals make it possible to miss the patient's deterioration between those intermittent measurements [14]. Second, EWSs are generic tools, and disease and population differences may influence its efficacy [13,15]. Thirdly, poor compliance to EWS protocols has been mentioned as a limitation in several studies [4,[16], [17], [18], [19], [20]].

The protocol compliance, consisting of the quality and frequency of vital sign measurements and subsequent protocol-driven measures initiated by nurses, are the most important factors in the identification of clinical deterioration [21,22]. Therefore, hospitals have implemented various strategies to educate nurses and physicians on the EWS, emphasizing when, how and why to use it [23]. Education initiatives often include comprehensive training programs, regular workshops, and integration of EWS protocols into daily clinical routines [24,25]. These programs are designed to ensure that healthcare professionals understand the significance of the EWS in identifying patient deterioration early and the subsequent steps required for timely intervention.

Nevertheless, compliance remains an issue. A previous study divided poor compliance in three themes: 1) ‘calculation accuracy’ defined as an uncompleted measurements to calculate an EWS, 2) ‘monitoring frequency’ defined as the EWS is not according the follow-up measurement frequency, and 3) ‘clinical response’ defined as consultation of a physician of RRT when mandated by protocol [17]. Reasons mentioned for non-compliance are time constraints, understaffing, nurses' confidence in their ability to cope with deteriorating patients, and the large number of patients with elevated EWSs [17,26,27].

However, most studies are limited to assessing EWS compliance in patients only with adverse events, which contrarily have a low incidence in nursing wards [28]. Although poor compliance is also suggested with and without signs of clinical deterioration, current research does not yet provide a complete understanding of EWS compliance and variation on nursing wards. Examining compliance in both patients with and without complications provides a comprehensive understanding of how the EWS is utilized in everyday practice, not just in critical situations. This broader perspective can reveal underlying issues that might otherwise go unnoticed, ensuring that the EWS system functions effectively as a preventive measure rather than solely a reactive tool. Moreover, given the generic nature of EWS, there may be variation in compliance across specialties. Understanding these variations is essential for developing targeted interventions for specific nursing wards to improve compliance and ensure the benefits of the EWS system in diverse clinical settings. Therefore, this study aimed to explore the variation in nurses' adherence to the EWS protocol in patients with and without complications and between different nursing wards.

2 Methods

2.1 Study design

A retrospective, single-centre cohort study was performed, with a review of all patient files for a 1-month period (January 2021) in three nursing wards in a 1245-bed teaching hospital in the Netherlands. Patient files were reviewed throughout the entire admission period up to 30 days after discharge. The study has been reported according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist (Appendix 1) [29].

2.2 Study population

The study population included all patients admitted to one of the participating nursing wards (internal medicine (IM), surgery, and pulmonary disease) in January 2021 who were 18 years or older and had a minimum length of stay (LOS) of 48 h. Patients who were excluded from the standardised EWS protocol (Appendix 2), such as terminal patients or patients with COVID-19, were also excluded in this study.

2.3 Data collection

The software CTcue (CTcue, Amsterdam, Netherlands), powered by artificial intelligence (AI), was used. CTcue can extract data from an electronic health record (EHR) based on preformulated queries. First, the patient cohort was identified based on population and exclusion criteria. This was followed by creating sufficient queries for each variable. Subsequently, these established queries were validated on content, and all variables were pseudonymously filtered from the patient records. All patient- and baseline-characteristics and measurements of vital signs and EWSs (including date and time) were collected. The data collection process was performed by an executive researcher (#1) in collaboration with a data manager. The validation, structuring and organising of all extracted data was performed by executive researcher (#2) and a statistician and crosschecked by an coordinating researcher (#2). Data on clinical response, including the presence of a description in physician or nurses’ reports of consulting a physician or RRT, were collected manually from the patient files by the executive researcher (#1) and crosschecked by the coordinating researcher (#2).

2.4 Outcomes

The mandatory parameters were measured once per day by a registered nurse (RN). An EWS 0–1 is followed within 24 h and an EWS ≥2 should be followed every 8 h over the next 24 h. Lastly, an EWS ≥3 should be followed by consulting a physician or RRT. Appendix 2 demonstrates the EWS protocol, and its cut-off points applied on the participating nursing wards. Regarding the training received by RNs on the EWS system, there is a standardized training module provided during onboarding to the ward. Additionally, RNs undergo yearly e-learning sessions that include simulation training to reinforce their understanding and application of the EWS protocol.

The primary endpoint was compliance to the EWS protocol divided in three categories: 1) EWS calculation accuracy, 2) monitoring frequency and 3) clinical response [17].

2.5 Calculation accuracy

Calculation accuracy was determined by evaluating the completeness of all mandatory vital signs required for the calculation of an EWS at all measurement moments. A measurement moment was defined as a specific date and time, including all measurements. Registration of all required measurements performed manual by the RN resulted automatically in an EWS calculation and registration in the EHR. Omitting any individual parameter results in the inability to calculate an EWS automatically. Therefore, we defined an incomplete measurement moment as one where no EWS was calculated, and will therefore contain missing parameters for its calculation. First, all measurement moments were evaluated on wheatear an EWS was calculated based on all vital signs. Second, the proportion of measured vital signs were determined for the incomplete measurement moments without an EWS.

2.6 Monitoring frequency

Monitoring frequency was evaluated by calculating proportions correctly followed by EWSs within the mandated timeframes. Follow-up of EWSs 0–1 was considered correct if the next EWS was calculated within 24 h. The follow-up of EWSs ≥2 were considered correct if the EWS was calculated within 8 h, 8–16 h and 16–24 h. With one missing part of this follow-up, it was considered incorrect. We also evaluated the proportion of follow-up per timeframe (8 h, 8–16 h and 16–24 h). The last EWS on admission (before discharge) was not included in this analysis because there was no follow-up after discharge.

2.7 Clinical response

If an EWS is ≥ 3, the protocol mandates that a physician is consulted within 30 min, with evaluation and medical policy to be determined within 1 h. It was not possible to retrospectively trace whether the clinical response actions had been conducted within the specified timeframes. Therefore, the first EWS of ≥ 3 per patient was evaluated for clinical response. The first EWS of ≥3 was chosen because it provided the first indication of possible clinical deterioration. The first EWS ≥3 was evaluated on presence of a description in physician or nurses’ reports of consulting a physician or RRT.

2.8 Other study parameters

For stratification of patients with and without complications, complication status was described as yes/no depending on absence or presence of complications according to the list of Hospital Complications [30] based on the assessment report of the physician together with the discharge letter in the EHR which were independently reviewed by both authors (Appendix 4) [30]. Other demographics and patient characteristics, such as the nursing ward, sex, age, medical history, and clinical outcomes (length of stay, readmission <30days and mortality <30 days) were also obtained to describe the study population.

2.9 Data analysis

All data were analysed using descriptive statistics and presented for the entire cohort, each nursing ward, and for patients with and without complications. Continuous variables were presented as mean and standard deviation (SD) considering the skewness of the data. Skewness was determined by plotting a histogram for visual inspection and the Shapiro-Wilk test. Categorical variables are presented as frequencies and percentages. Primary endpoints are presented for the entire cohort, complications versus non-complications, and each nursing ward (including a distinction between patients with and without complications). Fishers' exact test was applied to test for differences between primary endpoints in patients with and without complications (two groups). A Fisher-Freeman-Halton Exact Test (FFHET) was used to test for differences between the three nursing wards. Post-hoc analyses were performed using Fishers’ exact test with a Bonferroni correction for multiple testing. A two-tailed significance level of <.05 (5 %) was applied in all the tests. In a post-hoc analysis with Bonferroni correction, a significance level of .017 (α/k, .05/3) was applied. All statistical analyses were performed with IBM SPSS Statistics 27.0 (Armonk, NY, IBM Corp.).

2.10 Sample size

Due to the explorative nature of this study and the limited available evidence on this specific topic, no formal sample size calculation was conducted [31]. Most importantly, all subgroups (complications vs. no complications and different nursing wards) had to be large enough to accurately describe variations in compliance. The ratio of complications to non-complications across the population base was unclear. Data on surgical complications were most available: postoperative complications occurred in up to 33 % of patients undergoing colorectal procedures [32]. The registration of patients on the nursing wards of interest, showed approximately 200 patients per month and approximately 25 % of the patient with complications. Considered the above, this study focussed on a 1-month cohort, January 2021.

2.11 Ethical considerations

The Medical Ethics Review Committee of Isala reviewed the study protocol (protocol 20211219) and stated this study did not involve Dutch Medical Research Involving Human Subjects Acts (non-WMO) and received waiver for obtain informed consent. The study was conducted in accordance with the Declaration of Helsinki and Data Protection Regulation (GDPR).

3 Results

3.1 Study characteristics

After exclusion, the cohort consisted of 210 patients (Appendix 3. Of these, 60 (28.6 %) patients were admitted to the IM ward, 98 (46.7 %) to the surgical ward, and 52 (24.8 %) to the pulmonary ward (Table 1). The mean LOS was 11.7 (SD 11.1) days, complication rate was 24.3 % (n = 51), and mortality rate was 6.7 % (n = 14) (Table 1; Appendix 4). A total of 5864 EWS measurements were included in the analysis.Table 1 Patient characteristics.

Table 1	Total	Nursing ward	Complications	
Variable	(n = 210)	Internal medicine (n = 60)	Surgical (n = 98)	Pulmonary (n = 52)	Complications (n = 51)	No complications (n = 159)	
Demographics	
 Gender (male), n (%)	112 (53.3)	33 (55.0)	54 (55.1)	25 (48.1)	31 (60.8)	81 (50.9)	
 Age (years), mean (sd*)	68.7 (13.8)	67.9 (15.5)	68.8 (13.7)	69.6 (11.9)	69.5 (14.8)	68.5 (13.5)	
Medical History	
 COPD†, n (%)	66 (31.4)	11 (18.3)	21 (21.4)	34 (65.4)	16 (31.4)	50 (31.4)	
 CHF‡, n (%)	28 (13.3)	6 (10.0)	13 (13.3)	9 (17.3)	4 (7.8)	24 (15.1)	
 Diabetes mellitus, n (%)	57 (27.1)	12 (20.0)	34 (34.7)	11 (21.2)	16 (31.4)	41 (25.8)	
 Hypertension, n (%)	112 (53.3)	31 (51.7)	56 (57.1)	25 (48.1)	28 (54.9)	84 (52.8)	
 Peripheral Vascular Disease, n (%)	46 (21.9)	4 (6.7)	33 (33.7)	9 (17.3)	10 (19.6)	36 (22.6)	
 Renal disease (eGFR§
<30), n (%)	21 (10.0)	6 (10.0)	12 (12.2)	3 (5.8)	7 (13.7)	14 (8.8)	
 CVD¶, n (%)	53 (25.2)	13 (21.7)	22 (22.4)	18 (34.6)	15 (29.4)	38 (23.9)	
 Dementia, n (%)	3 (1.4)	0 (.0)	2 (2.0)	1 (1.9)	1 (2.0)	2 (1.3)	
 Connective tissue disease (RA/SLE#)) n (%)	17 (8.1)	2 (3.3)	7 (7.1)	8 (15.4)	6 (11.8)	11 (6.9)	
 Peptic ulcer disease, n (%)	9 (4.3)	6 (10.0)	1 (1.0)	2 (3.8)	1 (2.0)	8 (5.0)	
 Liver disease, n (%)	17 (8.1)	13 (21.7)	2 (2.0)	2 (3.8)	1 (2.0)	16 (10.1)	
 Hemiplegia, n (%)	2 (1.0)	0 (.0)	1 (1.0)	1 (1.9)	1 (2.0)	1 (.6)	
 Leukemia, n (%)	3 (1.4)	0 (.0)	3 (3.1)	0 (.0)	0 (.0)	3 (1.9)	
 Lymphoma, n (%)	5 (2.4)	1 (1.7)	2 (2.0)	2 (3.8)	0 (.0)	5 (3.1)	
 AIDS+, n (%)	0 (.0)	0 (.0)	0 (.0)	0 (.0)	0 (.0)	0 (.0)	
 Myocardial infarction, n (%)	30 (14.3)	6 (10.0)	20 (20.4)	3 (5.8)	6 (11.8)	24 (15.1)	
Clinical Outcomes	
 Length of Stay, mean (sd*)	11.7 (11.1)	11.2 (9.0)	13.1 (13.8)	9.6 (6.7)	19.2 (16.6)	9.3 (7.3)	
 Mortality <30 days	14 (6.7)	6 (10.0)	1 (1.0)	7 (13.5)	10 (19.6)	4 (2.5)	
 Readmission <30 days	46 (21.9)	19 (31.7)	17 (17.3)	10 (19.2)	9 (17.6)	37 (23.3)	
Complication rate, n (%)	51 (24.8)	13 (21.7)	23 (23.5)	15 (28.9)	n/a	n/a	
List of abbreviations.

*Standard Deviation.

†Chronic Obstructive Pulmonary Disease.

‡ Congestive Heart failure.

§ Estimated glomerular filtration rate.

¶ Cerebro Vascular Accident.

# Rheumatoid Arthritis/Systemic Lupus Erythematosus.

+ Acquired immunodeficiency syndrome n/a: not applicable

4 Main outcomes

4.1 Calculation accuracy

Of all included measurement moments (n = 5846), 70.6 % contained an EWS and were considered complete (Table 2). Of the 29.4 % incomplete measurement moments, oxygen saturation (60.5 %) and heart rate (53.7 %) were the most frequently measured and respiratory frequency the least frequently measured (22.8 %) (Table 2).Table 2 Early Warning Score (EWS) compliance per nursing ward.

Table 2Endpoint	Total	Nursing ward	P value	
		Internal medicine (n = 60)	Surgical (n = 98)	Pulmonary (n = 52)		
Calculation accurac	
Number of measurement moments, n	5846	1552	2802	1492		
Proportion completed EWSs†, n (%)	4125 (70.6)	1125 (72.5)	2039 (72.8)	961 (64.4)	<.001*	
Number of incomplete measurement moments, n	1721	427	763	531		
Proportion measured parameters within incomplete sets	
 Blood pressure, n (%)	858 (49.9)	286 (67.0)	378 (49.5)	194 (36.5)	<.001*	
 Heartrate, n (%)	9224 (53.7)	267 (62.5)	402 (52.7)	255 (48.0)	<.001*	
 Temperature, n (%)	794 (46.1)	177 (41.5)	438 (57.4)	179 (33.7)	<.001*	
 Oxygen saturation, n (%)	1042 (60.5)	231 (54.1)	371 (48.6)	419 (78.9)	<.001*	
 Respiratory frequency, n (%)	393 (22.8)	69 (16.2)	165 (21.6)	204 (29.9)	<.001*	
Monitoring frequency	
Total number of EWS† 0–1, n	2650	731	1421	500		
Proportion EWS† 0–1 correct followed, n (%)	2080 (78.4)	567 (77.6)	1085 (76.4)	428 (85.6)	<.001*	
Total number of EWS† ≥2	1265	335	521	409		
Proportion EWS† ≥2 correct followed, n (%)	331 (26.1)	71 (21.2)	163 (31.3)	97 (23.7)	.002*	
 <8u, n (%)	708 (56.0)	184 (54.9)	306 (58.7)	218 (53.3)	.229	
 8-16u, n (%)	889 (70.3)	233 (69.6)	384 (73.7)	272 (66.5)	.055	
 16-24u, n (%)	817 (64.6)	203 (60.6)	339 (65.1)	275 (67.2)	.164	
Clinical response	
Number of clinical responses (N)	73	23	22	28		
Clinical response correct n (%)	50 (68.5)	17 (73.9)	17 (77.3)	16 (57.1)	.304	
List of abbreviations.

*Statistically significant difference with a two-tailed significance level of p < .05, † Early Warning Score.

In general, patients with complications had a significantly higher proportion of completed EWSs (75.3 %) than those without complications (67.1 %, p < .001). However, within all incomplete measurement periods, more individual parameters were measured in patients without complications (Table 3). Likewise, when stratified by ward, the internal and surgical wards showed a higher proportion of completed EWSs in patients with complications (p < .001) (Table 4). In contrast, no significant differences were found in the pulmonary ward (p = .337).Table 3 Early Warning Score (EWS) compliance in patients stratified by complications.

Table 3Endpoint	Complications	P value	
	Complications (n = 51)	No complications (n = 159)		
Calculation accuracy	
Number of measurement moments,	2485	3361		
Proportion completed EWSs†, n (%)	1871 (75.3)	2254 (67.1)	<.001*	
Number of incomplete measurement moments, n	614	1107		
Proportion measured parameters within incomplete sets	
 Blood pressure, n (%)	271 (44.1)	587 (53.0)	<.001*	
 Heartrate, n (%)	297 (48.4)	527 (56.6)	.001*	
 Temperature, n (%)	308 (50.2)	486 (43.9)	.013*	
 Saturation, n (%)	351 (57.2)	691 (62.4)	.035	
 Respiratory frequency, n (%)	123 (20.0)	270 (24.4)	.042*	
Monitoring frequency	
Number of EWS† 0–1, n	1107	1545		
Proportion EWS† 0-1correct followed, n (%)	905 (78.8)	1175 (69.8)	<.001*	
Number of EWS† ≥2	713	522		
Proportion EWS† ≥2 correct followed, n (%)	206 (28.9)	125 (22.6)	.014*	
 <8u, n/N (%)	400 (55.4)	308 (55.8)	.954	
 8-16u, n/N (%)	531 (73.5)	358 (64.9)	<.001*	
 16-24u, n/N (%)	465 (64.4)	352 (63.8)	.594	
Clinical respons	
Number of clinical responses (N)	33	40		
Clinical response correct n (%)	28 (84.8)	22 (55.0)	.011*	
List of abbreviations.

*Statistically significant difference with a two-tailed significance level of <.05 (5 %), † Early Warning Score.

Table 4 Early Waning Score (EWS) Compliance on each nursing ward divided by complication vs no complications.

Table 4Endpoint	Internal Medicine	Surgical	Pulmonary	
	Complications (n = 13)	No complications (n = 47)	P-value	Complications (n = 23)	No complications (n = 75)	P-value	Complications (n = 15)	No complications (n = 37)	P-value	
Calculation accurac	
Number of measurement moments, n	619	933		1417	1385		449	1043		
Proportion completed EWSs†, n (%)	480 (77.5)	645 (69.1)	<.001*	1094 (77.2)	945 (68.2)	<.001*	297 (66.1)	664 (63.7)	.337	
Number of incomplete measurement moments, n	139	288		323	440		152	379		
Proportion measured parameters within incomplete sets	
Blood pressure, n (%)	69 (48.9)	211 (75.7)	<.001*	155 (48.0)	223 (50.7)	.465	48 (31.6)	146 (38.5)	.136	
Heartrate, n (%)	64 (46.0)	203 (70.5)	<.001*	164 (50.8)	238 (54.1)	.379	69 (45.4)	186 (49.1)	.501	
Temperature, n (%)	64 (46.0)	113 (39.2)	.208	177 (54.8)	261 (59.3)	.236	67 (44.1)	112 (29.6)	.002*	
Saturation, n (%)	62 (44.6)	169 (58.7)	.007*	170 (52.6)	222 (50.5)	.558	119 (78.3)	300 (79.2)	.815	
Respiratory frequency, n (%)	14 (10.1)	66 (22.9)	.017*	70 (21.7)	95 (21.6)	1	39 (25.7)	120 (31.7)	.208	
Monitoring frequenc	
Number of EWS† 0–1, n	287	444		658	763		162	338		
Proportion EWS† 0–1 correct followed, n (%)	216 (75.3)	351 (79.1)	.239	546 (83.0)	539 (70.6)	<.001*	143 (83.6)	285 (78.3)	.277	
Number of EWS† ≥2,	180	155		413	108		120	289		
Proportion EWS† ≥2 correct followed, n (%)	42 (23.3)	29 (18.7)	.349	133 (32.2)	30 (27.8)	.416	31 (26.8)	66 (22.8)	.525	
 <8u, n (%)	98 (54.4)	86 (55.5)	.912	238 (57.6)	68 (63.0)	.326	64 (53.3)	154 (53.3)	1	
 8-16u, n (%)	129 (71.7)	104 (67.1)	.405	317 (76.8)	67 (62.0)	.003*	85 (70.8)	187 (64.7)	.251	
 16-24u, n (%)	114 (63.3)	89 (57.4)	.313	273 (66.1)	66 (61.1)	.365	78 (65.0)	197 (68.2)	.564	
Clinical Respons	
Number of clinical responses,	9	14		12	10		12	16		
Correct clinical response, n (%)	8/(88.9)	9 (64.3)	.340	11 (91.7)	6 (60.0)	.135	9 (75.0)	7 (43.8)	.136	
List of abbreviations.

*Statistically significant difference with a two-tailed significance level of <.05 (5 %), † Early Warning Score.

When evaluating variation among nursing wards, we found a significantly lower proportion of completed EWSs in the pulmonary ward (64.4 %) than in the internal (72.8 %) and surgical wards (72.5 %) (p < .001) (Table 2). While focusing on the measured parameters within the incomplete measurements, we found variations among the different nursing wards. The pulmonary ward measured oxygen saturation (78.9 %) and respiratory rate (29.9 %) significantly more often than the other nursing wards (p < .001) (Appendix 5). The internal medicine (IM) ward measured blood pressure (66.0 %) and heart rate (62.5 %) more frequently than other nursing wards (p < .001). Additionally, the surgical ward measured temperature significantly more often compared to the other nursing wards (57.4 %) (p < .001). These differences are detailed in Table 2 and Appendix 5.

4.2 Monitoring frequency

Of the 3915 included EWSs for analysis of this endpoint, 67.7 % of EWSs scored 0–1 and 32.3 % EWSs scored ≥2 (Table 2). Of all EWSs 0–1, 78.4 % were correctly followed within 24 h. Of EWSs ≥2, 26.1 % were correctly followed. EWSs ≥2 was most frequently followed within 8–16 h (70.3 %) (Table 2).

The proportion of correct follow-up of EWSs 0–1 was significant higher in patients with complications than in patients without complications (78.8 % vs. 69.8 %, p < .001) (Table 2). Likewise, the proportion of correct follow-up of EWSs ≥2 was higher in patients with complications (28.9 % vs. 22.6 %, p = .014). Both groups showed the best follow-up to an EWS ≥2 within 8–16 h (Table 3). At the surgical ward, correct follow-up of EWS 0–1 (83.0 % vs. 70.6 %; p = .001) and EWS ≥2 between 8 and 16 h (76.8 % vs. 62.0 %; p = .003) were higher in patients with complications. There were no further significant differences between patients with and without complications stratified by the ward (Table 4).

Among the different nursing wards, the proportion EWS 0–1 correct followed was significantly higher in the pulmonary ward (85.6 %) than in the internal and surgical wards (77.6 % and 76.4 %, respectively, p < .001) (Table 3). In contrast, the surgical ward showed a significant higher proportion correct followed EWSs ≥2 (31.3 %, p = .002).

4.3 Clinical response

Seventy-three patients had an EWS of ≥3 during admission. Of these, 50 (68.5 %) clinical responses included a nurses’ or physicians report stating a physician was consulted as response to this EWS and were therefore considered correct (Table 2). The proportion correct clinical responses was significantly better in patients with complications (84.8 %) compared to patients without complications (55.0 %, p = .011) (Table 3). The proportion of correct clinical responses on the IM and surgical wards were 73.9 % and 77.3 %, respectively. This was a higher proportion compared to the pulmonary ward (57.1 %). However, this was not statistically significant (p = .304) (Table 2).

5 Discussion

The aim of this study was to describe variation in compliance of the EWS protocol in patients who did and did not experience complications and differences between general nursing wards. To our knowledge this is the first study to provide an more in-depth analysis of current practice of EWS compliance on nursing wards, including its differences among nursing wards and a broader range of complications. This study showed significant variations among all categories of EWS compliance, calculation accuracy, monitoring frequency and clinical response. The compliance to the EWS protocol appeared significantly better in patient with complications. But also, across the different nursing ward we found significant variations on the measured vital signs within incomplete measurement moments.

In our study, approximately 71 % of all measurement moments were completed. In comparison with previous studies, this result varied between 76 and 90 % [17,18,[33], [34], [35]]. The implementation of automated EWS calculation systems appears to improve completed EWS assessments over time [36,37]. In this study, the hospital also implemented an automated EWS calculation system. However, it is suggested that the improvement in calculation accuracy did not translate into an improvement in patient outcomes [37]. Compliance with escalation protocols when scores are elevated is more problematic. Therefore, it remains to be debated whether improving overall compliance to this part of the protocol will significantly improve the detection of clinical deterioration. Especially because nurses rely heavily on EWS scores but also perform holistic physical assessments to detect patient deterioration earlier [38,39]. Overreliance on scores alone may compromise patient safety [[40], [41], [42]]. There were also some differences in the vital signs collected between the general wards. Heart rate and blood pressure were the most commonly recorded vital signs, and it is unclear why these rates were high. One possible explanation is that a cuff blood pressure measurement automatically generates a heart rate measurement. This is consistent with other findings from a study of EWS compliance in hospitals [43]. In contrast to heart rate and blood pressure measurements, the frequency of measured respiratory rate was low (22.4 %). This has also been found in previous studies [18,44] and may be explained by a lack of understanding of its physiology and time constraints in performing measurements according to protocol; respiratory frequency is known to be an undervalued vital sign by nurses, resulting in poor recording [45]. Given the early detection of clinical deterioration, the low proportion of measured respiratory frequency is encouraging improvement on all wards.

There were also significant differences in the measured vital signs between the different nursing units. Nurses on the pulmonary ward are likely to focus more on oxygen saturation and respiratory rate measurements, whereas the surgical ward focuses considerably more on temperature and the IM ward on blood pressure and heart rate. These differences may mean that each specialty has its own relevant observations in relation to common diseases or complications within these specialties. The generic aspect of the EWS, as previously discussed by Downey et al. [7], therefore provides less opportunity for a tailored approach. In addition, there is evidence that generic EWSs have lower predictive value than specialty-specific scores, and that their sensitivity can be increased by adding variables [46]. Nevertheless, it should be noted there is considerable evidence for respiratory rate as a sensitive parameter for detection of clinical deterioration in a wide range of settings [47].

We also found a large difference in compliance with monitoring frequency within EWSs of 0–1 (78.4 %) and EWSs of ≥2 (26.1 %). We found significantly higher compliance in patients with complications compared to those without (29 % and 23 %, respectively). Although this was statistically significant higher, the compliance remains inadequate. A higher score indicates an increased risk of deterioration. It is therefore alarming that it is precisely the higher EWSs that are poorly adhered to in terms of monitoring frequency. Especially as the intermittent nature of the EWS is already a known limitation [7]. In addition to these findings, we also found that for all subgroups of EWSs, follow-up was best adhered to within 8–16 h ≥ 2. However, as the EWS is a tool for ‘early’ detection of clinical deterioration, we expected follow-up to be best adhered to within 8 h. When compared to other studies, they also found that higher EWSs were significantly associated with a lower likelihood of being monitored according to protocol [18,48]. Time constraints, heavy workloads and understaffing were mentioned by nurses as reasons for non-compliance [17,26,38,49]. On the other hand, nurses with shorter work experience found EWS particularly helpful in prioritizing care [50].

Notably, regardless of the severity of complications, we found better compliance with the EWS protocol in patients with complications. It is known that it is difficult for nurses to comply with EWS protocols due to heavy workloads, previous experiences and challenges in obtaining medical assistance [51]. Although this study did not focus on differences in compliance before and after the manifestation of the complication, better compliance was observed throughout the admission of patients with complications. In particular, we found a large difference in compliance with the clinical response (55 % versus 85 %). The interview study by Petersen et al. which focused on barriers to informing physicians, concluded that nurses considered themselves competent to manage patients without assistance [20,48]. This may explain why nurses had low compliance with the clinical response in patients without complications. Given these findings, it highlights the role of nurses in allocating attention to patients who are about to deteriorate and those who are not considered at risk, regardless of the EWS, as the majority of nurses manipulated or inaccurately documented vital signs to achieve a desired EWS score and would almost override the protocol if it did not agree with their clinical judgement [38,41]. It emphasizes the autonomy of nurses to make clinical decisions based on their clinical judgement.

6 Limitations

The retrospective nature of this study provides an accurate description of current practice, but there are several limitations when considering the results of our study. First, the generalisability of the results is limited due to the single-centre study design. Studying variations in EWS compliance across different centers or over time could be relevant for a first step for future research. Second, although this study included almost 6000 measurements, the subgroups of patients with and without complications stratified by ward resulted in a relatively small sample. Although we found statistical significant differences, this may have compromised the achievement of statistical significance. Third, the distribution of patients within the nursing wards was uneven. Almost 50 % of the included patients were admitted to the surgical ward. Although the statistical power is based on the smallest sample, this did not affect the ability to detect significant variations between the different nursing wards. Fourth, this study focused on completed EWSs and ‘incomplete measurements' without an EWS. Clinical decisions to deviate from the protocol were not taken into account and may have resulted in a narrower understanding of current protocol compliance. Fifth, the study was conducted during the COVID-19 pandemic which may affected on the compliance rates in the pulmonary ward.

7 Recommendations for future research

The findings of this study indicate that the current standards for the identification of clinical deterioration is suboptimal and require enhancement. The introduction of the EWS may have been an overly simplistic solution for the complex issue of clinical deterioration [38]. For future improvements of the identification of clinical deterioration, it is crucial to understand the reasons behind low protocol compliance. Significant variations in protocol compliance should be analysed, and improvement strategies should be tailored to specific groups and contexts [42]. Furthermore, individual factors (e.g. age, sex, medical condition) and contextual factors (day/night shifts, nurse-to-patient ratios) that may influence compliance should be further explored in depth. Nursing wards may also benefit from the application of new strategies to detect complex and dynamic clinical situations. We recommend further research into the thorough development of new strategies to detect clinical deterioration using new innovative technologies. Previous studies have already suggested application of continuous vital signs monitoring increases the accuracy of detecting clinical deterioration in nursing wards [[52], [53], [54], [55]]. Furthermore, trend analysis of intermittent (or continuous) vital signs measurements, enhanced by artificial intelligence and machine learning technologies, may enable the development of more sophisticated and personalised EWSs by combining vital signs with other variables such as patient characteristics [56,57]. Besides a proper algorithm, successful implementation of EWS systems relies heavily on other key factors, including education [42], inter-professional collaboration [58], interoperability with IT systems , and managerial appreciation [59]. These elements collectively contribute to the effective integration and utilization of EWS systems in clinical practice.

8 Conclusion

In conclusion, the results of this study suggest suboptimal compliance with the EWS protocol with a large amount of variation in relation to patients with and without complications and between different general wards. Nurses tended to be more compliant with the EWS protocol in patients with complications. In addition, nursing wards appear to focus on the most important vital signs in relation to the speciality of the ward. It is essential to integrate EWS with nurses' clinical assessments to enhance early detection of deterioration.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Data availability statement

Data will be made available on request.

CRediT authorship contribution statement

Jobbe PL. Leenen: Writing – review & editing, Writing – original draft, Validation, Supervision, Investigation, Data curation, Conceptualization. Chantal L. Mondria: Writing – original draft, Visualization, Project administration, Methodology, Formal analysis, Data curation.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A Supplementary data

The following are the Supplementary data to this article:Multimedia component 1

Multimedia component 1

Multimedia component 2

Multimedia component 2

Multimedia component 3

Multimedia component 3

Multimedia component 4

Multimedia component 4

Multimedia component 5

Multimedia component 5

Acknowledgements

The authors would like to thank Dr. Gijs Patijn for critically review the study protocol and Dr. Mireille Edens for assistance in the data analysis.

Appendix A Supplementary data to this article can be found online at https://doi.org/10.1016/j.heliyon.2024.e36147.
==== Refs
References

1 Buist M. Bernard S. Nguyen T.V. Moore G. Anderson J. Association between clinically abnormal observations and subsequent in-hospital mortality: a prospective study Resuscitation 62 2 2004 Aug 137 141 15294398
2 Gerry S. Bonnici T. Birks J. Kirtley S. Virdee P.S. Watkinson P.J. Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology BMJ 369 2020 May m1501
3 Smith M.E.B. Chiovaro J.C. O'Neil M. Kansagara D. Quinones A.R. Freeman M. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review Ann Am Thorac Soc 11 9 2014 Nov 1454 1465 25296111
4 Difonzo M. Performance of the afferent limb of Rapid response systems in managing deteriorating patients: a systematic review Crit Care Res Pract 2019 2019 6902420
5 Subbe C.P. Kruger M. Rutherford P. Gemmel L. Validation of a modified early warning score in medical admissions QJM 94 10 2001 Oct 521 526 11588210
6 Alam N. Hobbelink E.L. van Tienhoven A.J. van de Ven P.M. Jansma E.P. Nanayakkara P.W.B. The impact of the use of the Early Warning Score (EWS) on patient outcomes: a systematic review Resuscitation 85 5 2014 May 587 594 24467882
7 Downey C.L. Tahir W. Randell R. Brown J.M. Jayne D.G. Strengths and limitations of early warning scores: a systematic review and narrative synthesis Int. J. Nurs. Stud. 76 2017 Nov 106 119 28950188
8 Hollis R.H. Graham L.A. Lazenby J.P. Brown D.M. Taylor B.B. Heslin M.J. A role for the early warning score in early identification of critical postoperative complications Ann. Surg. 263 5 2016 May 918 923 26692076
9 Jayasundera R. Neilly M. Smith T.O. Myint P.K. Are early warning scores useful predictors for mortality and morbidity in hospitalised acutely unwell older patients? A systematic review J. Clin. Med. 7 10 2018 Sep
10 Petersen J.A. Antonsen K. Rasmussen L.S. Frequency of early warning score assessment and clinical deterioration in hospitalized patients: a randomized trial Resuscitation 101 2016 Apr
11 Petersen J.A. Mackel R. Antonsen K. Rasmussen L.S. Serious adverse events in a hospital using early warning score – what went wrong? Resuscitation 85 12 2014 Dec
12 Wood C. Chaboyer W. Carr P. How do nurses use early warning scoring systems to detect and act on patient deterioration to ensure patient safety? A scoping review Int. J. Nurs. Stud. 94 2019 Jun
13 Downey C.L. Tahir W. Randell R. Brown J.M. Jayne D.G. Strengths and limitations of early warning scores: a systematic review and narrative synthesis Int. J. Nurs. Stud. 76 2017 Nov
14 Beckett D. Gordon C. Paterson R. Chalkley S. Macleod D. Bell D. Assessment of clinical risk in the out of hours hospital prior to the introduction of Hospital at Night Acute Med. 8 1 2009 33 38 21607208
15 Barlow G. Nathwani D. Davey P. The CURB65 pneumonia severity score outperforms generic sepsis and early warning scores in predicting mortality in community-acquired pneumonia Thorax 62 3 2007 Mar 1
16 Clifton D.A. Clifton L. Sandu D.M. Smith G.B. Tarassenko L. Vollam S.A. “Errors” and omissions in paper-based early warning scores: the association with changes in vital signs--a database analysis BMJ Open 5 7 2015 Jul e007376
17 Credland N. Dyson J. Johnson M.J. What are the patterns of compliance with Early Warning Track and Trigger Tools: a narrative review Appl. Nurs. Res. 44 2018 Dec 39 47 30389058
18 Eddahchouri Y. Koeneman M. Plokker M. Brouwer E. van de Belt T.H. van Goor H. Low compliance to a vital sign safety protocol on general hospital wards: a retrospective cohort study Int. J. Nurs. Stud. 115 2021 Mar 103849
19 Ludikhuize J. Smorenburg S.M. de Rooij S.E. de Jonge E. Identification of deteriorating patients on general wards; measurement of vital parameters and potential effectiveness of the Modified Early Warning Score J. Crit. Care 27 4 2012 Aug 424.e7-13
20 Petersen J.A. Mackel R. Antonsen K. Rasmussen L.S. Serious adverse events in a hospital using early warning score - what went wrong? Resuscitation 85 12 2014 Dec 1699 1703 25238741
21 De Meester K. Das T. Hellemans K. Verbrugghe W. Jorens P.G. Verpooten G.A. Impact of a standardized nurse observation protocol including MEWS after Intensive Care Unit discharge Resuscitation 84 2 2013 Feb 184 188 22796310
22 De Meester K. Haegdorens F. Monsieurs K.G. Verpooten G.A. Holvoet A. Van Bogaert P. Six-day postoperative impact of a standardized nurse observation and escalation protocol: a preintervention and postintervention study J. Crit. Care 28 6 2013 Dec 1068 1074 24011975
23 Mølgaard R.R. Jørgensen L. Grønkjær M. Madsen J.Ø. Christensen E.F. Voldbjerg S.L. Nurses' and physicians' ideas on initiatives for effective use of the early warning score: a participatory study Glob Qual Nurs Res 10 2023 23333936231210148
24 Wood C. Chaboyer W. Carr P. How do nurses use early warning scoring systems to detect and act on patient deterioration to ensure patient safety? A scoping review Int. J. Nurs. Stud. 94 2019 Jun 166 178 31002971
25 Saab M.M. McCarthy B. Andrews T. Savage E. Drummond F.J. Walshe N. The effect of adult Early Warning Systems education on nurses' knowledge, confidence and clinical performance: a systematic review J. Adv. Nurs. 73 11 2017 Nov 2506 2521 28440892
26 Bunkenborg G. Poulsen I. Samuelson K. Ladelund S. Akeson J. Mandatory early warning scoring--implementation evaluated with a mixed-methods approach Appl. Nurs. Res. 29 2016 Feb 168 176 26856510
27 Ludikhuize J. Dongelmans D.A. Smorenburg S.M. Gans-Langelaar M. de Jonge E. de Rooij S.E. How nurses and physicians judge their own quality of care for deteriorating patients on medical wards: self-assessment of quality of care is suboptimal Crit. Care Med. 40 11 2012 Nov 2982 2986 22890255
28 Anderson O. Davis R. Hanna G.B. Vincent C.A. Surgical adverse events: a systematic review Am. J. Surg. 206 2 2013 Aug 253 262 23642651
29 von Elm E. Altman D. Egger M. Pocock S. Gøtzsche P. Vandenbroucke J. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies BMJ 335 7624 2007 806 808 10.1136/bmj.39335.541782.AD.BMJ 335(7624):806–8 17947786
30 Australian Commission on Safety and Quality in Health Care Hospital-acquired Complications (HACs) List 2019 version 3.0
31 Jones S.R. Carley S. Harrison M. An introduction to power and sample size estimation Emerg. Med. J. 20 2003
32 Giglia M.D. Stein S.L. Overlooked long-term complications of colorectal surgery Clin. Colon Rectal Surg. 32 3 2019 May 204 211 31061651
33 Odell M. Detection and management of the deteriorating ward patient: an evaluation of nursing practice J. Clin. Nurs. 24 1–2 2015 Jan 173 182 25039312
34 Smith A.F. Oakey R.J. Incidence and significance of errors in a patient “track and trigger” system during an epidemic of Legionnaires' disease: retrospective casenote analysis Anaesthesia 61 3 2006 Mar 222 228 16480345
35 Pedersen N.E. Rasmussen L.S. Petersen J.A. Gerds T.A. Østergaard D. Lippert A. A critical assessment of early warning score records in 168,000 patients J. Clin. Monit. Comput. 32 1 2018
36 Jones S. Mullally M. Ingleby S. Buist M. Bailey M. Eddleston J.M. Bedside electronic capture of clinical observations and automated clinical alerts to improve compliance with an Early Warning Score protocol Crit Care Resusc 13 2 2011 Jun 83 88 21627575
37 Mestrom E. De Bie A. Steeg M.V. Driessen M. Atallah L. Bezemer R. Implementation of an automated early warning scoring system in a surgical ward: practical use and effects on patient outcomes PLoS One 14 5 2019 e0213402
38 Langkjaer C.S. Bove D.G. Nielsen P.B. Iversen K.K. Bestle M.H. Bunkenborg G. Nurses' experiences and perceptions of two early warning score systems to identify patient deterioration-A focus group study Nurs Open 8 4 2021 Jul 1788 1796 33638617
39 Saab M.M. McCarthy B. Andrews T. Savage E. Drummond F.J. Walshe N. The effect of adult Early Warning Systems education on nurses' knowledge, confidence and clinical performance: a systematic review J. Adv. Nurs. 73 11 2017 Nov 2506 2521 28440892
40 Jensen J.K. Skår R. Tveit B. Introducing the National Early Warning Score - a qualitative study of hospital nurses' perceptions and reactions Nurs Open 6 3 2019 Jul 1067 1075 31367432
41 Le Lagadec M.D. Massey D. Byrne A.L. Connor J. Flenady T. Nurse by numbers: the impact of early warning systems on nurses’ higher-order thinking, a quantitative study J. Adv. Nurs. 2024 May 10 10.1111/jan.16235
42 Wood C. Chaboyer W. Carr P. How do nurses use early warning scoring systems to detect and act on patient deterioration to ensure patient safety? A scoping review Int. J. Nurs. Stud. 94 2019 Jun 166 178 31002971
43 Eddahchouri Y. Koeneman M. Plokker M. Brouwer E. van de Belt T.H. van Goor H. Low compliance to a vital sign safety protocol on general hospital wards: a retrospective cohort study Int. J. Nurs. Stud. 115 2021 Mar
44 Cretikos M.A. Bellomo R. Hillman K. Chen J. Finfer S. Flabouris A. Respiratory rate: the neglected vital sign Med. J. Aust. 188 11 2008 Jun 657 659 18513176
45 Scott J.B. Kaur R. Monitoring breathing frequency, pattern, and effort Respir. Care 65 6 2020 Jun 793 806 32457171
46 Mann K.D. Good N.M. Fatehi F. Khanna S. Campbell V. Conway R. Predicting patient deterioration: a review of tools in the digital hospital setting J. Med. Internet Res. 23 2021
47 Scott J.B. Kaur R. Monitoring breathing frequency, pattern, and effort Respir. Care 65 6 2020
48 Petersen J.A. Antonsen K. Rasmussen L.S. Frequency of early warning score assessment and clinical deterioration in hospitalized patients: a randomized trial Resuscitation 101 2016 91 96 10.1016/j.resuscitation.2016.02.003 26891861
49 Stafseth S.K. Grønbeck S. Lien T. Randen I. Lerdal A. The experiences of nurses implementing the modified early warning score and a 24-hour on-call mobile intensive care nurse: an exploratory study Intensive Crit. Care Nurs. 34 2016 Jun 25 33 26631354
50 Spångfors M. Molt M. Samuelson K. National Early Warning Score: a survey of registered nurses' perceptions, experiences and barriers J. Clin. Nurs. 29 7–8 2020 Apr 1187 1194 31887247
51 Wood C. Chaboyer W. Carr P. How do nurses use early warning scoring systems to detect and act on patient deterioration to ensure patient safety? A scoping review Int. J. Nurs. Stud. 94 2019 Jun 166 178 31002971
52 Brekke I.J. Puntervoll L.H. Pedersen P.B. Kellett J. Brabrand M. The value of vital sign trends in predicting and monitoring clinical deterioration: a systematic review PLoS One 14 1 2019 e0210875
53 Vincent J.L. Einav S. Pearse R. Jaber S. Kranke P. Overdyk F.J. Improving detection of patient deterioration in the general hospital ward environment Eur. J. Anaesthesiol. 35 5 2018 May 325 333 29474347
54 Leenen J.P.L. Ardesch V. Patijn G. Remote home monitoring of continuous vital sign measurements by wearables in patients discharged after colorectal surgery: observational feasibility study JMIR Perioper Med 6 2023;May 5 e4
55 Leenen J.P.L. Rasing H.J.M. Kalkman C.J. Schoonhoven L. Patijn G.A. Process evaluation of a wireless wearable continuous vital signs monitoring intervention in 2 general hospital wards: mixed methods study JMIR Nurs 6 2023 May e44061
56 Cho K.J. Kwon O. Kwon J.M. Lee Y. Park H. Jeon K.H. Detecting patient deterioration using artificial intelligence in a Rapid response system Crit. Care Med. 48 4 2020 Apr e285 e289 32205618
57 Khanna A.K. Hoppe P. Saugel B. Automated continuous noninvasive ward monitoring: future directions and challenges Crit. Care 23 1 2019 May 194 31146792
58 Lyons P.G. Chen V. Sekhar T.C. McEvoy C.A. Kollef M.H. Govindan R. Clinician perspectives on barriers and enablers to implementing an inpatient oncology early warning system: a mixed-methods study JCO Clin Cancer Inform 7 2023 Jan e2200104
59 Richard A. Frank O. Schwappach D. Chief physicians' attitudes towards early warning score systems in Switzerland: results of a cross-sectional survey J. Eval. Clin. Pract. 24 2 2018 Apr 331 337 29114964
