
==== Front
Eur Heart J Digit Health
Eur Heart J Digit Health
ehjdh
European Heart Journal. Digital Health
2634-3916
Oxford University Press UK

10.1093/ehjdh/ztae058
ztae058
Original Article
AcademicSubjects/MED00200
Eurheartj/23
Eurheartj/24
Eurheartj/45
Eurheartj/47
The use of imputation in clinical decision support systems: a cardiovascular risk management pilot vignette study among clinicians
https://orcid.org/0000-0001-5465-4868
Haitjema Saskia Central Diagnostic Laboratory, University Medical Center Utrecht, Utrecht University, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands

Nijman Steven W J Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

Verkouter Inge Department Clinical Care & Research, Ortec B.V., Zoetermeer, The Netherlands

Jacobs John J L Department Clinical Care & Research, Ortec B.V., Zoetermeer, The Netherlands

Asselbergs Folkert W Department of Cardiology, Amsterdam Cardiovascular Sciences, Amsterdam University Medical Centre, University of Amsterdam, Amsterdam, The Netherlands
Department of Cardiology, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
Institute of Health Informatics, University College London, London, UK

Moons Karel G M Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

Beekers Ines Department Clinical Care & Research, Ortec B.V., Zoetermeer, The Netherlands

Debray Thomas P A Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands
Institute of Health Informatics, University College London, London, UK

Bots Michiel L Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, The Netherlands

Corresponding author. Tel: +31 (0)88 75 65 333, Email: s.haitjema@umcutrecht.nl
9 2024
10 8 2024
10 8 2024
5 5 572581
19 1 2024
06 5 2024
01 6 2024
26 8 2024
© The Author(s) 2024. Published by Oxford University Press on behalf of the European Society of Cardiology.
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial License (https://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact journals.permissions@oup.com.

Abstract

Aims

A major challenge of the use of prediction models in clinical care is missing data. Real-time imputation may alleviate this. However, to what extent clinicians accept this solution remains unknown. We aimed to assess acceptance of real-time imputation for missing patient data in a clinical decision support system (CDSS) including 10-year cardiovascular absolute risk for the individual patient.

Methods and results

We performed a vignette study extending an existing CDSS with the real-time imputation method joint modelling imputation (JMI). We included 17 clinicians to use the CDSS with three different vignettes, describing potential use cases (missing data, no risk estimate; imputed values, risk estimate based on imputed data; complete information). In each vignette, missing data were introduced to mimic a situation as could occur in clinical practice. Acceptance of end-users was assessed on three different axes: clinical realism, comfortableness, and added clinical value. Overall, the imputed predictor values were found to be clinically reasonable and according to the expectations. However, for binary variables, use of a probability scale to express uncertainty was deemed inconvenient. The perceived comfortableness with imputed risk prediction was low, and confidence intervals were deemed too wide for reliable decision-making. The clinicians acknowledged added value for using JMI in clinical practice when used for educational, research, or informative purposes.

Conclusion

Handling missing data in CDSS via JMI is useful, but more accurate imputations are needed to generate comfort in clinicians for use in routine care. Only then can CDSS create clinical value by improving decision-making.

Graphical Abstract

Graphical Abstract

CDSS
Imputation
Prediction models
Cardiovascular risk
U-Prevent
==== Body
pmcIntroduction

The use of prediction models in routine clinical care can provide actionable insights to improve clinical decision-making.1–5 Their current use is however limited and frustrated by missing data in the electronic health records (EHR) and the inability of the integrated clinical decision support system (CDSS) to appropriately handle these missing data.6

Missing data are a hallmark of routine care datasets that are used for the development and validation of prediction models. Consequently, recommended approaches (e.g. multiple imputation) for handling missing data have been developed and made part of reporting guidelines.7–12 However, these approaches are not suitable for deployed CDSS that are ‘live’ used in clinical practice.13 First, common imputation strategies are developed for valid statistical inference of, for example, model coefficients and not for application in single patients. Second, they require access to raw data from multiple individuals, which is not always possible in clinical practice.

Recently developed strategies such as joint modelling imputation (JMI) alleviate these issues as they allow for valid real-time risk prediction in routine clinical care when predictors in an individual patient are missing.14 In brief, JMI is suitable for individual predictions, does not need large amounts of raw data, and can achieve near-real time handling of missing data that makes it attractive for use in CDSS.14–16 Joint modelling imputation uses a two-step approach: first, population characteristics (i.e. means and covariance) are estimated in raw individual patient data and stored in the EHR-system; and second, the CDSS handles any missing data by drawing imputations using the stored population characteristics. However, the use of JMI requires careful interpretation by CDSS users, as imputations may be very uncertain (e.g. when most predictors are missing). Similarly, JMI is highly dependent on the data on which it is conditioned and likely requires local tailoring when trained on data different from the patients it may be used on. Furthermore, as the use of imputation is not widely adopted yet in clinical practice, assessment of the acceptance of using a CDSS combined with JMI in routine clinical care is needed.

Previous studies on imputation of missing data in clinical prediction models focused on the performance of the imputation method and did not study the acceptance in clinicians.17–19 In the current study, we assessed the acceptance of real-time imputation of missing patient data in a CDSS for cardiovascular risk management (CVRM). For this, we studied acceptance on three different axes (perceived clinical realism, comfortableness, and added clinical value) by means of a vignette study with three use cases among physicians as the intended users.

Methods

Participants and setting

The target population of the vignette study was the potential users of our cardiovascular CDSS developed in the UMC Utrecht; in other words, clinicians that work with patients in a field where cardiovascular risk prediction is used. Study participants were eligible if they were clinicians or had previous experience as a clinician [fully completed their studies to become a clinical specialist (i.e. no clinicians in training)] within the departments of cardiology, vascular medicine, or internal medicine at the UMC Utrecht, The Netherlands.2,13 Potential study participants were recruited via e-mail.

Study design

To simulate the handling of missing data as it would occur in clinical practice, a real-time imputation model was incorporated in a cardiovascular CDSS and evaluated using a vignette study to assess acceptance.

Joint modelling imputation

We used methods for performing JMI as developed in previous studies, though separately implemented in R to be used on clinical data.14–16 In general, JMI assumes multivariate normality. Any imputations are randomly sampled from a (multivariate) normal distribution, constructed from means and covariance in a training sample, which is conditioned on the observed predictor values (i.e. observed values in the patient). For binary variables, a logistic regression model is used to transform the drawn continuous values into discrete imputations. To implement JMI in clinical practice, the expectation (mean) of all variables in the data and its covariance matrix have to be available. On a patient-by-patient basis, we can then extract which variables are missing and which are not missing. From the variables that are not missing, we save the observed values in a separate vector. Then, we estimate the conditional multivariate normal distribution using the provided expectations, covariance matrix, dependent variables (i.e. names of the missing variable), the given non-missing variables, and all observed values. Imputations can then be drawn from the distribution. If only a single imputation is desired, the most likely value is given. In this vignette study, single imputations are given.

Cardiovascular clinical decision support system

We made use of a modified version of the medical device directive-certified U-Prevent cardiovascular CDSS that was built by Ortec and adapted for this study to facilitate auto data fill-in and recording.20 The purpose of U-Prevent is to allow the application of existing risk calculators in routine clinical care via the use of an incorporated CDSS. As an extension to the existing CDSS, we added real-time imputation (JMI) that was trained on Utrecht Cardiovascular cohort–Secondary Manifestation of ARTerial disease (UCC-SMART) study.1,21 As we aimed to evaluate the use of a CVRM prediction model for secondary prevention [i.e. Second Manifestations of ARTerial disease (SMART) risk score], all SMART risk score variables were included in the imputation model. In addition to the imputation method, the 95% confidence interval of the imputed value was incorporated in the user interface to communicate variable uncertainty to the clinician. For binary variables, the confidence intervals of the imputed values were presented on a probability scale. A preview of the extended CDSS dashboard and an example of the SMART risk score in U-Prevent can be found in Figure 1.

Figure 1 Example of risk profile via CDSS. SMART, Second Manifestations of ARTerial disease; CRP, C-reactive protein; HDL, high-density lipoprotein; LDL, low-density lipoprotein.

Vignettes

To mimic patient care, three vignettes were designed using available data from the Utrecht Cardiovascular cohort.22 A vignette describes a potential scenario, including information about the patient, in which the dashboard could be used (e.g. patient scheduled for a consult). We selected patient information to be made missing based on a combination of: variable types (i.e. binary and continuous), clinical burden on the workflow, and clinical interpretation (Table 1 and Figure 2). All vignettes deliberately had a resulting predicted risk of around 15%, which is the general threshold for treating cardiovascular disease (CVD) when using the prediction model SMART.23 To be able to summarize the interpretation of our vignettes, we scored them into self-made categories of clinical burden and expected ease of interpretation as follows: three categories for burden on the clinical workflow: low (i.e. when a phone call to the patient suffices), medium (i.e. if the patient is in the consultation room, the clinician can easily measure the variable), and high (i.e. the variable requires a lab test). Similarly, expected ease of clinical interpretation was categorized specifically to look at how well the resulting imputations, and predicted risk, may be interpreted by the user: easy (i.e. when variables are in the same unit scale and the expected uncertainty of the imputed values is low), moderate (i.e. when confidence intervals are wider), and difficult (i.e. when predictors are presented as probabilities).

Figure 2 Overview of vignette patients. SBP, systolic blood pressure; hs-CRP, high-sensitivity C-reactive protein; TC, total cholesterol; HDL, high-density lipoprotein cholesterol; LDL, lower-density lipoprotein cholesterol.

Table 1 Summary of vignettes and categories

	Missing variables	Variable type	Clinical burden	Expected ease of clinical interpretation	
Scenario 1	SBP	Continuous	Medium	Easy	
Hs-CRP	Continuous	High	
Scenario 2	Hs-CRP	Continuous	High	Moderate	
Years since first CVD event	Continuous	Low	
Cholesterol levels (LDL/HDL/TC)	Continuous	High	
Scenario 3	SBP	Continuous	Medium	Difficult	
Diabetes	Binary	Low	
Anti-thrombotic treatment	Binary	Low	
SBP, systolic blood pressure; hs-CRP, high-sensitivity C-reactive protein; CVD, cardiovascular disease; LDL, lower-density lipoprotein; HDL, high-density lipoprotein; TC, total cholesterol.

In short, scenario 1 was the most prevalent and the easiest to interpret, scenario 2 the most extreme in terms of missingness, and scenario 3 the easiest to ascertain in clinical practice yet with imputations that were difficult to interpret.

Questionnaire

Each study participant received a written explanation about the use of the CDSS and a protocol for its use in practice. Participants were asked to use the CDSS in combination with the implemented JMI and to fill out a questionnaire. Vignettes were presented to the participants in fixed order (scenarios 1, 2, and 3). For each vignette, a new questionnaire was provided at three different points in time and of potential relevance: (i) before imputation, (ii) after imputation, and (iii) after unveiling the true values of the missing patient characteristics (Figure 3). Each study participant therefore filled in nine questionnaires, if they fully completed the study.

Figure 3 Overview of survey structure and design. CI, confidence interval; JMI, joint modelling imputation.

We studied acceptance of the CDSS with incorporated JMI scrutinizing whether imputed values were perceived as clinically realistic, the users were comfortable with using them, and whether they provided added value (including the potential call to action).

Specifically, to evaluate clinical realism, participants were asked (after imputation) whether the imputed values were perceived as clinically realistic and whether the imputed values fell in their expected ranges (this was further evaluated by comparing the participants guess and the real predictor value). Similarly, to evaluate comfortableness, participants were asked (after imputation and after unveiling true values) whether they were comfortable using these imputed values for predicting cardiovascular disease risk in the patient. We furthermore assessed whether confidence intervals affected comfortableness. The added value of the imputed values was assessed by evaluating the potential to relieve clinical burden and the potential of direct the clinical benefit (measured before imputation, after imputation and after unveiling true values). As a baseline for calculating the added value of imputation, participants were asked what they thought the missing value would be (minimal clinical burden, no clinical benefit of imputation). Participants were subsequently asked whether they would have measured the missing predictor value (current burden on the workflow that could be relieved by imputation) and subsequently which potential use cases for JMI they would consider using in clinical practice (direct benefit for clinical situations). An overview which questionnaire these questions were asked can be found in Figure 3. Free-text comments were evaluated on their relevance to the study objectives and summarized in the Supplementary material online, tablesS14 and S15. Likewise, the complete list of all survey questions can be found in Supplementary material online, tablesS2–S10.

Statistical analysis

Since this study consisted of an early investigative evaluation of JMI integrated into a CDSS, the analysis was primarily descriptive, using summary statistics to describe a user’s view on imputation, prediction, and CDSS use. The survey was not designed to statistically test the effect on decision-making nor to evaluate the use of the employed prediction model (i.e. SMART). All analyses were performed using R (Version 4.1.0).

Results

Characteristics of the study participants

A total of 17 clinicians participated in the pilot, of which 13 completed all 9 questionnaires. All were between 30 and 50 years old, more than half had >6 years of clinical experience in their field (53%), and the prevailing clinical specialty was cardiology (41%). Further detailed information about the participants can be found in Supplementary material online, tableS1.

Imputation has a potential to relieve the burden of missing values in the clinical workflow

Clinical estimations of the missing predictor values, measured before imputation as a baseline for establishing benefit, were generally quite close to the real missing predictor values (Figure 4). However, all predictors, except high-sensitivity C-reactive protein (hs-CRP), would still have been measured if the patient was directly available in the hospital. Predictors that had low clinical burden, namely years since first CVD event, use of anti-thrombotic medication, and diabetes, would be obtained via contacting the patient directly. The only variable(s) from those with medium or high clinical burden that were deemed important enough for the patient to come to the hospital were cholesterol levels.

Figure 4 Comparison between clinician’s guess, real and imputed values. SBP, systolic blood pressure; hs-CRP, high-sensitivity C-reactive protein; CVD, cardiovascular disease; TC, total cholesterol; HDL, high-density lipoprotein cholesterol; LDL, lower-density lipoprotein cholesterol.

Imputed values were overall perceived as very realistic

The type of variable did not influence realism as both continuous, such as systolic blood pressure (SBP) (100%), and categorical, such as diabetes or anti-thrombotic treatment (both 77%), predictors were rated similar (reflected by scenario 3) (Table 2). Of particular importance, perceived realism was considered high for predictors with a medium and high clinical burden. For predictors with low clinical burden such as years since first CVD event (47%), diabetes (77%), or anti-thrombotic treatment (77%), perceived clinical realism was inconsistent.

Table 2 Summary of acceptance measures for the imputed missing variables in the three scenarios (shows percentage of participants that said yes)

Scenarios	Missing variables	Clinical burden	Expected ease of clinical Interpretation	Perceived clinical realism	Comfortableness	Added value	
Are the imputed values clinically realistic?	Do the clinically realistic values fall in line with expectations?	Is the difference in predicted risks, between true and imputed predictor values, acceptable?	Are you comfortable with predicting risk with these missing predictor values?	Are you comfortable with predicting risk after seeing true values?	Would you use JMI in similar patients?	Are you motivated to measure missing variables?	
Before (i), after (ii), or after seeing true values (iii)	ii	ii	iii	ii	iii	iii	i, ii	
S1	SBP	Medium	Easy	77%	46%	80%	67%	60%	87%	71%	
Hs-CRP	High	100%	67%	35%	
S2	Time since first CVD event	Low	Moderate	47%	29%	23%	29%	23%	77%	80%	
Hs-CRP	High	93%	93%	13%	
Total cholesterol	High	93%	86%	93%	
HDL-cholesterol	High	93%	93%	93%	
LDL-cholesterol	High	93%	60%	93%	
S3	Anti-thrombotic medication	Low	Difficult	77%	80%	92%	15%	54%	85%	69%	
Diabetes	Low	77%	100%	69%	
SBP	Medium	100%	54%	77%	
JMI, joint modelling imputation; SBP, systolic blood pressure; hs-CRP, high-sensitivity C-reactive protein; HDL, high-density lipoprotein; LDL, low-density lipoprotein; CVD, cardiovascular disease.

Overall clinicians are uncomfortable with predicted risks using imputed values

The level of comfortableness was, altogether, low (Table 2). When many variables were missing, the difference in predicted risk was deemed unacceptable (23%) (reflected by scenario 2) when compared to other scenarios (80% and 92% for scenarios 1 and 3, respectively). Solely when few, exclusively continuous predictors, were missing, participants were comfortable with predicting risk (67% in scenario 1). With too many predictors missing, independent of the clinical ease of predictors, few participants were comfortable (29%). Only when predictor variables were mostly binary (as reflected in scenario 3), the level of comfortableness changed substantially after revealing the true predictor values (from 18% to 54%). When confidence intervals were shown, comfortableness of physicians for predicting risk decreased, except for the vignette where comfortableness levels were already low (18% for vignette 3).

Imputation use cases do not include treatment purposes directly benefitting clinical situations

Across all scenarios, participants would use JMI in similar patients (Table 2). Use cases for JMI, common for each scenario, were learning purposes, informing of the patient and the determination of further research (e.g. additional clinical or diagnostic tests). Treatment purposes (e.g. to determine what treatment to start) were generally not chosen much as a use-case for JMI (18%, 8%, and 15% for each scenario, respectively).

A summary of all individual responses specifically to the axes of perceived realism, comfortableness, and added value can be found in Supplementary material online, tablesS11–S13.

Discussion

In this pilot vignette study, we evaluated the acceptance of using a real-time imputation method (JMI) built into an existing cardiovascular CDSS. Generally, imputed predictor values were found to be perceived clinically realistic, yet comfortableness was low, especially when many predictors were missing. Moreover, clinicians felt consistently motivated to measure any missing predictor variables, regardless of clinical ease, and would not use the CDSS with imputed values to guide treatment, impeding direct added value for clinical practice. To our knowledge, this is the first study to not only develop or use an imputation method, but also study its acceptance by clinicians.

Impact of imputation depends on type of variable

The view on comfortableness in predicting risk seemed dependent on the variable type of the missing predictor. Imputed continuous predictors resulted in most participants being comfortable, assuming not too many predictors were missing. In contrast, when binary predictors were missing, comfortableness was low. Possibly this is because these binary predictors are imputed with percentages, rather than a dichotomized imputed value and clinical interpretation is difficult (this was also mentioned in the free-text comments). Likewise, the use of a confidence interval (CI) for the imputed value was found to deteriorate the interpretability of the imputed value and the comfortableness in predicting cardiovascular risk. This indicates an important role for the CI as a signal whether or not to trust the imputed value. Across all three vignettes, in the free-text comments, the primary concerns were CIs that were too wide or unrealistic (e.g. negative numbers). As a result, in its current form, the imputation method may primarily be perceived as useful by users when it is used to impute continuous variables and not too many predictors are missing. However, a user’s comfortableness was not necessarily found to be reflective of imputation performance according to the study participants, as the differences in predicted risk are often still found to be acceptable (as reflected by the third scenario). As such, even though the imputation method may perform adequately, a user may expect it to underperform under certain circumstances and as such be less comfortable with using the imputation method.

Impact of imputation depends on missing variable and its relation with cardiovascular disease risk

The view on how useful JMI would be in clinical practice was found to be likely dependent on the importance of the predictor (as indicated by free-text comments). Cholesterol, for example, was noted as an important predictor and study participants specifically stated that imputation could not be relied upon. In comparison, hs-CRP was not relied upon much for treatment and thus study participants were not concerned when it was imputed.

Strengths and limitations

The number of prediction algorithms that are being developed, validated, and proposed for use in clinical practice as CDSSs has greatly increased over time. Previous research provided evidence for improved decision-making using validated prediction models in CDSS.24–30 Missing data, however, seem to be unanimously omitted from any considerations when using the CDSS in real-time. Our pilot vignette study is, to our knowledge, the first to assess clinical acceptance of imputation methods for missing data in CDSS. Previous studies on imputation of missing data in clinical prediction models focused on the performance of the imputation method and did not study the acceptance in clinicians.17–19

We carefully designed our vignette study to mimic clinical practice, for example using real world data from a clinical cohort study as starting point for our vignettes, and presenting the imputed results in a user interface that is well known for our clinicians. However, a vignette study is still artificial as compared to a clinical setting. Yet, we feel that in light of the current regulatory framework that prohibits the use of non-CE-marked Medical Device Software in clinical care, a vignette study was the best option to study our research question. Joint modelling imputation is a possible solution to the problem of missing data. Another solution for imputation in the absence of accessibility of directly identifiable data in the electronic healthcare system as a basis for imputation could be an anonymous or synthetic dataset that is built into the CDSS itself. Acceptance of innovation is known to be influenced by many things, among which hospital culture, which may threaten the external validity of our results. Lastly, our sample size is modest. We therefore may have been unable to capture nuanced differences across various scenarios of missing data or between age categories or specialties. More research is needed to address these issues.

Future perspectives

New technological developments, including imputation strategies, carry the potential to support physicians during clinical decision-making. Systematic reviews on the use and acceptance of CDSS almost all indicated that, in order to facilitate acceptance of CDSS among healthcare professionals, one needs to integrate end-users in the early stages of its development, to offer needs-adjusted training for the use and providing adequate infrastructure.31 Kaiser et al.32 provided a framework showing the various barriers that need to be considered to ensure the acceptance of all of stakeholders (Table 3). Only then successful implementation of a CDSS is to be expected. Our study is among the few studies on showing how clinicians deal with missing data and accept imputed information in a CDSS, which touches upon aspects of trust, experience training and culture.

Table 3 Implementation factors and barriers for clinical decision support systems (according to Kaiser et al.32)

Process	Resources and capabilities	Strategies	
Ease of use	Trust in system	Endorsement/championing	
Integration into workflow	IT infrastructure	Organizational culture	
Work time and pressure	Perceived usefulness	Organizing/planning	
Internal communication	Experience		
Technical support	Training and supervision		
System reliability	Workforce competencies		
	Financial resources		

In conclusion, we showed that missing data in CDSSs are useful, but more accurate imputations are needed to generate the necessary comfort in clinicians for use in routine clinical care. This is one of the requisites for a CDSS to create clinical value by improving clinical decision-making.

Supplementary Material

ztae058_Supplementary_Data

Acknowledgements

This study was conducted on behalf of the Utrecht Cardiovascular Cohort-CardioVascular Risk Management (UCC-CVRM) study group. Members of the UCC-CVRM Study group: F.W. Asselbergs, Department of Cardiology; G.J. de Borst, Department of Vascular Surgery; M.L. Bots (chair), Julius Center for Health Sciences and Primary Care; S. Dieleman, Division of Vital Functions (anesthesiology and intensive care); M.H. Emmelot, Department of Geriatrics; P.A. de Jong, Department of Radiology; A.T. Lely, Department of Obstetrics/Gynecology; I.E. Hoefer, Central Diagnostic Laboratory; N.P. van der Kaaij, Department of Cardiothoracic Surgery; Y.M. Ruigrok, Department of Neurology; M.C. Verhaar, Department of Nephrology & Hypertension; F.L.J. Visseren, Department of Vascular Medicine, University Medical Center Utrecht, and Utrecht University. The authors would like to thank Frank Tabak and Steven van Dijk of the Health software development team of ORTEC for creating the research site for this purpose.

Supplementary material

Supplementary material is available at European Heart Journal – Digital Health.

Author contribution

T.P.A.D., K.G.M.M., F.W.A., M.L.B., J.J.L.J., I.B., I.V., and S.W.J.N. conceived of the presented study design. S.W.J.N. contacted the potential study participants. J.J.L.J., I.B., and I.V. provided the research site of U-Prevent and its expansion for data imputation and data collection for this pilot study. S.W.J.N., S.H., I.B., and T.P.A.D. contributed to the interpretation of the results. S.W.J.N. and S.H. wrote the draft of the manuscript. All authors provided critical feedback and helped shape the research, analysis, and manuscript. All authors approved the final manuscript.

Funding The UCC-CVRM is primarily financed by the UMC Utrecht. A grant from the Netherlands Organization for Health Research and Development (#8480-34001) was obtained to develop feedback procedures. UCC website: www.umcutrecht.nl/ucc (in Dutch). Contact information UCC: ucc@umcutrecht.nl. S.W.J.N. is supported by a public–private study grant of the Netherlands Heart Foundation for the CVRM-IMPROVE project (#2018B006). This research project is financed by the PPP Allowance made available by Top Sector Life Sciences & Health to Netherlands Heart Foundation to stimulate public–private partnerships. T.P.A.D. is supported by the Netherlands Organisation for Health Research and Development (#91617050). All other authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

All data are incorporated into the article and its online supplementary material.
==== Refs
References

1 Visseren FLJ , MachF, SmuldersYM, CarballoD, KoskinasKC, BäckM, et al 2021 ESC guidelines on cardiovascular disease prevention in clinical practice. Eur Heart J 2021;42 :3227–3337.34458905
2 Fraccaro P , van der VeerS, BrownB, ProsperiM, O’DonoghueD, CollinsGS, et al An external validation of models to predict the onset of chronic kidney disease using population-based electronic health records from Salford, UK. BMC Med 2016;14 :104.27401013
3 van Rijn MHC , van de LuijtgaardenM, van ZuilenAD, BlankestijnPJ, WetzelsJFM, DebrayTPA, et al Prognostic models for chronic kidney disease: a systematic review and external validation. Nephrol Dial Transplant 2021;36 :1837–1850.33051669
4 Kappen TH , van KleiWA, van WolfswinkelL, KalkmanCJ, VergouweY, MoonsKGM. Evaluating the impact of prediction models: lessons learned, challenges, and recommendations. Diagn Progn Res 2018;2 :11.31093561
5 Ratna MB , BhattacharyaS, AbdulrahimB, McLernonDJ. A systematic review of the quality of clinical prediction models in in vitro fertilisation. Hum Reprod 2020;35 :100–116.31960915
6 Groenhof TKJ , BotsML, BrandjesM, JacobsJJL, GrobbeeDE, van SolingeWW, et al A computerised decision support system for cardiovascular risk management ‘live’ in the electronic health record environment: development, validation and implementation—the Utrecht Cardiovascular Cohort Initiative. Neth Heart J 2019;27 :435–442.31372838
7 Collins GS , ReitsmaJB, AltmanDG, MoonsK. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMC Med 2015;13 :1.25563062
8 Allotey PA , HarelO. Multiple imputation for incomplete data in environmental epidemiology research. Curr Environ Health Rep 2019;6 :62–71.31090043
9 Carpenter JR , KenwardMG. Multiple imputation and its application. 1st ed. Chichester, West Sussex, UK: John Wiley & Sons; 2013. p1.
10 Janssen KJM , DondersART, HarrellFE, VergouweY, ChenQ, GrobbeeDE, et al Missing covariate data in medical research: to impute is better than to ignore. J Clin Epidemiol 2010;63 :721–727.20338724
11 Janssen KJM , VergouweY, DondersART, HarrellFE, ChenQ, GrobbeeDE, et al Dealing with missing predictor values when applying clinical prediction models. Clin Chem 2009;55 :994–1001.19282357
12 Donders ART , van der HeijdenGJMG, StijnenT, MoonsKGM. Review: a gentle introduction to imputation of missing values. J Clin Epidemiol 2006;59 :1087–1091.16980149
13 Van Buuren S . Flexible imputation of missing data. 2nd ed. New York: CRC Press; 2018.
14 Nijman SWJ , HooglandJ, GroenhofTKJ, BrandjesM, JacobsJJL, BotsML, et al Real-time imputation of missing predictor values in clinical practice. Eur Heart J Digit Health 2021;2 :154–164.36711167
15 Hoogland J , van BarreveldM, DebrayTPA, ReitsmaJB, VerstraelenTE, DijkgraafMGW, et al Handling missing predictor values when validating and applying a prediction model to new patients. Stat Med 2020;39 :3591–3607.32687233
16 Nijman SWJ , GroenhofTKJ, HooglandJ, BotsML, BrandjesM, JacobsJJL, et al Real-time imputation of missing predictor values improved the application of prediction models in daily practice. J Clin Epidemiol 2021;134 :22–34.33482294
17 Kim M , KimTH, KimD, LeeD, KimD, HeoJ, et al In-advance prediction of pressure ulcers via deep-learning-based robust missing value imputation on real-time intensive care variables. J Clin Med 2023;13 :36.38202043
18 Cardoso P , DennisJM, BowdenJ, ShieldsBM, McKinleyTJ; the MASTERMIND Consortium. Dirichlet process mixture models to impute missing predictor data in counterfactual prediction models: an application to predict optimal type 2 diabetes therapy. BMC Med Inform Decis Mak 2024;24 :12.38191403
19 Zhang Y , GaoZ, WittrupE, GryakJ, NajarianK. Increasing efficiency of SVMp+ for handling missing values in healthcare prediction. PLOS Digit Health 2023;2 :e0000281.37384608
20 Visseren FLJ , DorresteijnJAN, van der GraafY. U-prevent u bent “in control.” 2018. https://www.u-prevent.nl/nl-NL (7 October 2019).
21 Simons PCG , AlgraA. Second Manifestations of ARTerial disease (SMART) study: rationale and design. Eur J Clin Epi 1999;9 :773–781.
22 Asselbergs FW , VisserenFL, BotsML, de BorstGJ, BuijsroggeMP, DielemanJM, et al Uniform data collection in routine clinical practice in cardiovascular patients for optimal care, quality control and research: the Utrecht Cardiovascular Cohort. Eur J Prev Cardiol 2017;24 :840–847.28128643
23 Dorresteijn JAN , VisserenFLJ, WassinkAMJ, GondrieMJA, SteyerbergEW, RidkerPM, et al Development and validation of a prediction rule for recurrent vascular events based on a cohort study of patients with arterial disease: the SMART risk score. Heart 2013;99 :866–872.23574971
24 Goehler A , MooreC, Manne-GoehlerJM, ArangoJ, AmatoD, FormanL, et al Clinical decision support for ordering CTA-PE studies in the emergency department—a pilot on feasibility and clinical impact in a tertiary medical center. Acad Radiol 2019;26 :1077–1083.30389307
25 Gavrielides MA , MillerM, HagemannIS, AbdelalH, AlipourZ, ChenJ-F, et al Clinical decision support for ovarian carcinoma subtype classification: a pilot observer study with pathology trainees. Arch Pathol Lab Med 2020;144 :869–877.31816269
26 Sim LLW , BanKHK, TanTW, SethiSK, LohTP. Development of a clinical decision support system for diabetes care: a pilot study. PLoS One 2017;12 :e0173021.28235017
27 Keenan GM , LopezKD, YaoY, SousaVEC, StifterJ, FebrettiA, et al Toward meaningful care plan clinical decision support: feasibility and effects of a simulated pilot study. Nurs Res 2017;66 :388–398.28858147
28 Gudmundsson HT , HansenKE, HalldorssonBV, LudvikssonBR, GudbjornssonB. Clinical decision support system for the management of osteoporosis compared to NOGG guidelines and an osteology specialist: a validation pilot study. BMC Med Inform Decis Mak 2019;19 :27.30709348
29 Menon S , TarragoR, CarlinK, WuH, YonekawaK. Impact of integrated clinical decision support systems in the management of pediatric acute kidney injury: a pilot study. Pediatr Res 2021;89 :1164–1170.32620006
30 Topaz M , TrifilioM, MaloneyD, Bar-BacharO, BowlesKH. Improving patient prioritization during hospital-homecare transition: a pilot study of a clinical decision support tool. Res Nurs Health 2018;41 :440–447.30203417
31 Lambert SI , MadiM, SopkaS, LenesA, StangeH, BuszelloCP, et al An integrative review on the acceptance of artificial intelligence among healthcare professionals in hospitals. NPJ Digit Med 2023;6 :111.37301946
32 Kaiser C , AckerhansS, HuynhT, DenzingerJ, RieglerM, DumitrescuD, et al Challenges in the implementation of disruptive innovations in health care organizations. Stud Health Technol Inform 2023;309 :145–149.37869828
