
==== Front
PLoS One
PLoS One
plos
PLOS ONE
1932-6203
Public Library of Science San Francisco, CA USA

10.1371/journal.pone.0297703
PONE-D-23-43080
Research Article
Medicine and Health Sciences
Health Care
Health Information Technology
Clinical Decision Support Systems
Computer and Information Sciences
Information Technology
Health Information Technology
Clinical Decision Support Systems
Medicine and Health Sciences
Health Care
Health Care Providers
Physicians
People and Places
Population Groupings
Professions
Medical Personnel
Physicians
Medicine and Health Sciences
Epidemiology
Medical Risk Factors
Research and Analysis Methods
Mathematical and Statistical Techniques
Statistical Methods
Forecasting
Physical Sciences
Mathematics
Statistics
Statistical Methods
Forecasting
Medicine and Health Sciences
Geriatrics
People and Places
Population Groupings
Age Groups
Adults
Elderly
Biology and Life Sciences
Neuroscience
Cognitive Science
Cognitive Psychology
Decision Making
Biology and Life Sciences
Psychology
Cognitive Psychology
Decision Making
Social Sciences
Psychology
Cognitive Psychology
Decision Making
Biology and Life Sciences
Neuroscience
Cognitive Science
Cognition
Decision Making
Computer and Information Sciences
Software Engineering
Computer Software
Engineering and Technology
Software Engineering
Computer Software
Development of the ADFICE_IT clinical decision support system to assist deprescribing of fall-risk increasing drugs: A user-centered design approach
The ADFICE_IT clinical decision support system and deprescribing fall-risk increasing drugs
https://orcid.org/0000-0002-3641-6425
Groos Sara S. Data curation Formal analysis Methodology Project administration Writing – original draft 1 2 *
de Wildt Kelly K. Data curation Formal analysis Methodology Project administration Writing – review & editing 1 2
van de Loo Bob Data curation Methodology Project administration Writing – review & editing 1 2 3
Linn Annemiek J. Project administration Supervision Writing – review & editing 4
https://orcid.org/0000-0002-2679-8095
Medlock Stephanie Methodology Software Supervision Writing – review & editing 2 5 6
https://orcid.org/0000-0001-6793-9600
Shaw Kendrick M. Software Writing – review & editing 6 7 8
Herman Eric K. Software Writing – review & editing 9
Seppala Lotta J. Data curation Methodology Project administration Writing – review & editing 1 2
Ploegmakers Kim J. Data curation Methodology Project administration Writing – review & editing 1 2
van Schoor Natasja M. Conceptualization Funding acquisition Project administration Supervision Writing – review & editing 2 3
van Weert Julia C. M. Conceptualization Supervision Writing – review & editing 4
https://orcid.org/0000-0002-6477-6209
van der Velde Nathalie Conceptualization Funding acquisition Project administration Supervision Writing – review & editing 1 2
1 Internal Medicine, Section of Geriatric Medicine, Amsterdam UMC Location University of Amsterdam, Amsterdam, the Netherlands
2 Amsterdam Public Health Research Institute, Amsterdam, the Netherlands
3 Epidemiology and Data Science, Amsterdam UMC Location Vrije Universiteit Amsterdam, Amsterdam, the Netherlands
4 Amsterdam School of Communication Research/ASCoR, University of Amsterdam, Amsterdam, the Netherlands
5 Department of Medical Informatics, Amsterdam UMC Location University of Amsterdam, Amsterdam, the Netherlands
6 Stichting Open Electronics Lab, Maarssen, The Netherlands
7 Department of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Boston, MA, United States of America
8 Harvard Medical School, Boston, MA, United States of America
9 Commons Caretakers BV, Amsterdam, The Netherlands
Baysal Mehmet Editor
Bursa Ali Osman Sonmez Oncology Hospital, TÜRKIYE
Competing Interests: The authors have declared that no competing interests exist.

* E-mail: s.s.groos@amsterdamumc.nl
5 9 2024
2024
19 9 e029770310 1 2024
20 8 2024
© 2024 Groos et al
2024
Groos et al
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Introduction

Deprescribing fall-risk increasing drugs (FRIDs) is promising for reducing the risk of falling in older adults. Applying appropriate deprescribing in practice can be difficult due to the outcome uncertainties associated with stopping FRIDs. The ADFICE_IT intervention addresses this complexity with a clinical decision support system (CDSS) that facilitates optimum deprescribing of FRIDs by using a fall-risk prediction model, aggregation of deprescribing guidelines, and joint medication management.

Methods

The development process of the CDSS is described in this paper. Development followed a user-centered design approach in which users and experts were involved throughout each phase. In phase I, a prototype of the CDSS was developed which involved a literature and systematic review, European survey (n = 581), and semi-structured interviews with clinicians (n = 19), as well as the aggregation and testing of deprescribing guidelines and the development of the fall-risk prediction model. In phase II, the feasibility of the CDSS was tested by means of two usability testing rounds with users (n = 11).

Results

The final CDSS consists of five web pages. A connection between the Electronic Health Record allows for the retrieval of patient data into the CDSS. Key design requirements for the CDSS include easy-to-use features for fast-paced clinical environments, actionable deprescribing recommendations, information transparency, and visualization of the patient’s fall-risk estimation. Key elements for the software include a modular architecture, open source, and good security.

Conclusion

The ADFICE_IT CDSS supports physicians in deprescribing FRIDs optimally to prevent falls in older patients. Due to continuous user and expert involvement, each new feedback round led to an improved version of the system. Currently, a cluster-randomized controlled trial with process evaluation at hospitals in the Netherlands is being conducted to test the effect of the CDSS on falls. The trial is registered with ClinicalTrials.gov (date; 7-7-2022, identifier: NCT05449470).

http://dx.doi.org/10.13039/501100001826 ZonMw 848017004 Amsterdams Universiteitsfonds: Gepersonaliseerde Medicatieaanpassing bij Oudere Vallers http://dx.doi.org/10.13039/100000002 National Institutes of Health T32-GM007592 https://orcid.org/0000-0001-6793-9600
Shaw Kendrick M. The ADFICE_IT project is supported by funding from the Netherlands Organization for Health Research and Development (ZonMw, Grant 848017004), The Hague and the Amsterdams Universiteitsfonds: Gepersonaliseerde Medicatieaanpassing bij Oudere Vallers. KMS was supported by US NIH grant T32-GM007592. ZonMw: https://www.zonmw.nl/en Amsterdams Universiteitsfonds: https://www.auf.nl/en NIH: https://www.nih.gov/grants-funding The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Data AvailabilityAll relevant data are within the manuscript.
Data Availability

All relevant data are within the manuscript.
==== Body
pmcIntroduction

Falls and fall-related injuries among older adults are a growing major public health problem [1]. In 2017, 11.7 million older adults in Western Europe requested medical treatment for an injury, of which 8.4 million were fall-related [2]. Injurious falls may result in admission to long term care, loss of independence, reduced mobility, fear of falling, and social isolation, significantly reducing the quality of life for older adults [3–5]. As a result, falls place a significant financial burden on healthcare systems. In Western countries, it is estimated that up to 1.5 percent of the total healthcare expenditures are attributed to fall-related medical care costs [3, 6].

A prominent risk factor for falls in older adults is the use of certain medication classes known as fall-risk increasing drugs (FRIDs). The use or combined use of FRIDs, such as psychotropics and cardiovascular drugs, is associated with adverse effects including orthostatic hypertension, syncope, sedation, and dizziness that can cause accidental falls in older adults [7–10]. Therefore, appropriate deprescribing of FRIDs is recommended for lowering an older adult’s risk of incurring a medication-related fall [11]. Nevertheless, previous studies suggest that deprescribing approaches have yet to be optimized, as current approaches do not sufficiently address the complexity of FRIDs deprescribing for physicians [12, 13].

Deprescribing FRIDs is highly complex due to healthcare professional, patient, cultural and organizational reasons. For example first, physicians themselves may perceive difficulties with which, how and when a FRID or combination of FRIDs should be safely deprescribed. This results in the reluctancy to deprescribe, a phenomena that is especially prominent in the treatment of older patients with polypharmacy and multimorbidity [14, 15]. Second, the unique risk profiles of these patients can greatly increase the complexity of FRIDs deprescribing due to the weighing of competing clinical practice guidelines [15, 16]. As a result, physicians may have a tendency to overestimate the expected benefits of medications (i.e., effective treatment of the chronic condition) and underestimate the potential harms (i.e., an injurious fall) in these patients [17]. Third, physicians may also experience deprescribing reluctance from the patient. Research suggests that low medication-related knowledge in geriatric patients, such as the management of medications, can negatively influence appropriate deprescribing [18].

One way to better support physicians in the deprescribing of FRIDs is by means of a clinical decision support system (CDSS). Such a system has the ability to link individual patient characteristics to a computerized clinical knowledge base which uses information from guidelines to generate patient-specific recommendations back to the physician. These recommendations can subsequently be discussed together with the patient [19, 20]. In the context of FRIDs, a CDSS has the potential to support physicians through a structured deprescribing approach by first signaling which medications pose a risk to the individual patient, and subsequently advising how and when to safely deprescribe each medication (i.e., through guideline integration). In turn, the decision to deprescribe certain medications can be discussed together with the patient to foster joint medication management between physicians and patients. Such shared decision-making has been shown to improve medication-related knowledge and medication adherence in older patients, and could thus enhance the effectiveness of fall preventive care in these patients [21].

Previous studies suggest that medication-related CDSS can improve care outcomes for older patients in a variety of contexts (e.g., inappropriate medication use by patients and polypharmacy, inappropriate prescribing by physicians, falls) [22–24]. However, these studies have not used the potential of a CDSS in deprescribing optimally, such as incorporating data-driven methods, like prediction models, in the system’s clinical knowledge base [24]. Prediction models combine data for multiple risk factors in order to calculate the risk of a future outcome, and help inform subsequent decision making [25]. Thus, in the context of deprescribing optimally, such models could serve as an adjunct to decision-making by allowing the physician and patient to weigh the various treatment options on the basis of the patient’s risk of falling within 12 months. Moreover, such estimates of fall risk could help patients be more aware of their risk of falling and as such motivate them to follow the advice recommended by the physician.

The acceptance of CDSSs among physicians is still hindered by a number of barriers, such as insufficient knowledge with system use, time-consuming, alert fatigue and poor integration into workflow [24, 26–28]. A possible explanation for the considerable number of usability barriers is the lack of involvement by physicians during the development of these systems. A user-centered design approach is an iterative method that involves users of a system in each stage of the development process. Such an approach has been found to enhance the ease of use and usefulness of clinical study tools–the two important determinants that can influence technology adoption by clinicians [29, 30].

The current paper aims to outline how user and expert insights were used to inform the development of a data-driven CDSS that aims to provide optimal support for physicians during the deprescribing of FRIDs by generating a personalized fall-risk estimate and guideline-based medication advice tailored to the health conditions of each individual patient. Following a user-centered design approach, we showcase how users were successfully involved in different stages of the development process to increase the system’s acceptance and adoption in its intended clinical care setting later on. This CDSS is part of the ADFICE_IT (Alerting on adverse Drug reactions: Falls prevention Improvement through developing a Computerized clinical support system: Effectiveness of Individualized medicaTion withdrawal) intervention, a novel deprescribing intervention that aims to prevent medication-related falls in older adults by means of a CDSS for clinicians and an online portal for patients [31].

Materials and methods

Ethics statement

The Medical Ethics Research Committee of the Amsterdam University Medical Center (Amsterdam UMC, location University of Amsterdam; W19_310 # 19.368) declared that the Medical Research Involving Human Subjects Act did not apply to this study. All study participants gave written informed consent prior to data collection.

Medical research council framework

The overall ADFICE_IT intervention is developed and evaluated following the four phases of the Medical Research Council framework (MRC): (I) the development, (II) the feasibility, (III) the implementation, and (IV) the evaluation phase. The MRC is a guiding theoretical framework for developing, pilot-testing, implementing and evaluating complex health interventions [32]. This paper summarizes the (I) development and (II) feasibility phases for our CDSS. An overview of the studies conducted in these phases is provided in Table 1.

10.1371/journal.pone.0297703.t001 Table 1 The development process of the ADFICE_IT CDSS.

MRC Phase I: Development	
User interface		
Dec. 1 2018 –July 15 2019	European survey: Barriers and facilitators in using a CDSS for fall risk management for older adults [34]	
Dec. 17 2019 –Jan. 3 2020	Semi-structured interviews*: Clinician needs for effective implementation and trustworthy decision making	
2020	Literature review*: Risk communication needs of physicians	
2020	Systematic review: Barriers and facilitators influencing medication-related CDSS acceptance according to clinicians [33]	
Knowledge base		
Used latest edition 2017	Dutch fall guideline: Effect of medication on fall risk in older adults [37]	
Feb. 5 2019 –Feb. 28 2020	Modified Delphi study: STOPPFall (Screening Tool of Older Persons Prescriptions in older adults with high fall risk): a Delphi study by the EuGMS Task and Finish Group on FRIDs [35]	
2020–2021	Logical elements rule method*: Formalizing clinical rules for the CDSS [38]	
Reasoning Engine and Software		
2021–2022	Agile methodology using test-driven development*: Unit tests, acceptance tests, verification, and validation testing	
Prediction model		
2020–2021	Development: The ADFICE_IT models for predicting falls and recurrent falls in community-dwelling older adults: Pooled analyses of European cohorts with special attention to medication [36]	
MRC Phase II: Feasibility	
CDSS prototype 1		
Sept. 10–24 2020	Usability testing round 1*: Identification of usability problems and adjustments to prototype	
CDSS final version		
June 28 –July 18 2021	Usability testing round 2*: Identification of remaining usability problems and development of final version	
* The methods and results are described in detail in this study.

Phase I: Development

The aim of this phase was to cultivate a robust (theoretical) understanding about how to develop the CDSS components [30], which consisted of the (1) development of the user interface, (2) development of the clinical knowledge base, (3) development of the prediction model, and (4) development of the software. A detailed description of the methods for the systematic review, European online survey, modified Delphi study, and development of the fall-risk prediction model have been published elsewhere by the research team [33–36].

Development of the user interface

To develop the user interface, a literature review on risk communication needs and a systematic review on barriers and facilitators were carried out, and extended with empirical research by means of a European online survey (n = 581) distributed among physicians and semi-structured interviews with clinicians (n = 19) from different hospitals in the Netherlands. The literature review assessed the risk communication needs of physicians. References for the review were retrieved from Google Scholar using the following search terms: “doctor,” “physician,” “healthcare provider,” “clinical decision support system,” “clinical support system,” “satisfaction,” “preferences,” “usability,” “user centered design,” “risk information,” “risk communication,” “medication,” “drug,” “drug on drug.” References were excluded if (1) the sample did not compromise of physicians; (2) the study did not discuss a medication-related CDSS; (3) the study was not published between 2014 and 2020. The systematic review assessed barriers and facilitators to CDSS use [33]. The latter barriers and facilitators to use analysis was expanded on in the survey (n = 581), which was distributed among European physicians, nurse practitioners, and physician assistants who in their clinical practice (primary, secondary, and tertiary care) see older adults at risk of falling [34]. In-person, semi-structured interviews with clinicians were conducted to assess the needs for effective implementation and trustworthy decision making with the help of a topic-list developed for this study. For example, clinicians were asked how the system should communicate a patient’s fall risk; or how the system can best support clinicians in making informed decisions about whether or not to deprescribe. Clinicians (n = 19) were eligible to participate if they (1) are working at a hospital in the Netherlands where the ADFICE_IT intervention was set to be implemented and (2) regularly perform a multifactorial fall risk assessment in older adults at risk of falling in an (geriatric) outpatient setting. All interviews were conducted in Dutch, voice-recorded, and transcribed verbatim. The analysis of these studies guided the development of the user interface of the CDSS, such as the system’s functionality, design, and content composition.

Development of the knowledge base

The STOPPFall (Screening Tool of Older Persons Prescriptions in older adults with high fall risk) tool [35] and the Dutch fall guideline [37] were used to identify relevant FRIDs in different medication classes. STOPPFall entails a FRIDs list with accompanying guidelines for deprescribing, and was constructed using a modified Delphi technique through consensus effort with 24 panelists from 13 European countries [35]. Both STOPPFall and the Dutch fall guideline form the basis of the system’s clinical knowledge base. To extend the knowledge base specific deprescribing advice for each class of FRIDs was identified from over 30 guidelines, and formalized using an adaptation of the Logical Elements Rule Method (LERM). LERM is a validated method for formalizing clinical rules for decision support (e.g., a CDSS). The method follows a step-by-step approach in which clinical rules are formulated by an informatics knowledge expert, and close collaboration from a clinical expert is sought throughout rule formalization to ensure that the intent of the guidelines are maintained [38]. Specifically, rather than using conjunctive normal form as specified in LERM, the criteria for each rule were formalized as medications to include (e.g., a class of FRIDs), medications to exclude (e.g., non-FRIDs within that class), and conditions (e.g. diagnoses or laboratory values that modify the advice).

Development of the reasoning engine and software

Reasoning engine and software development followed an agile methodology, including using test-driven development. Both unit tests (which test specific functions individually) and acceptance tests (which test larger parts of the software as a whole) were used. The software was developed in Node JS, using Express and MariaDB. Jest and TestCafe are used for testing [39–43]. For transparency, all software was developed as open-source software. Experienced developers were involved in creating the software architecture and establishing the development principles. Given the low expected load, clarity of code was prioritized over efficiency and scalability. Importantly, good traceability between the specification for the knowledge base and the implementation of these rules in the software was maintained throughout development, including the provision of evidence behind the recommendations to the end user.

Verification and validation testing were also conducted. Regarding verification, test cases (with specified input and expected output) were developed for each rule in the specification. If any output was not as expected, the corresponding part of the logic was checked for errors and, if needed, corrected. This was repeated until a 100% pass rate was achieved. These tests were added to the test suite and are run every time a change is made to the software. Validation testing took place in two rounds. First, hypothetical patient cases were constructed based on the specification. Specifically, cases were designed such that each text that the CDSS can produce appeared for at least one patient. The output was reviewed by NvdV to confirm that the advice was clinically sound and reflected the intent of the underlying guidelines. Any identified problems were corrected and the test was repeated. Second, informed consent was obtained from 10 patients and data from these patients was entered into the CDSS. NvdV and an expert in geriatric pharmacy reviewed the cases and commented if they did not agree with the advice. Since clinicians can have differing opinions on the same case, we aimed for 90% agreement with these recommendations.

Development of the prediction model

To enable the estimation of a patient’s risk of falling within 12 months, a fall-risk prediction model was developed. The development of the model is described in detail in Van de Loo, Seppala et al. [36], but briefly the model was developed based on a harmonized dataset comprised of two Dutch and one German cohort studies of community-dwelling older adults (65+), namely Longitudinal Aging Study Amsterdam (LASA), B-vitamins for Prevention of Osteoporotic Fractures study (B-PROOF), and Activity and Function in the Elderly in Ulm Study (ActiFE Ulm). For background, LASA is a prospective cohort study to determine predictors and consequences of aging in older adults in the Netherlands [44]. B-PROOF is a Dutch randomized, double-blind, placebo-controlled trial among older adults with an elevated homocysteine concentration [45]. ActiFE Ulm is a population-based cohort study among community dwelling older adults in Ulm and nearby regions in southwestern Germany [46]. From these cohorts, 5722 older adults (65+) for whom medication and follow-up data were recorded were included in the development of the prediction model. This prediction model comprises a part of the CDSS’s knowledge base. The outcome variable was defined as any fall (one or more falls) within a one-year follow-up. Candidate predictors were selected based on previously reported risk factors for falls that are easily obtained from the Electronic Health Record (EHR), such as sociodemographic variables; measures of emotional, cognitive and physical functioning; self-reported chronic conditions; variables related to lifestyle; biomarkers; and use of certain medications. Logistic regression with backward elimination was used to develop the model. The prediction model was internally validated using an internal-external cross-validation procedure, in which the performance of the model was tested in each of the development cohorts separately following Steyerberg and Harrel [47]. Performance was assessed using the C-statistic, whereby a C-statistic value of 0.5 indicates no discrimination and a value of 1 indicates perfect discrimination. Calibration plots were used to assess the agreement between predicted risks and observed outcomes. The prediction model was externally validated in data from geriatric outpatients (refer to Van de Loo, Heymans et al. for detailed methods) [48].

Phase II: Feasibility

Usability testing of the CDSS prototype

The feasibility phase assessed whether the prototype of the CDSS had any usability problems that needed to be fixed prior to developing the final version of the CDSS. This was achieved through two usability testing rounds with geriatric physicians (n = 11) from Dutch hospitals (i.e., the primary users of the system). Usability research argues that five participants are sufficient for detecting 80% of a product’s usability problems [49, 50]. Each physician was presented with three hypothetical patient cases that varied in fall risk (e.g., high versus low risk). Next, physicians were asked to carry out a scripted navigation for each case, which consisted of realistic CDSS task scenarios, resembling aspects of clinical documentation (e.g., analyzing a patient’s fall-risk, selecting relevant treatment options, making a referral). Throughout navigation, physicians were prompted to “think aloud” and verbalize their thought process while carrying out the tasks (i.e. using a concurrent think aloud method). Additionally, Camtasia 9, a usability software, was used to record the screen and mouse movements of each physician, including facial expressions and vocalizations (i.e., questions, expressions of confusion) [51].

Usability problems were identified for each physician session, and coded according to the Nielsen usability problems severity rating ranging from 0 = “I don’t agree that this is a usability problem at all” to 4 = “Usability catastrophe: imperative to fix this before product can be released” [52]. The categorization and potential negative impact of each identified usability problem was assessed following the augmented scheme for classifying and prioritizing usability problems [53]. Next, usability problems from all sessions were merged, and for each problem the occurrence of that problem was noted. This led to an overview of usability problems ordered on both severity and occurrence, ranging from the most severe and most often occurring problems to the least severe and least occurring problems. The analysis of the results from this study led to adjustments to the CDSS, resulting in a second version of the prototype. The aforementioned procedure was repeated in the second usability study among six physicians of which three did not participate in the first usability study. This allowed us to pinpoint remaining usability issues that needed to be improved prior to developing the final version of the CDSS.

Results

The analysis of the results from all studies guided the development of the CDSS. In this article, we report in detail the results of the literature review on risk communication needs, semi-structured interviews, software development, and usability testing rounds. The results of the systematic review on barriers and facilitators, European online survey, modified Delphi study, and development of the fall-risk prediction model are reported in detail elsewhere by the research team [33–36].

Phase I: Development

Results from the literature review

The aim of the literature review was to identify the risk communication needs of physicians. A total of six articles were included. The literature suggests that physicians prefer a holistic- and patient-specific approach to communicating health-related advice [26]. This approach should include information that is actionable, directive and transparent [54–57]. When visualizing risks, information-orientated graphs were perceived as easier to interpret by physicians, and were found to enhance shared decision-making between physicians and patients [56–58]. Similarly, using a traffic-light coloring system for the presentation of risk is believed to facilitate information processing, which can lead to a more rapid assessment of risk-based information by physicians [54, 56, 58]. Additionally, favorable effects were found on information processing outcomes (e.g., enhanced attention, reduced cognitive load) when the formatting of information and use of terminology was consistent, and when text density and visual clutter was reduced [56, 57].

Results from the semi-structured interviews

The aim of the interviews with clinicians was to identify the information-related needs for effective and trustworthy decision making, and the requirements for successful implementation of the CDSS in practice. A total of 19 clinicians employed by different hospitals in the Netherlands participated of which 15 were geriatric physicians followed by two nurse practitioners, a general practitioner in training, and a physical therapist. Results showed that fall risk information should be displayed in color as either a number or percentage. With regard to effective patient-physician communication, a visual graph was viewed as beneficial for the patient, including the ability to print out a patient-friendly handout. This handout should include information about the patient’s personalized fall risk and treatment plan that was discussed during the consultation. Opportunities to read about the benefits and side effects of deprescribing a medication, and having access to additional information (e.g., via hyperlinks) were viewed as important in the decision making to deprescribe. Moreover, information about how a patient’s fall risk is calculated (i.e., the prediction model) was perceived as vital information by physicians that would also enhance the system’s credibility. Lastly, for successful implementation of the CDSS in practice, physicians stressed the importance of reducing completion time (e.g., limiting the amount of mouse clicks).

Development of the user interface and knowledge base

Table 2 shows the aggregated system needs of clinicians obtained from the literature review on risk communication needs, systematic review on barriers and facilitators, European online survey, and semi-structured interviews with clinicians. These key requirements were subsequently operationalized into system features used for the development of the user interface of the first CDSS prototype for usability testing. Regarding the knowledge base of the system, the final set of clinical rules covering 22 classes of FRIDs, with specific deprescribing advice based on diagnoses, lab values, and concurrent medications, including the final fall-risk prediction model were integrated into the CDSS.

10.1371/journal.pone.0297703.t002 Table 2 Aggregated needs of clinicians.

Key requirements for CDSS
R: Literature reviews
S: Survey
I: Semi-structured interviews	
    • Limit repeated and uninformative alerts (R, S).
    • Include easy-to-use interactive features that cater towards physicians fast-paced work environment (R, I).
    • Provide physicians with actionable recommendations on how a patient’s fall risk can be reduced (R).
    • Make information transparent and credible by including hyperlinks that direct physicians to information surrounding the fall-risk prediction model and other medication-related information (I).
    • The CDSS should provide a holistic overview of the patient (e.g., comorbidities) (R).
    • The format of the CDSS should be consistent (R).
    • The presentation of risk should be presented in a text-format using concise and to-the-point language (e.g., using short sentences with standardized terminology) (R).
    • Information should be presented in a systematic manner that complements the consultation workflow of physicians (R, S).
    • A patient’s fall risk should be accompanied with information-orientated graphs (e.g., bar graphs, icon arrays, pie-charts) (R, I).
    • The information-orientated graphs should employ a traffic-light coloring system (i.e., red for high risk, yellow for medium, risk, and green for no risk) (R, I).
    • The CDSS should be efficient in use (i.e., fast completion time, limited clicks and text-entry fields) (I, S).	

The results of the prediction model are described in detail in Van de Loo, Seppala et al. [36] and Van de Loo, Heymans et al. [48]. The final prediction model consisted of the following 14 predictors: educational status (low, middle, high), depression (different validated scales were combined using z-scores), body mass index, grip strength (in kg), gait speed (in meter per second), number of functional limitations (from 0 to 5), systolic blood pressure (in mmHg), at least one fall in the previous 12 months, at least two falls in the previous 12 months, fear of falling (from 0 = not afraid to 2 = very afraid), smoking status (0 = never to 2 = current smoker), use of calcium channel blockers, use of antiepileptics, and use of drugs for urinary frequency and incontinence (see Van de Loo, Seppala et al. for harmonization guide) [36]. Performance of the prediction model was comparable to other fall-risk prediction models with a mean C-statistic value of 0.65 in the cohorts used for model development (range 0.61–0.67) [36, 59]. Calibration plots revealed good agreement between the predicted risks and observed outcomes [36]. External validation of the model showed similar performance in geriatric outpatients as in the cohorts that were used for development, with a C-statistic of 0.66 [48].

Development of the reasoning engine and software

Following agile development principles, development of the software started by building the smallest part that would be useful, which was the reasoning engine and an interface for verification and validation testing that showed the advice text for the doctor, checkbox options, corresponding patient-friendly text, and references all on one screen. Features were then added in order of priority. This, in combination with test-driven development, lead to a modular architecture with good separation between the connection to the EHR, the logic, and the user interface. This allowed developers to readily modify the interface based on feedback received from phase II. Development as open source software facilitated seeking and incorporating input from external expert developers (see acknowledgements). Regarding the system’s security, the software is designed to be hosted within the hospital network, and dependencies are minimized to lower the security footprint and improve maintainability. The CDSS logic is kept server side to ensure that it cannot be changed from the browser, and that the information seen by the user is always consistent with the saved data. Features were added to limit data being cached in the user’s browser. After verification testing and the first round of validation testing were completed with no remaining errors detected, the second round of validation testing with real patient data was conducted. The 10 patients were taking 42 different medications, triggering 50 distinct rules. This resulted in 100% agreement with these recommendations from one clinician and 95% agreement from the other, which exceeded the minimum threshold of 90% agreement.

Phase II: Feasibility

Results from usability testing rounds

To identify usability problems early on, the prototype of the CDSS was evaluated in two separate usability testing rounds with geriatric physicians. In the first round of usability testing, data saturation was reached after five physicians. The physicians (n = 5) identified a total of 74 usability problems with the CDSS, with a mean Nielsen’s severity rating classification of 2.49 (i.e., between minor to major usability problems) [52]. Regarding major to severe usability problems, physicians perceived difficulties with the general navigability of the system (e.g., no “back button”) and the medical terminology used within the system (e.g., what is meant by “lowest dose” or “minimal effective dose” for deprescribing medications). After these usability problems were solved, the CDSS underwent the second round of usability testing with physicians (n = 6, three of whom had not participated in the first round). Data saturation was reached after the inclusion of the sixth physician. The number of total usability problems identified by physicians was 11 and the mean Nielsen’s severity rating was 1.87 (i.e., between cosmetic to minor problems) [52]. The most pressing usability problems were improved (e.g., fixing hyperlinks and task-orientated buttons), and the final version of the CDSS was developed. After each usability testing round, the user interface and clinical knowledge base of the CDSS were further optimized, which led to the development of the final CDSS.

The final ADFICE_IT CDSS

The source code for the ADFICE_IT software is available at https://github.com/adfice-it. A connection between the EHR and the CDSS was made for the extraction of patient data into the CDSS. This data is used to provide patient-specific advice and to calculate a patient’s personalized fall risk estimate. The final version of the CDSS is depicted in Figs 1–6. The CDSS consists of 5 web pages. Page 1 (titled “Start” in Fig 1) is the landing page of the CDSS. On this page the physician can (1) check whether relevant patient information (e.g., age, morbidities, list of medications) was correctly extracted from the EHR, and add missing data for calculating the fall risk estimate; (2) view a graphical representation of the patient’s personalized fall risk estimate by means of a gradient scale; (3) view a model for shared decision-making that can be implemented during the consultation (in Fig 2); and (4) view a user guide of the system (titled “Handleiding”). Additionally, physicians have access to the patient identifier and personalized fall risk estimate during the entire consultation, as this information is displayed in the form of a horizontal menu bar on all subsequent pages.

10.1371/journal.pone.0297703.g001 Fig 1 Start page of the ADFICE_IT CDSS.

10.1371/journal.pone.0297703.g002 Fig 2 Shared decision-making model of the ADFICE_IT CDSS.

10.1371/journal.pone.0297703.g003 Fig 3 Preparation page of the ADFICE_IT CDSS.

10.1371/journal.pone.0297703.g004 Fig 4 Consult page of the ADFICE_IT CDSS.

10.1371/journal.pone.0297703.g005 Fig 5 Advice page of the ADFICE_IT CDSS.

10.1371/journal.pone.0297703.g006 Fig 6 Wrap-up page of the ADFICE_IT CDSS.

Page 2 (titled “Preparation” in Fig 3) lists each medication taken by the patient and structurally provides the physician with patient-specific deprescribing advice for each listed medication in an attempt to facilitate optimal deprescribing for the physician. Page 2 also provides relevant non-medication related information for preventing falls in older patients (e.g., referral to fall prevention interventions, leaflets, etc.). The clinician can select the treatment options and/or non-medication related information they want to discuss with the patient. Moreover, hyperlinks to third-party sources are provided for additional information about the listed medications (i.e., “Farmacotherapeutisch Kompas”) and related deprescribing advice (i.e., the guidelines used to formulate that advice).

Page 3 (titled “Consult” in Fig 4) provides an overview of the treatment options that the physician selected to discuss together with the patient (i.e., the deprescribing advice or leaflets selected in page 2). Based on the discussion with the patient, a final treatment plan is determined. Page 4 (titled “Advice” in Fig 5) displays the final treatment plan in a patient-friendly format, which is printable by the physician and accessible by the patient via the patient portal. In page 5 (titled “Wrap-up” in Fig 6), the physician is able to copy a summary of the consultation into the patient’s EHR, for future storage.

Discussion

This study outlined the user-centered development of a CDSS to support physicians in the optimum deprescribing of FRIDs in older adults (65+). The system was developed for the ADFICE_IT intervention. In the development phase (phase I), a robust theoretical understanding about key components of the CDSS was cultivated. For this, both existing and new evidence was relied upon, in which collaboration with users of the system was sought throughout different stages of development. In phase I, a fall-risk prediction model was developed and internally validated. This prediction model was later integrated into the CDSS. Phase 1 also included the development of the knowledge, reasoning engine and software. Moreover, verification and validation testing were conducted to check for and subsequently correct errors relating to the generation of deprescribing advice. Together, the results from phase I guided the development of the first prototype of the CDSS. In the feasibility phase (phase II), two separate usability testing rounds with geriatric physicians were conducted to identify and address remaining usability issues within the CDSS that could hinder successful implementation of the system later on.

A key strength of this study was the ability to integrate the aggregated deprescribing guidelines and the fall-risk prediction model into the clinical knowledge base of our CDSS. For physicians, deprescribing FRIDs is an often complex task as differences in risk of adverse drug events make it hard to determine whether reducing a particular FRID or combination of FRIDs will result in a significant change in preventing a fall [12, 13]. According to Bloomfield et al., it is exactly this particular complexity in FRIDs deprescribing that has not been effectively addressed in past interventions [12]. This CDSS addresses this need by systematically guiding physicians through the deprescribing of one or more FRIDs. The system does this by first signaling a FRID from a patient’s medication list, and subsequently leverages the stored deprescribing guidelines to advise physicians on how the identified FRID can be deprescribed optimally. Additionally, physicians can leverage the fall-risk prediction model as an adjunct to decision-making by assessing different advice on the basis of the patient’s estimated risk of falling within a 12-month period.

Another key strength of our study was implementing a user-centered design approach to development. This, in our opinion, will greatly influence the acceptance of the system in its intended clinical care setting, as its functionalities and content are tailored to the needs and preferences of its users, namely geriatric physicians who treat older fall risk patients in hospitals. Past studies have shown low CDSS acceptance rates and a high number of usability barriers when physicians are not involved in the development process [24, 26–28]. Since the outcomes of the ADFICE_IT intervention are dependent on physicians’ use of the CDSS, insights from the system’s users were gathered, at an early stage and throughout the development process of the system, which led to several advantages. For example, the test-driven development allowed us to make changes to the code with confidence that earlier added functionality was not being disrupted. This also led to a more modular architecture within the code base, which facilitated making the changes suggested by users during the usability testing rounds. Additionally, while the decision to favor clarity over performance aided in this process, it should be noted that this does carry a limitation whereby the software–in its current form–is not scalable to support hundreds of concurrent users. Future versions of the software should consider enhancing scalability to support a higher number of concurrent users. Moreover, future development could improve testing of the client software further (e.g., unit and acceptance tests) by concentrating manipulation of the display to a few parts of the web page, which could allow for easier testing during development. While employing a rigorous verification and validation process led to an improved version of the CDSS after each feedback round, a limitation of this process was the involvement of one of the two clinicians during earlier verification and validation steps. This may have resulted in higher agreement, even though the system did perform well with the second clinician, who was not otherwise involved in the development process. Future versions of the CDSS should consider incorporating additional clinicians in the validation process to enhance robustness and, consequently, strengthen the acceptability and applicability of the CDSS recommendations among clinicians. Additionally, while the steps described in this paper closely align with the quality lifecycle model described in the ISO/IED 25010:2023 standard, standards such as ISO/IEC 25010 could help guide improvements to the software.

A final strength of our CDSS is that the system displays several different treatment options (i.e., advice) for each identified FRID (i.e., using guideline-based reasoning), which provides an opportunity for physicians to discuss preferred treatment options together with the patient. This can foster joint medication management, which could help in tackling the observed low rates of deprescribing compliance among patients [60]. In older patients, Van Weert et al. showed that shared decision-making was effective at enhancing both medication-related knowledge and adherence rates, which we believe could have a positive effect on the outcomes of the ADFICE_IT intervention as well [21]. Specifically, the effectiveness of our CDSS (along with the patient portal) is currently being tested in a multicenter, cluster-randomized controlled trial with process evaluation in several Dutch hospitals [61].

Conclusions

This study followed a user-centered design approach to development to develop a CDSS intended for use in hospitals with older fall-risk patients in the Netherlands. The CDSS supports physicians in the optimum deprescribing of FRIDs to prevent falls in older adults (65+), leveraging aggregated deprescribing guidelines, a validated fall-risk prediction model, and shared decision-making. Moreover, due to the continued involvement of users, we were able to ensure that the CDSS was both useful and user-friendly as each new feedback round led to an improved version of the system. The CDSS (and patient portal) is currently being tested in a multicenter, cluster-randomized controlled trial with process evaluation at hospitals in the Netherlands (i.e., the ADFICE_IT intervention).

The authors thank Tsvetan Yordanov, Rutger Bazen and Stephen Madson for their contributions to the software, and Leonie Westerbeek for her contribution to the user interface. The authors thank Eveline Poelgeest, Gerrit Jan Hafkamp, Irene Gomez Bruinewoud, Hester van der Kroon, Hanna Willems, Suzanne Bleker, Oscar Smeekes and Stephanie van der Woude for their contributions to testing the feasibility of the ADFICE_IT CDSS.

10.1371/journal.pone.0297703.r001
Decision Letter 0
Baysal Mehmet Academic Editor
© 2024 Mehmet Baysal
2024
Mehmet Baysal
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Submission Version0
17 Jun 2024

PONE-D-23-43080Development of the ADFICE_IT clinical decision support system to assist deprescribing of fall-risk increasing drugs: A user-centered design approachPLOS ONE

Dear Dr. Groos,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

The main problme with the manuscript it mostly presents data that has already been published. However; numerous components covered in the manuscript is worth giving a try. Especially for clinically desicion supporting systems.

Please submit your revised manuscript by Aug 01 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Mehmet Baysal

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

2. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. 

When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section.

3. Thank you for stating the following financial disclosure: 

 [The ADFICE_IT project is supported by funding from the Netherlands Organization for Health Research and Development (ZonMw, Grant 848017004), The Hague and the Amsterdams Universiteitsfonds: Gepersonaliseerde Medicatieaanpassing bij Oudere Vallers. KMS was supported by US NIH grant T32-GM007592.

ZonMw: https://www.zonmw.nl/en

Amsterdams Universiteitsfonds: https://www.auf.nl/en

NIH: https://www.nih.gov/grants-funding ].  

Please state what role the funders took in the study.  If the funders had no role, please state: ""The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript."" 

If this statement is not correct you must amend it as needed. 

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

4. We noted in your submission details that a portion of your manuscript may have been presented or published elsewhere. [This manuscript summarizes the (I) development and (II) feasibility phases of the ADFICE_IT CDSS which consisted of a scoping literature review, systematic literature review, European survey, and semi-structured interviews with physicians, including the aggregation and testing of deprescribing guidelines and the development of the fall-risk prediction model. In phase II, the feasibility of the ADFICE_IT CDSS was tested by means of two usability testing rounds. In this manuscript, we do not report the results

from the systematic literature review, European online survey, modified Delphi study, and development of the fall-risk prediction model as these have been published elsewhere by the research team. Instead, we outline how the aggregated results from these studies and others helped inform the final design/development requirements of the CDSS. Thus, we believe that this does not constitute dual publication.] Please clarify whether this publication was peer-reviewed and formally published. If this work was previously peer-reviewed and published, in the cover letter please provide the reason that this work does not constitute dual publication and should be included in the current manuscript.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The manuscript describes the development of the ADFICE_IT clinical decision support system (CDSS) aimed at assisting physicians in deprescribing fall-risk increasing drugs (FRIDs) to reduce fall risk in older adults. The development process utilized a user-centered design approach involving systematic literature review, surveys, interviews, and usability testing with end users. Overall, the manuscript is a valuable contribution to the field of geriatric care and clinical decision support systems. It provides a robust framework for the development and implementation of similar tools in clinical practice. I only have a few minor questions and advices.

1. Introduction

The introduction provides a comprehensive overview of the importance of addressing fall risks in older adults and the potential of CDSS to support deprescribing practices. The background is well-researched, citing relevant studies that highlight the complexities of deprescribing FRIDs and the need for decision support tools.

The rationale for the study is clearly articulated.

The background information is well-supported by recent literature.

The objectives of the study are clearly stated, focusing on the development of a CDSS to support the deprescribing of FRIDs and its user-centered design process.

2. Methods

The methods section is detailed, describing a multi-phase approach that includes literature reviews, surveys, interviews, and usability testing according to the Medical Research Council framework. Comprehensive description of each phase of the development process is provided.

- Please explain what you mean with "Dutch fall clinics" as these are not common in all Eurpean countries.

- What was the profession of the physicians (geriatrics?)

- The sample size for the usability testing (n = 11) is relatively small, which might limit the generalizability of the findings.

- More detail on the selection criteria for survey and interview participants would enhance reproducibility.

- "harmonized dataset (n = 5722) of two Dutch cohorts and a German cohort study of community dwelling older adults (65+)" Please briefly explain and characterize both cohorts.

4. Results

The results are presented in a clear and structured manner, detailing the findings from literature reviews, surveys, interviews, and usability testing rounds.

- final prediction model consisted of the following 14 predictors: Briefly explain how these variables were measured (e.g. fear of falling, gait speed etc). Why did you enter both, one fall and two falls within last 12 months as independent variable in the model?

-prediction model: drugs for urinary frequency and incontinence: what exactly? Only drugs with central side effects or all drugs for urge incontinence?

- The presentation of results from the prediction model development could benefit from more detailed statistical analysis and validation metrics.

5. Discussion

The discussion is well-rounded, addressing the implications of the findings for clinical practice. It acknowledges the limitations of the study and suggests directions for future research.

- While limitations are acknowledged, there could be more emphasis on the potential biases introduced by the involvement of the same clinicians in both development and testing phases.

Reviewer #2: Thank you for the opportunity to review this manuscript.

This is an important contribution to the literature, giving an overview of the complete development and evaluation processes of a CDSS to assist deprescribing of fall-risk increasing drugs. The results of the multicenter, cluster randomized controlled trial with process evaluation at hospitals in the Netherlands will be of great interest.

My primary critique of this lengthy manuscript is that it provides primarily previously published data with very limited new information such as the results of the scoping review, presented in 11 lines (209-300) on page 15. Despite the appreciated summary in table 1, the many components described in the article required multiple back and forth between the methods and results sections.

A few specific questions / comments arose as I reviewed the manuscript:

P.4. line 71, the study registration number could be provided.

p.7 line 157, the patient portal is only mentioned here and briefly in the discussion. I suggest to either delete or develop further.

p.10, lines 190-191, the sentence ‘The scoping literature review assessed the risk communication needs of physicians.’ is the only methods information I found for this component. This is insufficient to describe the methods of a scoping review.

P.11, line 194, more information on the selection process of the participating physicians (GP, specialists, how they were selected…) is essential.

P.13, line 238, 10 patients seem minimal. Enough to cover the wide scope of recommendations?

p.13, line 259, will the system be used by other physicians such as GP? Could usability differ between GP and geriatric physicians?

p.14, lines 260-261, they were presented 3 cases but only completed at least one?

P.15, lines 289-300, I do not have enough methods information to appreciate the results of the scoping review.

P.15, lines 301-312, I do not have enough information. For example, the reader must go back to the abstract for the n=19. Who were those physicians? How were the interviews conducted? Was a standard questionnaire used?

p.16, line 314, how was this hierarchy decided?

p.16, lines 321-325, are all these variables (e.g. educational status, grip strength, fear of falling) readily available in a majority of EHR?

P.18, line 336, was ‘input from external expert developers’ received?

P.18, lines 343-345, what are the numbers? 95% and 100% agreement on what?

p. 18, line 348, ’74 individual usability problems’. What is the context? The three hypothetical cases? How many of those cases were evaluated by the 5 physicians?

P. 18, line 354, I have to go back to the abstract to find the n=11 and understand that these are 6 new physicians.

P.18, line 355, give the number of problems instead of the 16%.

p.19, lines 364, the figures were unreadable. Unfortunately, this portion of the manuscript could not be assessed.

Reviewer #3: The manuscript presents a technically sound piece of scientific research that describes the development and evaluation of the ADFICE_IT clinical decision support system (CDSS) designed to assist in the deprescribing of fall-risk increasing drugs (FRIDs) in older adults. The study is well-grounded, with data supporting the conclusions drawn. The experiments were conducted rigorously, with appropriate controls, replication, and sample sizes.

Strengths:

1. User-Centered Design Approach: The involvement of end users throughout the development process ensures that the CDSS is practical, useful, and likely to be accepted in clinical settings. This approach has been shown to improve technology adoption by healthcare professionals.

2. Integration of Deprescribing Guidelines and Risk Prediction Models: The CDSS leverages a combination of deprescribing guidelines and a validated fall-risk prediction model, which addresses the complexity involved in deprescribing FRIDs. This systematic approach can significantly aid physicians in making informed decisions to reduce fall risks in older adults.

3. Iterative Development and Usability Testing: The use of agile methodologies and iterative feedback from usability testing rounds allowed the authors to refine the CDSS continuously. This resulted in a user-friendly system with minimized usability barriers, enhancing its effectiveness and efficiency in clinical practice.

4. Detailed Validation Process: The CDSS underwent thorough verification and validation testing, including both unit tests and real patient data validation. This rigorous process ensured the reliability and accuracy of the system's recommendations.

Areas for Improvement:

1. Scalability: Although clarity of code was prioritized over efficiency and scalability, future versions of the software should consider enhancing scalability to support a higher number of concurrent users. This will be particularly important for broader implementation across multiple clinical settings.

2. Broader Validation: While the involvement of one clinician in the early validation steps was beneficial, incorporating a more diverse group of clinicians in the initial stages could further strengthen the robustness of the validation process. This would ensure that the CDSS recommendations are universally acceptable and applicable.

3. Comprehensive Documentation: Providing more detailed documentation and user guides could facilitate easier implementation and training for new users. This would help in overcoming initial barriers to adoption and ensure that users can fully utilize the system's capabilities.

Overall Impression:

The manuscript is well-written and presented in clear, standard English. The data underlying the findings are fully available, adhering to PLOS ONE's data availability policy. The development and evaluation of the ADFICE_IT CDSS represent a significant advancement in supporting the safe deprescribing of FRIDs to prevent falls in older adults. The user-centered design approach and rigorous testing processes enhance the system's practicality and reliability.

The manuscript's clear and systematic presentation, combined with the robustness of the research methods, makes it a valuable contribution to the field. It addresses a critical aspect of geriatric care, and its findings have the potential to improve patient outcomes significantly.

Recommendation: Minor revision

Reasoning:

The manuscript presents a significant and well-executed study on the development and evaluation of the ADFICE_IT clinical decision support system (CDSS) for assisting in the deprescribing of fall-risk increasing drugs (FRIDs) in older adults. The study is technically sound, with robust data supporting the conclusions, and it follows rigorous experimental protocols. The user-centered design approach and iterative usability testing have resulted in a practical and user-friendly system that addresses a critical aspect of geriatric care.

However, there are a few areas that could benefit from further improvement before publication:

Scalability: The manuscript should address potential scalability issues and discuss plans for future enhancements to support a higher number of concurrent users.

Comprehensive Documentation: Providing more detailed documentation and user guides will facilitate easier implementation and training, helping to overcome initial adoption barriers.

Alignment with Standars: Consider the pertinence of describing in the manuscript how the ADFICE_IT CDSS aligns with the quality characteristics defined in ISO/IEC 25010:2011. Including this information can strengthen the manuscript by demonstrating that the CDSS meets recognized benchmarks for software quality, thus enhancing its credibility and reliability. If the authors can access this standard, it is recommended to review it to ensure comprehensive alignment.

Addressing these minor issues will enhance the manuscript's clarity and ensure that the CDSS is well-positioned for broader implementation and use.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

10.1371/journal.pone.0297703.r002
Author response to Decision Letter 0
Submission Version1
8 Aug 2024

We express our gratitude to the reviewers for their insightful feedback, which has been invaluable in enhancing the quality of our manuscript. In an attached file ("Response to Reviewers") you will find our responses (in bold) to each reviewer’s comment together with references to relevant changes we made to the manuscript. The full edited manuscript with tracked changes is attached as well.

Attachment Submitted filename: Response to Reviewers.docx

10.1371/journal.pone.0297703.r003
Decision Letter 1
Baysal Mehmet Academic Editor
© 2024 Mehmet Baysal
2024
Mehmet Baysal
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Submission Version1
22 Aug 2024

Development of the ADFICE_IT clinical decision support system to assist deprescribing of fall-risk increasing drugs: A user-centered design approach

PONE-D-23-43080R1

Dear Dr. Groos,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Mehmet Baysal

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #3: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #3: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #3: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #3: The manuscript presents a significant and well-conducted study on the development of a clinical decision support system (CDSS) aimed at optimizing the deprescribing of fall-risk increasing drugs in older adults. The authors have thoroughly addressed the reviewers' comments, resulting in a manuscript that is technically sound and methodologically rigorous.

The authors acknowledge some limitations in the current version of the software, particularly regarding scalability and the breadth of clinical validation. While the software is functional, they recognize the need for future improvements to support a higher number of concurrent users and to expand validation to include a more diverse group of clinicians, which will enhance the robustness and generalizability of the system's recommendations. Additionally, the authors mention the potential alignment with recognized standards such as ISO/IEC 25010 in future versions, which could further strengthen the system's credibility and reliability.

Overall, the manuscript is a valuable contribution to the field, and with these forward-looking plans, it will be well-positioned for publication.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #3: No

**********

10.1371/journal.pone.0297703.r004
Acceptance letter
Baysal Mehmet Academic Editor
© 2024 Mehmet Baysal
2024
Mehmet Baysal
https://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
26 Aug 2024

PONE-D-23-43080R1

PLOS ONE

Dear Dr. Groos,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Mehmet Baysal

Academic Editor

PLOS ONE
==== Refs
References

1 World Health Organization: Falls. https://www.who.int/news-room/fact-sheets/detail/falls (2021). Accessed 25 Oct 2022.
2 Haagsma JA , Olij BF , Majdan M , van Beeck EF , Vos T , Castle CD , et al . Falls in older aged adults in 22 European countries: incidence, mortality and burden of disease from 1990 to 2017. Inj Prev. 2020;26 (1 ):i67–74. https://injuryprevention.bmj.com/content/26/Suppl_2/i67 doi: 10.1136/injuryprev-2019-043347 32111726
3 Hartholt KA , van Beeck EF , Polinder S , van der Velde N , van Lieshout EMM , Panneman MJM , et al . Societal consequences of falls in the older population: Injuries, healthcare costs, and long-term reduced quality of life. J Trauma. 2011;71 (3 ):748–53. doi: 10.1097/TA.0b013e3181f6f5e5 21045738
4 Peeters G , Bennett M , Donoghue OA , Kennelly S , Kenny RA . Understanding the aetiology of fear of falling from the perspective of a fear-avoidance model–A narrative review. Clin Psychol Rev. 2020;79 :101862. doi: 10.1016/j.cpr.2020.101862 32442854
5 Petersen N , König H-H , Hajek A . The link between falls, social isolation and loneliness: A systematic review. Arch Gerontol Geriatr. 2020;88 :104020. doi: 10.1016/j.archger.2020.104020 32018091
6 Heinrich S , Rapp K , Rissmann U , Becker C , König HH . Cost of falls in old age: A systematic review. Osteoporos Int. 2010 Jun 19;21 (6 ):891–902. http://link.springer.com/10.1007/s00198-009-1100-1 19924496
7 de Vries M , Seppala LJ , Daams JG , van de Glind EMM , Masud T , van der Velde N , et al . Fall-risk-increasing drugs: A systematic review and meta-analysis: I. Cardiovascular Drugs. J Am Med Dir Assoc. 2018;19 (4 ):371.e1–9. doi: 10.1016/j.jamda.2017.12.013 29396189
8 Seppala LJ , Wermelink AMAT , de Vries M , Ploegmakers KJ , van de Glind EMM , Daams JG , et al . Fall-risk-increasing drugs: A systematic review and meta-analysis: II. Psychotropics. J Am Med Dir Assoc. 2018;19 (4 ):371.e11–7. doi: 10.1016/j.jamda.2017.12.098 29402652
9 Seppala LJ , van de Glind EMM , Daams JG , Ploegmakers KJ , de Vries M , Wermelink AMAT , et al . Fall-risk-increasing drugs: A systematic review and meta-analysis: III. Others. J Am Med Dir Assoc. 2018;19 (4 ):372.e1–8. doi: 10.1016/j.jamda.2017.12.099 29402646
10 Michalcova J , Vasut K , Airaksinen M , Bielakova K . Inclusion of medication-related fall risk in fall risk assessment tool in geriatric care units. BMC Geriatr. 2020;20 (1 ):1–11. doi: 10.1186/s12877-020-01845-9 33158417
11 Van der Velde N , Minhas JS . Appropriate deprescribing in older people: a challenging necessity Commentary to accompany themed collection on deprescribing. Age Ageing. 2021;50 (5 ):1516–9. doi: 10.1093/ageing/afab142 34260681
12 Bloomfield HE , Greer N , Linsky AM , Bolduc J , Naidl T , Vardeny O , et al . Deprescribing for community-dwelling older adults: A systematic review and meta-analysis. J Gen Intern Med. 2020;35 (11 ):3323–32. doi: 10.1007/s11606-020-06089-2 32820421
13 Lee J , Negm A , Peters R , Wong EKC , Holbrook A . Deprescribing fall-risk increasing drugs (FRIDs) for the prevention of falls and fall-related complications: A systematic review and meta-analysis. BMJ Open. 2021;11 (2 ):e035978. doi: 10.1136/bmjopen-2019-035978 33568364
14 Bell HT , Steinsbekk A , Granas AG . Factors influencing prescribing of fall-risk-increasing drugs to the elderly: A qualitative study. Scand J Prim Health Care. 2015;33 (2 ):107–14. doi: 10.3109/02813432.2015.1041829 25965505
15 Tinetti ME , Gordon C , Sogolow E , Lapin P , Bradley EH . Fall-risk evaluation and management: Challenges in adopting geriatric care practices. Gerontologist. 2006;46 (6 ):717–25. doi: 10.1093/geront/46.6.717 17169927
16 Drickamer MA , Levy B , Irwin KS , Rohrbaugh RM . Perceived needs for geriatric education by medical students, internal medicine residents and faculty. J General Intern Med. 2006;21 :1230–4. doi: 10.1111/j.1525-1497.2006.00585.x 17105521
17 Hoffmann TC , Del Mar C . Clinicians’ expectations of the benefits and harms of treatments, screening, and tests: A systematic review. JAMA Intern Med. 2017;177 :407–19. doi: 10.1001/jamainternmed.2016.8254 28097303
18 Laing SS , Silver IF , York S , Phelan EA . Fall prevention knowledge, attitude, and practices of community stakeholders and older adults. J Aging Res. 2011;2011 :1–9. doi: 10.4061/2011/395357 21915377
19 Goud R , van Engen-Verheul M , de Keizer NF , Bal R , Hasman A , Hellemans IM , et al . The effect of computerized decision support on barriers to guideline implementation: A qualitative study in outpatient cardiac rehabilitation. Int J Med Inform. 2010;79 (6 ):430–7. doi: 10.1016/j.ijmedinf.2010.03.001 20378396
20 Sim I , Gorman P , Greenes RA , Haynes RB , Kaplan B , Lehmann H , et al . Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001;8 (6 ):527–34. doi: 10.1136/jamia.2001.0080527 11687560
21 Van Weert JCM , van Munster BC , Sanders R , Spijker R , Hooft L , Jansen J . Decision aids to help older people make health decisions: a systematic review and meta-analysis. BMC Med Inform Decis Mak. 2016;16 (1 ):1–20. doi: 10.1186/s12911-016-0281-8 26754574
22 Clyne B , Bradley MC , Hughes C , Fahey T , Lapane KL . Electronic prescribing and other forms of technology to reduce inappropriate medication use and polypharmacy in older people: A review of current evidence. Clin Geriatr Med. 2012;28 (2 ):301–22. doi: 10.1016/j.cger.2012.01.009 22500545
23 Dalton K , O’Brien G , O’Mahony D , Byrne S . Computerised interventions designed to reduce potentially inappropriate prescribing in hospitalised older adults: a systematic review and meta-analysis. Age Ageing. 2018;47 (5 ):670–8. doi: 10.1093/ageing/afy086 29893779
24 Damoiseaux-Volman B , van der Velde N , Ruige S , Romijn J , Abu-Hanna A , Medlock S . Effect of interventions with a clinical decision support system for hospitalized older patients: Systematic review mapping implementation and design factors. JMIR Med Inform 2021;9 (7 ):e28023. doi: 10.2196/28023 34269682
25 Moons KGM , Altman DG , Reitsma JB , Ioannidis JPA , Macaskill P , Steyerberg EW , et al . Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): Explanation and elaboration. Ann Intern Med. 2015;162 (1 ):W1. doi: 10.7326/M14-0698 25560730
26 Kilsdonk E , Peute LW , Jaspers MWM . Factors influencing implementation success of guideline-based clinical decision support systems: A systematic review and gaps analysis. Int J Med Inform. 2017;98 :56–64. doi: 10.1016/j.ijmedinf.2016.12.001 28034413
27 Moxey A , Robertson J , Newby D , Hains I , Williamson M , Pearson S-A . Computerized clinical decision support for prescribing: provision does not guarantee uptake. J Am Med Inform Assoc. 2010;17 (1 ):25–33. doi: 10.1197/jamia.M3170 20064798
28 Trinkley KE , Blakeslee WW , Matlock DD , Kao DP , Van Matre AG , Harrison R , et al . Clinician preferences for computerised clinical decision support for medications in primary care: a focus group study. BMJ Health Care Inform. 2019;26 (1 ):e000015. doi: 10.1136/bmjhci-2019-000015 31039120
29 Gagnon M-P , Desmartis M , Labrecque M , Car J , Pagliari C , Pluye P , et al . Systematic review of factors influencing the adoption of information and communication technologies by healthcare professionals. J Med Syst. 2010;36 (1 ):241–77. doi: 10.1007/s10916-010-9473-4 20703721
30 Walden A , Garvin L , Smerek M , Johnson C . User-centered design principles in the development of clinical research tools. Clin Trials. 2020;17 (6 ):703–11. doi: 10.1177/1740774520946314 32815381
31 Van de Loo B , Linn AJ , Medlock S , Belimbegovski W , Seppala LJ , Van Weert JCM , et al . AI-based decision support to optimize complex care for preventing medication-related falls. Nat Med. 2024; 30 : 620–621. doi: 10.1038/s41591-023-02780-z 38273147
32 Craig P , Dieppe P , Macintyre S , Michie S , Nazareth I , Petticrew M . Developing and evaluating complex interventions: the new Medical Research Council guidance. Int J Nurs Stud. 2013;50 (5 ), 587–92. doi: 10.1016/j.ijnurstu.2012.09.010 23159157
33 Westerbeek L , Ploegmakers KJ , De Bruijn GJ , Linn AJ , Van Weert JCM , Daams JG , et al . Barriers and facilitators influencing medication-related CDSS acceptance according to clinicians: A systematic review. Int J Med Inform. 2021;152 :104506. doi: 10.1016/j.ijmedinf.2021.104506 34091146
34 Ploegmakers KJ , Medlock S , Linn AJ , Lin Y , Seppala LJ , Petrovic M , et al . Barriers and facilitators in using a clinical decision support system for fall risk management for older people: a European survey. Eur Geriatr Med. 2022; 13 , 395–405. doi: 10.1007/s41999-021-00599-w 35032323
35 Seppala LJ , Petrovic M , Ryg J , Bahat G , Topinkova E , Szczerbińska K , et al . STOPPFall (Screening Tool of Older Persons Prescriptions in older adults with high fall risk): A Delphi study by the EuGMS Task and Finish Group on fall-risk-increasing drugs. Age Ageing. 2021;50 (4 ):1189–99. doi: 10.1093/ageing/afaa249 33349863
36 Van de Loo B , Seppala LJ , van der Velde N , Medlock S , Denkinger M , de Groot LCPGM , et al . Development of the ADFICE_IT models for predicting falls and recurrent falls in community-dwelling older adults: pooled analyses of European cohorts with special attention to medication. J Gerontol A Biol Sci Med Sci. 2022;77 (7 ):1446–54. doi: 10.1093/gerona/glac080 35380638
37 Richtlijnendatabase. Effect medicijnen op valrisico ouderen–Richtlijn. 2017. https://richtlijnendatabase.nl/richtlijn/preventie_van_valincidenten_bij_ouderen/effect_medicijnen_op_valrisico_ouderen.html. Accessed 26 Oct 2022.
38 Medlock S , Opondo D , Eslami S , Askari M , Wierenga P , de Rooij SE , et al . LERM (Logical Elements Rule Method): A method for assessing and formalizing clinical rules for decision support. Int J Med Inform. 2011;80 (4 ):286–95. doi: 10.1016/j.ijmedinf.2011.01.014 21333589
39 Node.js Foundation. Node.js. 2019. https://nodejs.org/en/. Accessed 26 Oct 2022.
40 Express. Fast, unopinionated, minimalist web framework for Node.js. Expressjs.com. 2017. http://expressjs.com/. Accessed 26 Oct 2022.
41 MariaDB Foundation. MariaDB server: The open source relational database. MariaDB.org. 2015. https://mariadb.org/. Accessed 26 Oct 2022.
42 Jest. Delightful JavaScript Testing. Jestjs.io. 2017. https://jestjs.io/. Accessed 26 Oct 2022.
43 TestCafe. End-to-end testing, simplified. testcafe.io. https://testcafe.io/. Accessed 26 Oct 2022.
44 Hoogendijk EO , Deeg DJH , de Breij S , et al . The Longitudinal Aging Study Amsterdam: cohort update 2019 and additional data collections. Eur J Epidemiol. 2020;35 (1 ):61–74. doi: 10.1007/s10654-019-00541-2 31346890
45 Van Wijngaarden JP , Dhonukshe-Rutten RA , van Schoor NM , et al . Rationale and design of the B-PROOF study, a randomized controlled trial on the effect of supplemental intake of vitamin B12and folic acid on fracture incidence. BMC Geriatr. 2011;11 (1 ):80. doi: 10.1186/1471-2318-11-80 22136481
46 Denkinger MD , Franke S , Rapp K , et al . Accelerometer-based physical activity in a large observational cohort―study protocol and design of the activity and function of the elderly in Ulm (ActiFE Ulm) study. BMC Geriatr. 2010;10 (1 ):50. doi: 10.1186/1471-2318-10-50 20663209
47 Steyerberg EW , Harrell FE . Prediction models need appropriate internal, internal–external, and external validation. J Clin Epidemiol. 2016;69 :245–7. doi: 10.1016/j.jclinepi.2015.04.005 25981519
48 Van de Loo B , Heymans MW , Medlock S , Boyé NDA , van der Cammen TJM , Hartholt KA , et al . Validation of the ADFICE_IT models for predicting falls and recurrent falls in geriatric outpatients. J Am Med Dir Assoc. 2023 Dec;24 (12 ):1996–2001. doi: 10.1016/j.jamda.2023.04.021 Epub 2023 May 30. .37268014
49 Alroobaea R , Mayhew PJ . How many participants are really enough for usability studies?. 2014 science and information conference. IEEE, 2014. doi: 10.1109/SAI.2014.6918171
50 Lewis JR . Sample sizes for usability studies: Additional considerations. Human factors. 1994;36 (2 ): 368–378. doi: 10.1177/001872089403600215 8070799
51 TechSmith Camtasia. Screen Recorder & Video Editor. TechSmith. 2022. https://www.techsmith.com/video-editor.html. Accessed 26 Oct 2022.
52 Nielsen J. Usability engineering. Boston: Academic Press; 1993.
53 Khajouei R , Peute LWP , Hasman A , Jaspers MWM . Classification and prioritization of usability problems using an augmented classification scheme. J Biomed Inform. 2011;44 (6 ):948–57. doi: 10.1016/j.jbi.2011.07.002 21782036
54 Kennedy G , Gallego B . Clinical prediction rules: A systematic review of healthcare provider opinions and preferences. Int J Med Inform. 2019;123 :1–10. doi: 10.1016/j.ijmedinf.2018.12.003 30654898
55 Khong PCB , Holroyd E , Wang W . A critical review of the theoretical frameworks and the conceptual factors in the adoption of clinical decision support systems. Comput Inform Nurs. 2015;33 (12 ): 555–70. doi: 10.1097/CIN.0000000000000196 26535769
56 Leichtling G , Hildebran C , Novak K , Alley L , Doyle S , Reilly C , et al . Physician responses to enhanced prescription drug monitoring program profiles. Pain Med. 2019;21 (2 ):e9–21. doi: 10.1093/pm/pny291 30698811
57 Miller K , Mosby D , Capan M , Kowalski R , Ratwani R , Noaiseh Y , et al . Interface, information, interaction: a narrative review of design and functional requirements for clinical decision support. JAMIA. 2018;25 (5 ):585–92. doi: 10.1093/jamia/ocx118 29126196
58 Van Maurik IS , Visser LN , Pel-Littel RE , Buchem MM van , Zwan MD , Kunneman M , et al . Development and usability of ADappt: Web-based tool to support clinicians, patients, and caregivers in the diagnosis of mild cognitive impairment and Alzheimer disease. JMIR Form Res. 2019;3 (3 ):e13417. doi: 10.2196/13417 31287061
59 Gade GV , Jørgensen MG , Ryg J , et al . Predicting falls in community-dwelling older adults: a systematic review of prognostic models. BMJ Open. 2021; 11 (5 ):e044170. doi: 10.1136/bmjopen-2020-044170 33947733
60 Boyé NDA , van der Velde N , de Vries OJ , van Lieshout EMM , Hartholt KA , Mattace-Raso FUS , et al . Effectiveness of medication withdrawal in older fallers: Results from the improving medication prescribing to reduce risk of falls (IMPROveFALL) trial. Age Ageing. 2017;46 (1 ):142–6. doi: 10.1093/ageing/afw161 28181639
61 de Wildt KK , van de Loo B , Linn AJ , Medlock SK , Groos SS , Ploegmakers KJ et al (2023) Effects of a clinical decision support system and patient portal for preventing medication-related falls in older fallers: Protocol of a cluster randomized controlled trial with embedded process and economic evaluations (ADFICE_IT). PLoS ONE 18 :e0289385. doi: 10.1371/journal.pone.0289385 37751429
