
==== Front
Bone Joint Res
Bone Joint Res
BJR
Bone & Joint Research
2046-3758
The British Editorial Society of Bone & Joint Surgery London

10.1302/2046-3758.139.BJR-2024-0135.R1
BJR-2024-0135.R1
General Orthopaedics
Glucose
Glucose Variability
Complications
Glycemic Control
Glycated Haemoglobin
General Orthopaedicsgeneral-orthopaedicsbj17439Traumabj11388Orthopaedic diseasesbj731Anatomybj1763Basic sciencebj9776Medical specialistsbj6858Hipbj11386Orthopaedic devicesbj13736Randomized controlled trialsbj11128Nonunionbj3413Clavicle fracturesbj3454Cliniciansbj14720Sepsisbj9771Medical devicesbj3410Claviclebj12081Patient reported outcome measuresThe Clinical Practice Integration of Artificial Intelligence (CPI-AI) framework
a proposed application of IDEAL principles to artificial intelligence applications in trauma and orthopaedics
https://orcid.org/0000-0002-1443-5908
Farrow Luke MBChB, MRCS SCREDS Clinical Lecturer, Speciality Registrar 1 2 luke.farrow@abdn.ac.uk

Meek Dominic MBChB, BSc (Hons), MD, FRCS (Tr&Orth) Consultant Orthopaedic Surgeon 3 rmdmeek@doctors.org.uk

Leontidis Georgios BSc, MSc, PhD Director - Interdisciplinary Centre for Data & Artificial Intelligence 1 georgios.leontidis@abdn.ac.uk

Campbell Marion BSc (Hons), MSc, PhD, CStat, FSS, FFPH, FSCT, FRSE, FMedSci Professor of Health Services Research 1 m.k.campbell@abdn.ac.uk

Harrison Ewen OBE, FRSE, FMedSci Professor of Surgery and Data Science 4 ewen.harrison@ed.ac.uk

Anderson Lesley PhD, MPHe, BSc(Hons), PGCHET, FHEA Chair of Health Data Science 1 lesley.anderson@abdn.ac.uk

1 Institute of Applied Health Sciences, University of Aberdeen, Aberdeen, UK
2 Grampian Orthopaedics, Aberdeen Royal Infirmary, Aberdeen, UK
3 Department of Orthopaedics, Queen Elizabeth University Hospital, Glasgow, UK
4 Centre of Medical Informatics, University of Edinburgh, Edinburgh, UK
Correspondence should be sent to Luke Farrow. E-mail: luke.farrow@abdn.ac.uk
L. Farrow reports receipt of a grant from the Chief Scientist Office (CSO) Scotland Clinical Academic Fellowship, of which the submitted work will form part of his associated PhD thesis, related to this study. M. Campbell reports leadership as Chair of the Medical Research Council (MRC)/National Institute for Health and Care Research (NIHR) Better Methods Better Research funding panel, not related to this study. M. Campbell also reports work as an advisor for the IDEAL Collaboration Council, not related to this study.

18 9 2024
9 2024
13 9 507512
© 2024 Farrow et al.
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ https://online.boneandjoint.org.uk/TDM This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial No Derivatives (CC BY-NC-ND 4.0) licence, which permits the copying and redistribution of the work only, and provided the original author and source are credited. See https://creativecommons.org/licenses/by-nc-nd/4.0/

Despite the vast quantities of published artificial intelligence (AI) algorithms that target trauma and orthopaedic applications, very few progress to inform clinical practice. One key reason for this is the lack of a clear pathway from development to deployment. In order to assist with this process, we have developed the Clinical Practice Integration of Artificial Intelligence (CPI-AI) framework – a five-stage approach to the clinical practice adoption of AI in the setting of trauma and orthopaedics, based on the IDEAL principles (https://www.ideal-collaboration.net/). Adherence to the framework would provide a robust evidence-based mechanism for developing trust in AI applications, where the underlying algorithms are unlikely to be fully understood by clinical teams.

Cite this article: Bone Joint Res 2024;13(9):507–512.

Artificial intelligence
Machine learning
CPI-AI
trauma
Randomized Controlled Trial
nonunions
clavicle fracture
clinicians
sepsis
medical devices
clavicle
patient-reported outcome measure (PROM)
Hip
http://dx.doi.org/10.13039/100014589 Chief Scientist Office odf2nlm-version2.0
price$2.00
principal-institutionUniversity of Aberdeen, Aberdeen, UK
article-typeGeneral Orthopaedics
coi-statementL. Farrow reports receipt of a grant from the Chief Scientist Office (CSO) Scotland Clinical Academic Fellowship, of which the submitted work will form part of his associated PhD thesis, related to this study. M. Campbell reports leadership as Chair of the Medical Research Council (MRC)/National Institute for Health and Care Research (NIHR) Better Methods Better Research funding panel, not related to this study. M. Campbell also reports work as an advisor for the IDEAL Collaboration Council, not related to this study.
==== Body
pmcArticle focus

Safety, reliability, and transparency concerns are some of the major barriers to clinical practice integration of artificial intelligence (AI) in in trauma and orthopaedics.

This study provides a robust evidence framework for new AI applications, following a similar pathway to that for the integration of new surgical devices or drugs.

Adherence to the pathway should help to provide strong support for the safe and effective future integration of AI into all aspects of trauma and orthopaedics.

Key messages

This study sets out a five-stage framework which spans from concept outline to post-deployment model surveillance.

Key checkpoints to progression through each of the stages are highlighted, along with associated reporting guidelines.

Strengths and limitations

Strength: Based on the established IDEAL principles governing the introduction of surgical innovation.

Limitation: Widespread dissemination and uptake of the framework within AI research is required to realistically effect any potential clinical impact.

Introduction

Artificial intelligence (AI) has previously been highlighted as a potential innovation to embrace in trauma and orthopaedics, although caution behind the scale and scope of adoption has been recommended.1 With the increasing availability and use of large-scale data repositories, it is likely that the use of AI, with its associated abilities to analyze substantial quantities of information and include previously unavailable data sources (for example, digital images and unstructured clinical text information), will play a large role in future healthcare advancements.2

As potential applications of AI to healthcare expand, there is a clear need for key stakeholders (researchers, clinical staff, industry partners, policymakers, and patients) to understand the pipeline from development of AI algorithms to integration into clinical practice. It is important to have clear guidance on what good evidence looks like at each stage of development, to ensure informed appraisal of emerging evidence and associated claims of benefit. Currently there are hundreds, if not thousands, of developed AI tools that focus on clinical applications, but very few that progress to inform clinical practice.3,4

A number of AI reporting guidelines have been developed (e.g. CONSORT-AI, DECIDE-AI, and TRIPOD-AI), and provide a useful framework to support researchers to produce manuscripts that are of sufficient rigour for stakeholders to understand the potential application of these models.5-8 The vast majority of AI algorithms for use in healthcare are, however, produced without reference to any clear framework to inform potential users of their readiness (or not) for deployment into a useful clinical application. This is perhaps one of the major reasons why we continue to see limited clinical benefit from AI, despite its potential to transform the way in which healthcare operates at all points in the patient journey.9,10

Such guidance exists around other related areas, such as the Medical Research Council (MRC) framework for complex interventions11 and the mobile health (mHealth) evidence reporting and assessment (mERA)12 checklists, but these are not specific to AI and therefore potentially miss key aspects unique to healthcare innovation in this field. Other published advice relates to specific fields of AI application,13 or quality assurance,14 and do not depict a clear pipeline of the necessary journey from model development to safe use in clinical practice that are applicable to all domains of AI implementation within a healthcare setting. Governmental bodies, including the USA Food and Drug Administration (FDA) and NHS England,15,16 have both highlighted the clear need for further development around regulations for AI and digital health. A robust evidence pipeline is likely to play an integral part of that regulatory process, similar to current medical device and drug developments.

Having a recognized structured pathway for AI deployment (with clear stages of development and associated evidence of benefit required at each stage) would also provide key indicators for safety, and reduce the real danger associated with the potential use of improperly tested AI models within a clinical setting. Perhaps one of the best examples of this is the AI early sepsis prediction model embedded within the EPIC electronic health record system. Researchers found that even though the model was widely accepted, it still failed to identify a significant number of patients with sepsis during external validation, suggesting a lack of robustness and generalization capabilities of this (or a similar) model. The algorithm also set off a high number of false positive warnings for potential sepsis identification that may have led to “alert fatigue”.17 AI models have historically lacked proper cohort representation during development, leading to real-world issues with generalizability and subsequently raising ethical concerns related to poor model performance across different ethnic, racial, and sex categories.18,19

The IDEAL framework provides a well-established and comprehensive pathway from innovation to potential clinical adoption that has been used for several years to guide surgical innovation.20 This has included applications in the field of trauma and orthopaedics, such as development and clinical testing of the X-Bolt Dynamic Hip Plating System culminating in a large-scale randomized controlled trial (RCT) assessment.21-23 IDEAL forms a five-stage (six including the pre-clinical stage 0) concept of appropriate clinical integration and underpinning research for surgical therapy innovation: Stage 0 – Theoretical or in vitro testing of clinical utility and risk; Stage 1 – Proof of concept (In-vivo testing); Stage 2a – Development (Prospective Case-series); Stage 2b – Exploration (Prospective Cohort or feasibility RCT); Stage 3 – Assessment (Randomized Controlled Trial); and Stage 4 – Long term study (Surveillance to detect rare / late outcomes).

Although specific to surgical innovation, the IDEAL framework provides a valid and rigorous structure which provides key principles to inform the development of frameworks for other forms of innovation across the wider clinical spectrum, including AI, and which could be used in conjunction with other standards such as the BS 30440 Validation framework for the use of AI within healthcare.24

Successful use of the IDEAL framework as a tool for separate subscale development has already been demonstrated in the area of device innovation, culminating in the proposed IDEAL-D framework that provides guidance for the evaluation and regulation of medical devices.25

A proposed framework for AI

A framework to guide the development and evaluation of AI applications, which builds on the IDEAL principles together with insights from other validated AI quality assurance standards, has been proposed – the Clinical Practice Integration of Artificial Intelligence (CPI-AI) framework. As highlighted in Figure 1, it is important to understand that development is an iterative process, and that there may be natural flow forwards and backwards between stages to manage potential bias or model drift. Figure 2 provides a list of progression criteria to aid further understanding of suitability for progression of research through the CPI-AI stages.

Fig. 1 The Clinical Practice Integration of Artificial Intelligence (CPI-AI) framework. A proposed application of the IDEAL principles detailed by McCulloch et al.20

Fig. 2 Clinical Practice Integration of Artificial Intelligence (CPI-AI) progression criteria. A guide to aid determination of research integrity and suitability for stage progression.

Stage 0 – Concept outline

As with the IDEAL framework, this stage involves conceptualizing an AI algorithm with input from a diverse team of AI experts, clinicians, patients, and policymakers. This would involve evaluating relevant background literature and assessing the theoretical risks and benefits of the idea, including technological considerations such as infrastructure, interoperability, and data security. Consideration would also be given to feasibility and scope, potential user groups and output format, as well as ethics, transparency, and interpretability, as these will greatly influence the later stages of development. Generative AI may be used to aid concept development, but should include appropriate consideration to any later potential copyright implications.

Stage 1 – Algorithm development

This is currently by far the most frequently examined area of AI application to healthcare. In this stage, the AI algorithm is developed and initially tested at a basic level as a proof of concept. This will typically involve a single/multicentre study whereby a single large, or multiple smaller, dataset(s) is/are used to develop the AI tool.

AI-related reporting checklists for IDEAL Stage 1 are available and would be used to ensure that all key relevant information is contained within any publication of such work, for example STARD-AI (for diagnostic studies)26 or the TRIPOD+AI guidance (for prediction modelling).8 At this stage, no strong references would be made to the potential clinical applicability of any developed AI algorithms without more detailed evaluation conducted as with the later stages of the IDEAL framework. Given the high-stakes environment of AI healthcare applications, it would be anticipated that substantial attention would be paid to explainability and interpretability of the developed model. Active efforts would be made to confirm the results and performance of those early AI models to other available datasets.

An example of a CPI-AI stage 1 concept would be the development of an AI-based clinical prediction model, to determine risk of nonunion with nonoperative management of a clavicle fracture based on radiograph images at the time of injury. It would be anticipated that this would use local/national imaging databases with a ground truth available to determine who achieved union or not. Sample characteristics, as well as key metrics such as accuracy, recall, precision, f1 score, and specificity, would be detailed. Consideration would be given to how best to move towards CPI-AI Stage 2a.

Stage 2a – External validation

The next stage of algorithm development requires testing outside the dataset where the algorithm was originally trained, a process known as external validation. This is currently lacking in many studies, but is an essential part of appropriate use to ensure that the algorithm remains suitably effective within different populations.27 Multiple external validations are likely needed to account for demographic, clinical, and healthcare infrastructure variability. This external validation is in addition to the common AI development practice that involves splitting a dataset into train, validation, and test sets. The external validation will involve a completely separate dataset that will describe a different population and/or clinical setting.

This stage would likely highlight the need to tune certain parts of the model to better calibrate it within this wider set of patients and/or a new clinical infrastructure. This work is essential to ensuring fairness across patient groups and minimizing any bias present in the initial training data. Further testing would then be performed to demonstrate improved performance in the wider dataset. Again, reporting checklists such as those highlighted above (e.g. TRIPOD-AI and STARD-AI) would be used to ensure that this process is accurately documented and transparent. Reporting at this stage should also include calibration across different patient groups where appropriate, for example sex, age, or disease severity. The level of similarities and differences between the original and external validation cohorts should be highlighted.

An example of a CPI-AI Stage 2a for our clavicle fracture project would be testing of the developed algorithm on a different (multiple) local/national imaging database, with suitable tuning to ensure that the algorithm performs well on a wider subset of patients.

Stage 2b – Prospective assessment

Once confirmed that the algorithm is suitable for wider application, the project would move forwards to assess the feasibility of potential clinical practice implementation. This would be done in a prospective fashion with assessment of the intervention in a ‘live’ setting. This would include consideration of how use of the algorithm impacts on clinical decision-making, with initial evaluation performed as a ‘silent reading’ phase, so that impact on patient care is not realised while this process occurs. Attention would be placed on improving upon the explainability of the AI models (incorporating both local (individual AI outputs) and global (general population level) explainability methods), while tuning it for enhanced performance.

As per the DECIDE-AI reporting checklist,6 which has been developed specifically to address this type of early-stage clinical deployment, this stage would focus on three key areas: Clinical Utility, Safety, and Human Factors. Potential cost implications would also be important within financially constrained healthcare settings, which has typically been lacking previously. It would also be important to ensure that the interface is compliant with the necessary regulations regarding the processing and potential storage of patient data. Again, further updating of the model may be required following this prospective assessment and stakeholder feedback with re-testing and fluidity within Stage 2.

An example of a CPI-AI Stage 2b for our clavicle project would be initial ‘silent reading’ of the algorithm output, followed by live use in the fracture clinic to aid decision-making regarding the potential risks of nonunion and help determine optimal treatment strategy on a case-by-case basis. Saliency maps could be used for local explainability, to help demonstrate the regional importance of included images in algorithmic outcomes to reassure clinical teams and patients of the correct rationale for decision-making.

Stage 3 – Clinical impact assessment (RCT)

Similar to any other healthcare intervention, AI technologies require formal assessment of their impact on clinical practice, utilizing the most robust and unbiased methodology available. Typically, this would be a large-scale intervention in the form of a multicentre RCT, comparing the safety and outcomes of the use of the algorithm against current alternative best practice methodology. The effects on the wider system would also need to be assessed.

Potential outcomes examined would likely differ dependent on the type of the AI algorithm and its intended use, for example with the clavicle project example the primary outcome could be the proportion of patients who successfully avoided a nonunion and their linked patient-reported outcome measure (PROM) data over time (as one would anticipate that successful avoidance of nonunion would improve PROMs, particularly in the earlier stages of evaluation). Cost-effectiveness analyses would likely also be an important part of any assessment, balancing the cost of AI implementation and monitoring versus the potential savings from avoidance of unnecessary surgery.

Reporting checklists for a potential CPI-AI Stage 3 would include the CONSORT-AI statement,5 as well as the SPIRIT-AI checklist for RCT protocol design and publication.28

Stage 4 – Implementation and model surveillance

Following successful RCT analysis and confirmation of clinical and cost-effectiveness, implementation of the model is performed. Flexibility of the AI intervention may be required to widen adoption and should be evaluated iteratively. Supporting sustainability has previously been identified as a key driver of long-term behavioural change.29

It is also essential that any fully deployed AI model undergoes serial evaluation (through continued assessment of relative performance metric attainment) to confirm its execution is maintained over time. AI models are at particular risk of two major issues related to changes in how they function over time: concept drift and data drift.30

Concept drift relates to changes in how one labels data and interprets the algorithm findings. For example, clinicians may change the time at which a nonunion is declared to have occurred, or decide that painless radiological nonunions should be classified in the healed category.

Data drift is the more common concern, where the characteristics of the inputted data change over time. One such example would be an ageing population presenting different population demographics, or perhaps updated X-ray technology that changes the quality of the images assessed by the algorithm.

If model drift is identified, then retraining, and re-tuning, or adapting the model would likely be required to ensure adequate ongoing performance and maintenance of cost-effectiveness. Such techniques are currently under development and could be used in this setting, but further scrutiny is required for the problem in hand.31

Conclusion

AI algorithms are complex medical interventions and need to be appropriately evaluated as such. Our suggested CPI-AI framework would allow a clear pathway from development to clinical practice application of AI models in trauma and orthopaedics, with stage gates to ensure appropriate onward development of only effective algorithms – maximizing the potential that AI has to provide a more personalized and precise healthcare service.

Data Availability

The data that support the findings for this study are available to other researchers from the corresponding author upon reasonable request.

Author contributions

L. Farrow: Conceptualization, Methodology, Writing – original draft, Writing – review & editing

D. Meek: Conceptualization, Methodology, Writing – review & editing

G. Leontidis: Conceptualization, Methodology, Writing – review & editing

M. Campbell: Conceptualization, Methodology, Writing – review & editing

E. Harrison: Conceptualization, Methodology, Writing – review & editing

L. Anderson: Conceptualization, Methodology, Supervision, Writing – review & editing

Funding statement

The authors disclose receipt of the following financial or material support for the research, authorship, and/or publication of this article: a grant from the Chief Scientist Office Scotland Clinical Academic Fellowship, as reported by L. Farrow.

ICMJE COI statement

L. Farrow reports receipt of a grant from the Chief Scientist Office (CSO) Scotland Clinical Academic Fellowship, of which the submitted work will form part of his associated PhD thesis, related to this study. M. Campbell reports leadership as Chair of the Medical Research Council (MRC)/National Institute for Health and Care Research (NIHR) Better Methods Better Research funding panel, not related to this study. M. Campbell also reports work as an advisor for the IDEAL Collaboration Council, not related to this study.

Data sharing

The data that support the findings for this study are available to other researchers from the corresponding author upon reasonable request.

Acknowledgements

The authors are grateful to the original IDEAL authors for the outline on which this updated framework is based.

Open access funding

The authors report that they received open access funding for their manuscript from Chief Scientist Office Scotland.

Social media

Follow L. Farrow on X @docfarrow

© 2024 Farrow et al. This is an open-access article distributed under the terms of the Creative Commons Attribution Non-Commercial No Derivatives (CC BY-NC-ND 4.0) licence, which permits the copying and redistribution of the work only, and provided the original author and source are credited. See https://creativecommons.org/licenses/by-nc-nd/4.0/
==== Refs
References

1. Clement ND Simpson AHRW Artificial intelligence in orthopaedics Bone Joint Res 2023 12 8 494 496 10.1302/2046-3758.128.BJR-2023-0199 37553119
2. Kunze KN Orr M Krebs V Bhandari M Piuzzi NS Potential benefits, unintended consequences, and future roles of artificial intelligence in orthopaedic surgery research: a call to emphasize data quality and indications Bone Jt Open 2022 3 1 93 97 10.1302/2633-1462.31.BJO-2021-0123.R1 35084227
3. Ramkumar PN Pang M Polisetty T Helm JM Karnuta JM Meaningless applications and misguided methodologies in artificial intelligence-related orthopaedic research propagates hype over hope Arthroscopy 2022 38 9 2761 2766 10.1016/j.arthro.2022.04.014 35550419
4. Andaur Navarro CL Damen JAA Takada T et al. Risk of bias in studies on prediction models developed using supervised machine learning techniques: systematic review BMJ 2021 375 2281 10.1136/bmj.n2281 34670780
5. Liu X Cruz Rivera S Moher D Calvert MJ Denniston AK SPIRIT-AI and CONSORT-AI Working Group Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension Nat Med 2020 26 9 1364 1374 10.1038/s41591-020-1034-x 32908283
6. Vasey B Clifton DA Collins GS DECIDE-AI Steering Group DECIDE-AI: new reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence Nat Med 2021 27 2 186 187 10.1038/s41591-021-01229-5 33526932
7. Farrow L Zhong M Ashcroft GP Anderson L Meek RMD Interpretation and reporting of predictive or diagnostic machine-learning research in Trauma & Orthopaedics Bone Joint J 2021 103-B 12 1754 1758 10.1302/0301-620X.103B12.BJJ-2021-0851.R1 34847720
8. Cohen JF Bossuyt PMM TRIPOD+AI: an updated reporting guideline for clinical prediction models BMJ 2024 385 q824 10.1136/bmj.q824 38626949
9. Polisetty TS Jain S Pang M et al. Concerns surrounding application of artificial intelligence in hip and knee arthroplasty Bone Joint J 2022 104-B 12 1292 1303 10.1302/0301-620X.104B12.BJJ-2022-0922.R1 36453039
10. Varghese J Artificial intelligence in medicine: chances and challenges for wide clinical adoption Visc Med 2020 36 6 443 449 10.1159/000511930 33442551
11. Skivington K Matthews L Simpson SA et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance BMJ 2021 374 2061 10.1136/bmj.n2061 34593508
12. Agarwal S LeFevre AE Lee J et al. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist BMJ 2016 352 i1174 10.1136/bmj.i1174 26988021
13. Bizzo BC Dasegowda G Bridge C et al. Addressing the challenges of implementing artificial intelligence tools in clinical practice: principles from experience J Am Coll Radiol 2023 20 3 352 360 10.1016/j.jacr.2023.01.002 36922109
14. No authors listed Blueprint for Trustworthy AI Implementation Guidance and Assurance for Healthcare Coalition for Health AI 2023 https://www.coalitionforhealthai.org/papers/blueprint-for-trustworthy-ai_V1.0.pdf date last accessed 30 August 2024
15. No authors listed AI Regulation: Improving the regulatory approval process and building trust in robust standards NHS Transformation Directorate 2021 https://transform.england.nhs.uk/ai-lab/ai-lab-programmes/regulating-the-ai-ecosystem/ date last accessed 30 August 2024
16. No authors listed Executive Summary for the Patient Engagement Advisory Committee Meeting: Artificial Intelligence (AI) and Machine Learning (ML) in Medical Devices U.S. Food & Drug Administration 2021 https://www.fda.gov/media/151482/download date last accessed 10 September 2024
17. Wong A Otles E Donnelly JP et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients JAMA Intern Med 2021 181 8 1065 1070 10.1001/jamainternmed.2021.2626 34152373
18. Obermeyer Z Powers B Vogeli C Mullainathan S Dissecting racial bias in an algorithm used to manage the health of populations Science 2019 366 6464 447 453 10.1126/science.aax2342 31649194
19. Hundt A Agnew W Zeng V Kacianka S Gombolay M Robots Enact Malignant Stereotypes [abstract] FAccT '22: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 2022.
20. McCulloch P Altman DG Campbell WB et al. No surgical innovation without evaluation: the IDEAL recommendations Lancet 2009 374 9695 1105 1112 10.1016/S0140-6736(09)61116-8 19782876
21. Griffin XL Parsons N McArthur J Achten J Costa ML The Warwick Hip Trauma Evaluation One: a randomised pilot trial comparing the X-Bolt Dynamic Hip Plating System with sliding hip screw fixation in complex extracapsular hip fractures: WHiTE (One) Bone Joint J 2016 98-B 5 686 689 10.1302/0301-620X.98B5.37350 27143742
22. Kahane S Vaghela KR Stammers J Goldberg A Smitham P Biomechanical study comparing cut-out resistance of the X-Bolt® and dynamic hip screw at various tip-apex distances Surg Technol Int 2019 35 395 401 31524281
23. Griffin XL Achten J O’Connor HM Cook JA Costa ML WHiTE Four Investigators Effect on health-related quality of life of the X-Bolt dynamic plating system versus the sliding hip screw for the fixation of trochanteric fractures of the hip in adults: the WHiTE Four randomized clinical trial Bone Joint J 2021 103-B 2 256 263 10.1302/0301-620X.103B.BJJ-2020-1404.R1 33390029
24. No authors listed BS 30440:2023: Validation framework for the use of artificial intelligence (AI) within healthcare. Specification BSI Group 2022 https://landingpage.bsigroup.com/LandingPage/Standard?UPI=000000000030434912 date last accessed 10 September 2024
25. Sedrakyan A Campbell B Merino JG Kuntz R Hirst A McCulloch P IDEAL-D: a rational framework for evaluating and regulating the use of medical devices BMJ 2016 353 i2372 10.1136/bmj.i2372 27283585
26. Sounderajah V Ashrafian H Aggarwal R et al. Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group Nat Med 2020 26 6 807 808 10.1038/s41591-020-0941-1 32514173
27. de Vries CF Colosimo SJ Staff RT et al. Impact of different mammography systems on artificial intelligence performance in breast cancer screening Radiol Artif Intell 2023 5 3 e220146 10.1148/ryai.220146 37293340
28. Cruz Rivera S Liu X Chan A-W et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension Nat Med 2020 26 9 1351 1363 10.1038/s41591-020-1037-7 32908284
29. Khalil H Kynoch K Implementation of sustainable complex interventions in health care services: the triple C model BMC Health Serv Res 2021 21 1 143 10.1186/s12913-021-06115-x 33588823
30. Carter RE Anand V Harmon DM Pellikka PA Model drift: when it can be a sign of success and when it can be an occult problem Intell Based Med 2022 6 100058 10.1016/j.ibmed.2022.100058
31. Thota M Yi D Leontidis G LLEDA -- Lifelong Self-Supervised Domain Adaptation arXiv.org e-Print archive 2023 https://arxiv.org/pdf/2211.09027.pdf date last accessed 30 August 2024
