
==== Front
NPJ Digit Med
NPJ Digit Med
NPJ Digital Medicine
2398-6352
Nature Publishing Group UK London

39289474
1230
10.1038/s41746-024-01230-5
Article
A drug mix and dose decision algorithm for individualized type 2 diabetes management
http://orcid.org/0000-0001-9445-6516
Nambiar Mila 1
http://orcid.org/0000-0002-5482-2646
Bee Yong Mong bee.yong.mong@singhealth.com.sg

23
Chan Yu En 1
Ho Mien Ivan 13
Guretno Feri 1
Carmody David 23
Lee Phong Ching 23
http://orcid.org/0000-0002-7591-3141
Chia Sing Yi 4
Salim Nur Nasyitah Mohamed 4
http://orcid.org/0000-0001-5893-4306
Krishnaswamy Pavitra pavitrak@i2r.a-star.edu.sg

1
1 https://ror.org/053rfa017 grid.418705.f 0000 0004 0620 7694 Institute for Infocomm Research, Agency for Science, Technology and Research (A*STAR), Singapore, Singapore
2 https://ror.org/036j6sg82 grid.163555.1 0000 0000 9486 5048 Department of Endocrinology, Singapore General Hospital, Singapore, Singapore
3 https://ror.org/02j1m6098 grid.428397.3 0000 0004 0385 0924 Duke-NUS Medical School, Singapore, Singapore
4 https://ror.org/036j6sg82 grid.163555.1 0000 0000 9486 5048 Health Services Research Unit, Singapore General Hospital, Singapore, Singapore
17 9 2024
17 9 2024
2024
7 25423 9 2023
19 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Pharmacotherapy guidelines for type 2 diabetes (T2D) emphasize patient-centered care, but applying this approach effectively in outpatient practice remains challenging. Data-driven treatment optimization approaches could enhance individualized T2D management, but current approaches cannot account for drug-specific and dose-dependent variations in safety and efficacy. We developed and evaluated an AI Drug mix and dose Advisor (AIDA) for glycemic management, using electronic medical records from 107,854 T2D patients in the SingHealth Diabetes Registry. Given a patient’s medical profile, AIDA leverages a predict-then-optimize approach to identify the minimal drug mix and dose changes required to optimize glycemic control, subject to clinical knowledge-based guidelines. On unseen data from large internal, external, and temporal validation sets, AIDA recommendations were estimated to improve post-visit glycated hemoglobin (HbA1c) by an average of 0.40–0.68% over standard of care (P < 0.0001). In qualitative evaluations on 60 diverse cases by a panel of three endocrinologists, AIDA recommendations were mostly rated as reasonable and precise. Finally, AIDA’s ability to account for drug-dose specifics offered several advantages over competing methods, including greater consistency with practice preferences and clinical guidelines for practical but effective options, indication-based treatments, and renal dosing. As AIDA provides drug-dose recommendations to improve outcomes for individual T2D patients, it could be used for clinical decision support at point-of-care, especially in resource-limited settings.

Subject terms

Type 2 diabetes
Computer science
Applied mathematics
https://doi.org/10.13039/501100001348 Agency for Science, Technology and Research (A*STAR) H19/01/a0/023 – Diabetes Clinic of the Future H19/01/a0/023 – Diabetes Clinic of the Future H19/01/a0/023 – Diabetes Clinic of the Future H19/01/a0/023 – Diabetes Clinic of the Future H19/01/a0/023 – Diabetes Clinic of the Future H19/01/a0/023 – Diabetes Clinic of the Future H19/01/a0/023 – Diabetes Clinic of the Future H19/01/a0/023 – Diabetes Clinic of the Future H19/01/a0/023 – Diabetes Clinic of the Future H19/01/a0/023 – Diabetes Clinic of the Future Nambiar Mila Bee Yong Mong Chan Yu En Ho Mien Ivan Guretno Feri Carmody David Lee Phong Ching Chia Sing Yi Salim Nur Nasyitah Mohamed Krishnaswamy Pavitra issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Type 2 diabetes (T2D) is a progressive condition that requires regular optimization of pharmacotherapy for glycemic control1,2. Recent American Diabetes Association/European Association for the Study of Diabetes guidelines provide evidence-based recommendations to inform pharmacologic treatment for blood glucose2,3, and emphasize a patient-centered approach2–4. Yet, there remains considerable uncertainty on how best to tailor choice of treatment combination and intensity to individual patient needs and disease stage5,6. In practice, clinicians grapple with the complexities of aligning the many drug mix and dose choices to myriad patient-specific factors (e.g., demographics, weight, comorbidities), and to variable treatment acceptance and response profiles5–11. These difficulties are amplified in busy primary care clinics with large caseloads and limited access to specialist judgment12–14. Importantly, the above practice challenges compound inertia to intensify treatment and limit clinical outcomes12,13,15,16. Hence, there is a need for systematic approaches to support clinicians in choosing more effective yet practicable treatment regimens for individual patients.

The growing use of electronic medical records (EMR) presents new opportunities for decision support. As EMR data link diverse patient characteristics and prescription choices with real-world outcomes for large cohorts, they could be used to compare effectiveness of different treatment regimens and inform optimal treatment choices for individual patients. Previous works have applied machine learning and optimization techniques on EMR data to generate individualized treatment recommendations for improving T2D management17–21. Notably, these works focused on making broad (largely drug class level) treatment recommendations independently of specific drug and dose considerations typically required in clinical practice. However, guidelines and landmark trials highlight important drug- and dose-dependent variations in treatment efficacy, safety, and risk profiles22–28. This suggests that the ability to account for these variations could enable more clinically relevant recommendations. Inherently, this requires optimizing outcomes over drug and dose level actions—a more challenging prescriptive recommendation task whose feasibility is yet to be established.

In this study, we developed and validated an AI Drug mix and dose Advisor (AIDA) for optimizing glycemic management in type 2 diabetes. Specifically, we adopted a predict-then-optimize approach. Given a patient’s medical profile, we employed machine learning to model glycated hemoglobin (HbA1c) outcomes under different prescription regimens. Then, leveraging this prediction model, we developed a decision algorithm to identify the minimal drug mix and dose changes required to optimize glycemic control, subject to clinical knowledge-based guidelines. We developed and evaluated AIDA using EMR data for 107,854 T2D patients from the multi-institutional SingHealth Diabetes Registry (SDR)29. Our evaluations included a range of quantitative and qualitative analyses. First, we conducted extensive quantitative evaluations to estimate performance in relation to standard of care using unseen data from independent cohorts treated at different care sites and timeframes. Second, we performed qualitative evaluations to assess clinical relevance and practicability of AIDA’s treatment recommendations. Third, we characterized the impact of including drug-dose specifics on the practical relevance and guideline concordance of the generated treatment recommendations. Our decision algorithm systematically integrates data and guidelines to identify optimal prescription regimens for individualized management of T2D patients.

Results

We developed and demonstrated our drug mix and dose recommendation approach using EMR data from the SDR dataset. After inclusion criteria, the SDR had data for a total of 107,854 T2D patients with 1,039,869 prescription visits during 2013–2020 across 22 care sites. We used data from different sites and time periods for development and evaluation (see “Methods”). For model development, we used data for 66,623 patients (697,500 visits during 2013–2019 to any of 21 designated development sites). For evaluations, we considered three independent test cohorts: an internal validation cohort treated at the development sites (test set IV: 16,653 patients, 2013–2019 visits); an external validation cohort treated at an external primary care site (test set EV: 14,316 patients, 2013–2019 visits); and a temporal validation cohort treated during a future period (test set TV: 10,262 patients, 2020 visits). Cohort characteristics are provided in Table 1. The three test sets have distinct demographic and medical characteristics. Compared to test set IV, test set EV patients had different demographics, higher prevalence of hypertension and hyperlipidemia, lower prevalence of macrovascular complications, and lower frequency of prescriptions for newer drugs like sodium-glucose cotransporter-2 inhibitors (SGLT2-i) as well as lower frequency of prescriptions for injectables (e.g., insulins). Meanwhile, compared to test set IV, test set TV patients were younger, had lower diabetes durations, lower prevalence of hypertension, hyperlipidemia, and microvascular complications, and higher frequency of prescriptions for SGLT2-i but lower frequency of prescriptions for insulins. This diversity in our test sets enables evaluation of generalizability across diverse cohorts.Table 1 Demographic and clinical characteristics of the development and evaluation datasets

Feature	Model development seta	Internal validation test set IV	External validation test set EV	Temporal validation test set TV	P (test set IV vs. EV)	P (test set IV vs. TV)	
Patients (n)	66,623	16,653	14,316	10,262	N/A	N/A	
Visits (n)	697,500	16,653	14,316	10,262	N/A	N/A	
Years in SDR (mean (s.d.))	2.8 (1.9)	2.2 (1.9)	2.3 (1.9)	2.2 (1.2)	N/A	N/A	
Demographic and visit information	
 Age (years, mean (s.d.))	65.6 (11.2)	64.5 (12.0)	65.3 (11.2)	60.5 (12.2)	<0.0001	<0.0001	
 Sex: Male (%)	52.5%	52.5%	48.8%	58.2%	<0.0001	<0.0001	
Ethnicity (%)	
 Chinese (C)	70.2%	70.2%	65.6%	65.4%	<0.0001	<0.0001	
 Indian (I)	10.8%	10.8%	8.2%	12.3%	
 Malay (M)	14.7%	14.7%	20.9%	16.6%	
 Other (O)	4.3%	4.3%	5.2%	5.7%	
 Primary care (Polyclinic, %)	87.4%	78.6%	100.0%	74.0%	<0.0001	<0.0001	
Physical measurements and laboratory data	
 Pre-visit HbA1c (%, mean (s.d.))	7.4 (1.2)	7.4 (1.4)	7.3 (1.3)	7.5 (1.6)	<0.0001	<0.0001	
 HbA1c ≤ Individualized target (%)	54.1%	53.5%	55.4%	48.4%	0.0039	0.0039	
 Most recent BMI (kg/m2, mean (s.d.))	26.6 (4.9)	26.8 (5.2)	26.9 (5.2)	27.4 (5.6)	0.075	<0.0001	
 Most recent eGFR (mL/min/1.73 m2, mean (s.d.))	78.7 (24.6)	78.7 (26.2)	79.4 (24.2)	83.7 (23.8)	0.035	<0.0001	
Medical history	
 Diabetes duration (years, mean (s.d.))	10.9 (7.9)	9.9 (8.0)	9.5 (7.3)	5.3 (6.6)	0.0007	<0.0001	
 Hyperlipidemia (%)	93.6%	89.7%	94.1%	81.7%	<0.0001	<0.0001	
 Hypertension (%)	86.4%	81.9%	86.7%	70.3%	<0.0001	<0.0001	
 Microvascular complications (%)	30.5%	29.4%	27.8%	26.9%	0.0026	<0.0001	
 Macrovascular complications (%)	27.8%	30.6%	26.3%	30.0%	<0.0001	0.29	
 Severe hypoglycemia (%)b	1.4%	1.7%	1.5%	1.2%	0.18	0.0032	
Prescription historyc	
Previous visit severity level (%)d	
 1: One OAD	39.7%	43.6%	44.0%	51.5%	<0.0001	<0.0001	
 2: Multiple OADs	45.0%	42.1%	44.6%	37.0%	
 3: First injectables (GLP-1 RA/Long-acting insulin)	4.9%	5.5%	4.5%	6.8%	
 4: Premixed insulin	9.0%	7.3%	5.8%	3.1%	
 5: Rapid-acting insulin	1.5%	1.5%	1.0%	1.6%	
 On Metformin (%)	89.0%	86.0%	87.8%	87.9%	<0.0001	<0.0001	
 On GLP-1 RA (%)	0.2%	0.2%	0.0%	0.2%	0.0057	0.0015	
 On SGLT2-i (%)	5.0%	6.5%	2.4%	19.6%	<0.0001	<0.0001	
 On Insulin (%)	15.4%	14.4%	11.4%	11.3%	<0.0001	<0.0001	
The four datasets have no patient overlap (Supplementary Fig. 1). P values are for differences between test sets IV and EV, and between test sets IV and TV, and are computed using two-sample t tests with equal variance for continuous valued features and chi-square tests for categorical features, with corrections for multiple hypothesis testing using the Benjamini–Hochberg (False Discovery Rate) procedure.

Pre-visit HbA1c the last HbA1c reading before the visit, BMI body mass index, eGFR estimated glomerular filtration rate, OAD oral antidiabetic drugs, SGLT2i-i sodium-glucose cotransporter-2 inhibitors, GLP-1 RA glucagon-like peptide-1 receptor agonists.

aAs statistics are at the visit level, patients with longer medical histories could be over-represented.

bAdmissions with hypoglycemia as primary diagnosis.

cStatistics for previous visit prescriptions.

dSeverity levels mark critical treatment regimen transitions2,3.

Given a patient’s medical profile and previous prescriptions from the EMR (see “Methods”), the treatment recommendation task is to identify the drug mix and dose regimen that optimizes glycemic control. The ‘drug mix and dose regimen’ is defined as the set of generic name antidiabetic drugs and their daily dosages. The glycemic control objective is to achieve a post-visit HbA1c (mean HbA1c outcome two to six months after the visit) at or under the patient’s individualized target, subject to specified knowledge-based diabetes management (KDM) guidelines (see “Methods”)2,3. An example of the required form of the treatment recommendation is “metformin: 1000 mg/day, sitagliptin: 50 mg/day, insulin glargine: increase”.

AI Drug mix and dose Advisor (AIDA)

To generate recommendations, AIDA employs an intuitive search scheme informed by an HbA1c prediction model and the specified KDM guidelines (Fig. 1). Briefly, the search uses a heuristic that starts with the patient’s previous prescription regimen and considers changes stepwise in order of increasing intensity, while also incorporating medication prioritizations as per the KDM guidelines. The steps are: (1) drop medication and/or lower dose, (2) increase dose of an existing medication, (3) add one new strongly prioritized medication, (4) substitute one new medication, or (5) add one new medication. For each step, the algorithm defines and ranks feasible regimens according to the KDM guidelines, and leverages the HbA1c prediction model to assess whether post-visit glycemic control can be achieved under each feasible regimen. The search stops when it finds the first regimen predicted to bring the patient into glycemic control, and AIDA recommends the minimum dose estimated to be required for glycemic control (see “Methods”). If glycemic control is not predicted to be possible, AIDA recommends the regimen with the lowest predicted post-visit HbA1c. Supplementary Notes 1 and 2 describe the stepwise search and dose tuning procedures, respectively.Fig. 1 Architectural overview of AIDA.

Given a patient’s visit medical profile, AIDA identifies potential regimens, filters these down to a ranked list of feasible regimens using knowledge-based diabetes management guidelines, and then uses a heuristic search to identify the regimen that would optimize glycemic control relative to an individualized target T. The heuristic starts with the previous prescription regimen and considers feasible changes, stepwise, in order of increasing intensity while also incorporating medication prioritizations as per the guidelines. The steps are: (1) drop medication and/or lower dose if any contraindications or renal dosing requirements, (2) increase dose of an existing medication, (3) add one new strongly prioritized medication (e.g., SGLT2-i, GLP-1 RA) if eligible, (4) substitute one new medication, or (5) add one new medication. At each step, the search uses AIDA’s HbA1c prediction model to assess whether there is any feasible regimen that is expected to enable glycemic control (predicted post-visit HbA1c ≤ T)—if yes, AIDA selects the highest-ranking regimen expected to enable glycemic control and recommends the minimum dose required for glycemic control; else AIDA proceeds to consider options in the next step. If glycemic control is not predicted to be possible, AIDA recommends the feasible regimen with the lowest predicted post-visit HbA1c.

Overall, our stepwise approach makes this complex search problem tractable, and crucially, enables AIDA to identify the minimal drug mix and/or dose changes required to optimize glycemic control.

AIDA recommendations are associated with improvements in glycemic control

First, we evaluated the performance of AIDA’s HbA1c prediction model against observed glycemic outcomes under ‘standard of care’ (SoC) prescriptions (i.e., regimens prescribed by clinicians during outpatient visits). On test sets IV, EV, and TV, AIDA’s XGBoost regression model achieved RMSE of 0.89%, 0.88%, and 1.00%, respectively, for post-visit HbA1c prediction. These are lower than the RMSE values of 1.05%, 1.04%, and 1.33% achieved by a comparator that simply uses the most recent HbA1c measurement i.e., the pre-visit HbA1c (Supplementary Table 3). Applying the regression model to classify whether post-visit HbA1c is within the glycemic control target gave F1 scores of 0.82, 0.81, and 0.77. The most important features, based on the average performance gain, primarily pertained to the pre-visit and other recent HbA1c readings, and previous and current visit prescription information (Supplementary Table 4).

Next, we compared potential post-visit HbA1c outcomes under AIDA with the observed post-visit HbA1c outcomes under SoC. Specifically, we used an off-policy evaluation method (weighted importance sampling) to estimate reductions in post-visit HbA1c under AIDA, relative to the pre-visit HbA1c, and compared them to observed reductions in post-visit HbA1c under SoC (see “Methods”). For visits where AIDA recommends a treatment intensification, the estimated mean (95% confidence intervals) post-visit HbA1c reduction, relative to the pre-visit HbA1c, under AIDA was 0.71% (0.59%, 0.83%) for test set IV, 0.93% (0.72%, 1.12%) for test set EV and 0.96% (0.83%, 1.11%) for test set TV (all P < 0.0001, Fig. 2a). In comparison, the mean (95% confidence intervals) post-visit HbA1c reduction under the SoC was only 0.24% (0.21%, 0.27%) for test set IV, 0.25% (0.22%, 0.28%) for test set EV, and 0.56% (0.51%, 0.60%) for test set TV (all P < 0.0001, Fig. 2a). These improvements are also reflected in the resulting post-visit HbA1c outcomes under AIDA and SoC, which have different means with distinctly non-overlapping 95% confidence intervals (Fig. 2b). Correspondingly, the estimated mean (95% confidence intervals) post-visit HbA1c benefit for AIDA compared to the SoC was 0.47% (0.36%, 0.58%) for test set IV, 0.68% (0.48%, 0.86%) for test set EV, and 0.40% (0.28%, 0.53%) for test set TV (all P < 0.0001). Moreover, the 95% confidence lower bounds for estimated post-visit HbA1c outcomes under AIDA were well above typical thresholds for hypoglycemia. In addition, post-visit HbA1c reductions and outcomes estimated under a no-change recommendation generally follow expected trends, especially in relation to AIDA and SoC.Fig. 2 Quantitative comparison of post-visit glycemic outcomes under AIDA, standard of care (SoC) and a no-change recommendation in test sets IV, EV, and TV.

Comparisons focus on visits where AIDA recommended a treatment intensification relative to the previous prescription (8,802 test set IV, 7334 test set EV, and 5774 test set TV visits with mean pre-visit HbA1c of 8.20%, 8.10%, and 8.36%, respectively). a HbA1c changes (defined as the pre-visit HbA1c subtracted from the post-visit HbA1c) estimated under no-change recommendations (green), observed under the SoC prescriptions (blue), and estimated under AIDA recommendations (orange). Bars indicate mean HbA1c changes with 95% CIs. b Post-visit HbA1c outcomes estimated under no-change recommendations, observed under the SoC prescriptions, and estimated under AIDA recommendations. Bars indicate mean post-visit HbA1c with 95% CIs. More negative changes indicate higher reductions (improvements), and lower post-visit HbA1c indicates better glycemic control. Mean post-visit HbA1c reductions and outcomes under AIDA were better than those under SoC and no-change recommendations (P < 0.0001, one-sided paired Student’s t-test).

Notably, the above prediction and optimization results were consistent in sensitivity analyses across five random data splits (Supplementary Tables 5 and 6). Further, estimated post-visit HbA1c reductions and benefits under AIDA were significant for most of 17 clinically relevant subgroups (Supplementary Fig. 2).

AIDA recommends intensifying treatment to target more frequently than clinicians

We examined the breakdown of prescription change types (e.g., no change, dose increase, introduction of new oral antidiabetic drugs (OADs) and injectables) under AIDA and SoC as a function of pre-visit HbA1c (Fig. 3). For cases in glycemic control, both SoC and AIDA tend to recommend no changes. However, for cases not in glycemic control, SoC exhibits substantial inertia to intensify treatment whereas AIDA recommends intensifying treatment to just the extent required for glycemic control. Specifically, for cases not in glycemic control, SoC prescribes no change, dose increase, and new drugs in 48.4–58.4%, 27.8–30.1%, and 10.8–22.0% of visits, respectively (ranges indicate variation across the three test sets). In contrast, AIDA recommends dose increase (where sufficient, in 13.4–15.2% of visits, primarily when pre-visit HbA1c exceeds individualized target by less than 1%), introduction of new OADs (most common, in 62.4–63.2% of visits), and injectables (where essential, in 19.8–21.7% of visits, commonly when pre-visit HbA1c exceeds individualized target by over 1%).Fig. 3 Breakdown of recommended change types under AIDA and SoC.

Plots summarize prescription patterns (i.e., proportions of visits by recommended change types) under SoC (blue bars) and AIDA (orange bars) as a function of pre-visit HbA1c ranges, for all visits in test sets (a) IV, (b) EV, and (c) TV. For each bar, the different change types, depicted with gradient color palettes, are defined relative to the previous visit prescription as: 1. Decrease in regimen severity level; 2. No change in drug mix and dose; 3. Increase dose of any existing medication; 4. Introduce (either add or substitute) a new OAD; or 5. Introduce (either add or substitute) a new injectable. The numbers above the bars indicate the number of visits for each pre-visit HbA1c range.

AIDA recommendations are reasonable and precise

Having characterized quantitative performance, we also assessed the quality of AIDA’s treatment recommendations. Specifically, we conducted qualitative evaluations on a set of 60 diverse cases (Supplementary Table 8) with a panel of three senior practicing endocrinologists (see “Methods”). Each evaluator reviewed each case independently and assessed the extent to which AIDA’s recommendation is sensible (i.e., consistent with established clinical practice norms) and precise (i.e., provides sufficient specifics to inform practical drug mix and dose decisions). Following blinded reviews, cases where evaluators had different opinions were adjudicated for consensus opinion. AIDA recommendations were mostly rated as clinically reasonable (clinically sound in 39 of 60 cases and acceptable in 20 of 60 cases), as not sensible in 1 of 60 cases, and as sufficiently precise in 57 of 60 cases (Table 2).Table 2 Qualitative evaluation of AIDA recommendations

Metric	Evaluation panel consensus	Case counts	
How sensible	Good (Clinically Sound)	39	
Fair (Acceptable)	20	
Poor (Not Sensible)	1	
How precise	Sufficiently precise	57	
Somewhat precise	3a	
Not precise	0	
Case counts are based on consensus ratings by a panel of three senior practicing endocrinologists for the 60 cases evaluated.

aCases where AIDA introduced insulin without dose (see “Methods”).

AIDA’s ability to account for drug-dose specifics offers greater clinical relevance than comparators

We also assessed the value-add of including drug-dose specifics within AIDA by benchmarking recommendations from AIDA with those from two comparators that omit drug name and/or dose specifics. For the first comparator, we implemented a previously proposed Drug Class-combination Advisor (DCA) that recommends a line of therapy defined as a categorical combination of metformin, non-metformin oral agents, and/or injectables (e.g., “metformin, an injectable, and one other drug”)17. For the second comparator, we constructed a Drug Mix Advisor (DMA) that recommends a combination of generic drug names (e.g., “metformin, sitagliptin”) using the same approach as AIDA but without consideration of dose (see “Methods”).

First, the panel of three evaluators comparatively assessed recommendations from AIDA and the competing methods for the 60 cases (Table 3). Generally, AIDA was rated as clinically more sensible and more precise than the comparators. Specifically, evaluators found AIDA to be more sensible than DCA in 54 of 60 cases and more sensible than DMA in 14 of 60 cases. Moreover, even when AIDA’s recommendations were rated as similarly sensible to DCA and/or DMA recommendations (5 and 39 cases, respectively), there were important distinctions in actionability. Notably, as the comparators generate recommendations without drug name specifics (DCA) and/or dose information (DMA, DCA), evaluators noted that they (especially DCA) were imprecise, leave room for considerable ambiguity, and hence could have limited clinical utility (precision ratings, Table 3). Of the two comparators, the one with the more granular medication representation, i.e., DMA was rated better.Table 3 Comparative assessment of recommendations from AIDA and competing methods

Metric	Evaluation panel consensus	Drug Class-combination Advisor (DCA)	Drug Mix Advisor (DMA)	
How sensible	AIDA is more sensible	54	14	
AIDA is similarly sensible	5	39	
AIDA is less sensible	1	7	
How precise	AIDA is more precise	60	36	
AIDA is similarly precise	0	24a	
AIDA is less precise	0	0	
Counts indicate the number of cases where the recommendation from AIDA was rated as more, similarly, or less sensible or precise compared to the recommendation from the competing method (DCA, DMA). Case counts are based on consensus ratings by a panel of three senior practicing endocrinologists for the 60 cases evaluated.

aCases where AIDA introduced insulin without dose (see “Methods”), AIDA and/or DMA recommended no change, or DMA introduced drugs with fixed starting dose.

Therefore, to glean insights on the key advantages offered by AIDA, we further examined the 14 cases where the evaluation panel favored AIDA over DMA for any underlying themes. We found that AIDA’s advantages over DMA primarily stemmed from the ability to (A) optimize dose and (B) better prioritize medications—in line with specialist practice preferences and published clinical practice standards, as illustrated by examples in Table 4. For exemplar case A, AIDA recommended increasing dose whereas DMA recommended insulin addition. Evaluators favored AIDA’s recommendation as this minimal change is safer in a case of mild hyperglycemia and likely to be more acceptable to patients. Such differences where AIDA increased dose and DMA added medications tended to occur in milder hyperglycemia cases even in the larger test sets. For exemplar case B, AIDA recommended adding dapagliflozin (an SGLT2-i) while DMA recommended adding linagliptin. Evaluators favored AIDA’s recommendation as there is strong evidence on the benefits of prioritizing SGLT2 inhibitors in patients with cardiorenal conditions. Medication preference advantages were not only limited to cases with cardiorenal comorbidities, and also extended to better alignment to clinician preferences in other cases. These advantages lend some insights into how AIDA’s ability to optimize over drug-dose specifics could enhance the quality of treatment recommendations.Table 4 Illustration of main advantages of AIDA over competing methods

Type of advantage	Dose optimization	Better medication prioritization	
Exemplar ID	A	B	
Age (years)	59	77	
Gender	Male	Male	
Ethnicity	Indian	Chinese	
Pre-visit HbA1c (%)	7.4%	8.3%	
eGFR (mL/min/1.73 m2)	119	54	
Relevant comorbidities and complications	Diabetic foot/peripheral angiopathy

Macro- and micro-vascular complications

	Hypertension

Hyperlipidemia

	
Previous visit prescription	Glipizide: 5 mg

Metformin: 1000 mg

	Metformin: 500 mg	
AIDA Recommendation	Increase metformin dose: +1550 mg	Add dapagliflozin: +10 mg	
Drug Mix Advisor (DMA) Recommendation	Add insulin glargine	Add linagliptin	
Drug Class-combination Advisor (DCA) Recommendation	Add OAD	Add OAD	
Evaluation Panel Recommendation (Consensus)	Increase metformin dose or substitute glipizide with SGLT2 inhibitor	Add SGLT2 inhibitor	
Evaluation Panel Opinion (Consensus)	AIDA’s ability to optimize dose will improve the adoption of recommendations. AIDA’s recommendation is also safer than that of DMA as it has lower risk of hypoglycemia	AIDA’s recommendation is superior to that of DMA in terms of HbA1c lowering and cardio-renal protection in a patient with chronic kidney disease indication.	
The two main advantages arising from AIDA’s ability to account for drug and dose specifics, as identified in the comparative assessment with respect to competing methods (DCA, DMA). For each advantage, an illustrative patient visit example from the qualitative evaluation is provided. All treatment recommendations are presented as changes relative to the previous visit prescription.

Finally, we assessed the degree to which AIDA’s medication representation enables gains in guideline consistency. We considered the set of medication contraindications, drug combination restrictions and renal dosing guidelines specified within the treatment recommendation task, and compared rates of concordance with these guidelines for AIDA, DMA, and DCA (Table 5). In our test sets, the above guidelines were found to be applicable in 59.4–76.1% of visits. For these visits, rates of concordance with the above guidelines under AIDA, DMA, and DCA were 100% (by design), 73.0–77.1%, and 15.0–20.2%, respectively. In other words, AIDA could ensure guideline concordance in 22.9–27.1% more relevant visits than DMA and in 79.8–85.0% more relevant visits than DCA. The difference between AIDA and DMA was due to renal dosing, as DMA is unable to titrate medication doses to account for this safety requirement. Differences between AIDA and DCA were because DCA could represent only the metformin contraindication requirement. Therefore, AIDA’s ability to optimize over specific drug mix and dose choices enables a sizeable boost in the degree of concordance with clinical guidelines, enhancing utility and actionability.Table 5 Guideline concordance improvements enabled by AIDA over competing methods that do not account for drug and dose specifics in making recommendations

Test set	Relevant visits	Proportion of relevant visits with guideline concordance advantage(s) for AIDA over comparator	
DMA	DCA	
IV (16,653)	74.9% (12,478)	27.0% (3365)	85.0% (10,612)	
EV (14,316)	76.1% (10,888)	27.1% (2948)	84.2% (9173)	
TV (10,258)	59.4% (6098)	22.9% (1396)	79.8% (4865)	
Relevant visits are defined as those where any of the medication contraindications, drug combination restrictions and/or renal dosing guidelines specified within the task were applicable. For each test set, the improvement of AIDA over the comparator is defined as the difference between their guideline concordance rates (see “Methods”). Results are shown as proportions (number of visits).

Discussion

We introduced an AI Drug mix and dose Advisor (AIDA) that leverages real-world patient data alongside clinical knowledge-based guidelines to support physicians in individualized T2D management. To our knowledge, AIDA is unique in its ability to provide optimal drug mix and dose recommendations for individual T2D patients. We demonstrated that AIDA generates clinically reasonable and precise recommendations which were estimated to significantly improve HbA1c over standard of care in extensive evaluations on high-quality EMR data from diverse cohorts (>40,000 patients across several subgroups).

Previous works have proposed EMR-based treatment optimization approaches for individualized management of T2D17–21. However, these have been limited to making broader (largely drug class level) recommendations, independently of drug and dose specifics. As such, they leave key components of practical prescription decisions unresolved and place the onus on clinical end-users to filter amongst the many possibilities and identify the optimal treatment regimen. In contrast, AIDA’s recommendations are precise enough to inform the full spectrum of decisions on generic name(s), dosing, and change type, hence offering higher utility.

Interestingly, beyond the expected improvement in precision, AIDA’s ability to optimize over drug and dose level actions resulted in qualitatively distinct recommendations with many important acceptability, efficacy and safety advantages over comparators. For example, in the expert panel reviews, having the option to increase dose meant AIDA could avoid introducing new drugs and the associated costs and side effects for patients with mild hyperglycemia. Furthermore, relative to comparators, AIDA was more consistent with clinical practice preferences on medication prioritization, such as for patients with cardiovascular and renal comorbidities, highlighting potential to improve adoption of indication-based treatments with better outcomes2,3. Notably, AIDA’s medication representation and search approach uniquely enable it to incorporate multifaceted drug-specific and dose-dependent guidelines, resulting in safer, more guideline-concordant recommendations for a large proportion of visits. Beyond safety, explicitly accounting for these guidelines enables AIDA to estimate their impact on outcomes and thereby also generate more effective recommendations. For example, in cases requiring renal dose titration30, AIDA could consider the impact on glycemic outcomes and also introduce a more effective drug alongside. Together, these trends, observed in sizeable proportions of visits in our test sets, translate to greater ability to accommodate individual patient needs in alignment to practice preferences and clinical guidelines.

AIDA could be integrated with EMR systems and used to support individualized prescription decisions at point-of-care. Initial use cases could focus on primary care settings14, where most routine T2D management takes place. As AIDA can systematically assess if there is a need to intensify treatment, and then identify the minimal prescription changes to optimize glycemic control, it has considerable potential to mitigate therapeutic inertia12,13,15,16. Further, AIDA can be embedded into an interactive decision support dashboard (example in Supplementary Fig. 3) that provides individualized quantitative assessments to inform patient-clinician discussions on the best treatment options. Hence it could pave the way for data-driven shared decision-making paradigms31,32.

Furthermore, our approach is also relevant for individualizing multi-drug therapy in related chronic conditions. Our drug-dose recommendation methodology is anchored in the implicit structure of clinical decision-making in settings where the goal is to progressively intensify prescriptions to achieve a measurable treatment target. Therefore, it could readily be extended to conditions such as hypertension, hyperlipidemia, and rheumatoid arthritis33,34.

Our work has limitations. First, since AIDA relies entirely on EMR data, it is affected by data quality issues and lacks visibility on factors beyond the EMR that may influence treatment response or adoption. We have addressed some of these challenges with careful feature engineering and knowledge-base construction. However, future enhancements with large language models and/or interactive tools to access patient factors beyond structured EMR data, and/or integration with data from devices and mobile applications tracking blood sugar, physical activity, or diet would be beneficial. Second, although our results show that AIDA could improve glycemic control on external and temporal test sets, our quantitative evaluations are limited to counterfactual outcome estimates based on observational data generated under clinicians’ prescriptions. While we have adopted a widely used and theoretically principled off-policy evaluation method to estimate outcomes, follow-on randomized interventional studies with monitoring of acceptance, adherence and outcomes would more fully establish impact in real-world practice settings. Finally, AIDA currently focuses on lowering next visit HbA1c to an individualized glycemic target. While this approach is suitable for the average hyperglycemic patient, our qualitative evaluations highlighted the need to (1) expand the KDM guidelines for better management of patients with pre-visit HbA1c well within and well above their glycemic target2,3,35; and (2) offer multiple recommendations especially in cases where multifactorial considerations beyond the EMR such as capillary blood glucose readings or tolerance may be important. As clinical guidelines change regularly and new drugs are constantly being introduced, the knowledge-based guidelines and data pipelines underlying AIDA would need to be periodically refined to ensure up-to-date recommendations. These considerations are easy to implement in future revisions, as our approach is flexible enough to incorporate and align with any specified set of guidelines and treatment options.

In summary, we introduced a drug mix and dose decision algorithm to optimize patient-centered glycemic management in type 2 diabetes. Our approach generates meaningful, precise, and effective prescription recommendations aligned with clinical practice norms and guidelines, and has the potential to mitigate the considerable empiricism and inertia prevailing in routine diabetes care. With further validation, our algorithm could be deployed to systematically support clinicians in making improved treatment decisions for individual T2D patients at point-of-care, especially in primary care or resource-limited settings. More broadly, our work highlights that including drug-dose specifics in the medication representation is essential for the acceptability, safety, and efficacy of AI treatment advisories. Furthermore, our work motivates greater focus on guideline concordance characterizations and clinical expert evaluations to translate AI treatment recommendation systems for chronic disease management.

Methods

Study design and datasets

We used anonymized electronic medical records from the SingHealth Diabetes Registry (SDR)29. Our study was approved by the SingHealth Centralized Institutional Review Board (Protocol No. 2019/2414) and the A*STAR Institutional Review Board (Protocol No. 2019-079) with a waiver of informed consent, and fully complied with all relevant ethical regulations.

The SDR contains EMR data for 220,618 adult T2D patients with 3,513,887 ‘prescription visits’ (outpatient visits with glycemic control prescriptions) to any of 22 institutions (ten primary care clinics, nine acute hospitals and specialty centers, three community hospitals) within Singapore’s largest health cluster, SingHealth, between January 2013 to December 202029. Following previous work17, we excluded patients with a recorded history of type 1 or gestational diabetes as decisions on pharmacotherapy in these patients often require thorough reviews of self-monitored blood glucose readings, which were not available in the SDR. Other patients were included if they had outpatient prescription visits meeting the following criteria: (i) preceded by at least one prior prescription visit; (ii) preceded by at least one HbA1c measurement within 30 days before the visit; (iii) followed by at least one HbA1c measurement two to six months after the visit; and (iv) not followed by a visit with a prescription change (relative to the current visit prescription) before the next HbA1c measurement. These criteria, applied to enable meaningful modeling, yielded a total of 107,854 eligible T2D patients with 1,039,869 prescription visits.

We divided the set of eligible patients into four independent cohorts (Supplementary Fig. 1) to define a model development set (21 sites, visits during 2013–19), an internal validation set IV (21 development sites, 2013–19), an external validation set EV (external site designated as the primary care clinic with the largest number of prescription visits, 2013–19), and a temporal validation set TV (any of 22 sites, visits during 2020). The internal validation set reflects diverse care settings for assessing model reproducibility. The external site was designated as a large primary care clinic to assess generalizability in settings where the decision algorithm is likely to be adopted. The model development, internal validation and external validation sets were defined at the start of the study (when only 2013–19 data were available), while the temporal validation set was only defined prospectively when the 2020 data became available.

To ensure that patients with more visits are not over-represented in evaluations, we randomly sampled only one visit per patient in each test set. To ensure that the test sets reflect their respective originating data distributions, all sampling procedures were stratified36 based on age, gender, ethnicity, pre-visit HbA1c, visit year, care setting (primary care or hospital), and the incidence of comorbidities and complications relevant to T2D management. Specifically, we divided visits into subgroups (or strata) based on the above characteristics, and randomly sampled visits from each subgroup in proportion to the size of the subgroup.

Data preprocessing

For each dataset, we extracted a set of 38 EMR variables, including demographics, physical measurements, laboratory values, medical history, and prescription details (Supplementary Table 1). For every prescription visit, we defined a ‘visit medical profile’ comprising patient characteristics, pre-visit medical history, and the previous visit prescription; an in-visit ‘treatment recommendation’ represented as the prescribed set of generic name antidiabetic drugs (‘drug mix’), their daily dosages, and the associated regimen ‘severity level’ (defined based on the number of oral antidiabetic drugs (OADs) and/or type of injectables, as below); and a ‘post-visit HbA1c’ outcome used to define degree of glycemic control.

First, we processed and coded medical profile features at the prescription visit level, as follows. We represented physical measurements and laboratory values using the most recent measurements and the quartile summary statistics (25th, 50th, 75th percentiles) of measurements taken within 1 year before the visit. We denoted the last HbA1c reading before the visit as the ‘pre-visit HbA1c’. We mapped diagnosis codes from any previous visits to features indicating past incidence of relevant comorbidities (hypertension, hyperlipidemia, cardiovascular disease, cerebrovascular disease), complications (including macrovascular, microvascular, and other diabetes complications), and other conditions that could influence treatment decisions (including severe hypoglycemia, keto-/lactate acidosis, hepatic insufficiency, and pancreatitis). We used previous surgery codes to infer past incidence of lower limb amputation, bariatric surgery and vascular surgery. We also included features representing whether or not the patient was previously prescribed any lipid-lowering, antihypertensive, and antiplatelet drugs. We imputed missing data using mean values in the model development set. We represented the previous visit prescription as the prescribed set of generic name antidiabetic drugs (drug mix) and their daily dosages. Drug mix was denoted as a subset of 31 generic name antidiabetic drugs (OADs, injectables, combination drugs) prescribed within the model development set (Supplementary Table 2). Each prescription was associated with a regimen ‘severity level’ defined as: (1) 1 oral antidiabetic drug (OAD); (2) Multiple OADs; (3) First injectables (glucagon-like peptide-1 receptor agonists (GLP-1 RA) or long-acting insulin); (4) Premixed insulin; and (5) Rapid-acting insulin. Second, we encoded the in-visit treatment recommendation as the change in drug mix, daily dosage, and severity level, relative to the previous prescription. Third, we coded the post-visit HbA1c as the mean of all HbA1c readings taken two to six months after the visit.

Treatment recommendation task

Given input of a patient’s visit medical profile, the task is to recommend the drug mix and daily dose regimen that optimizes glycemic control (defined as achieving a post-visit HbA1c at or under the patient’s individualized HbA1c target), subject to clinical knowledge-based guidelines. We set individualized HbA1c targets based on age, diabetes duration, and medical history37. In general, patients had a glycemic target of 7.0%. However, for patients aged <50 years with diabetes duration of <10 years and no prior history of atherosclerotic cardiovascular disease, we adopted a tighter target of 6.5%. For patients aged 75–79 years, we set the target at 7.5%. For patients aged ≥80 years, we relaxed the target to 8.0%.

Treatment options

We defined the drug mix options as all possible subsets of 24 generic name antidiabetic medications: comprising all OADs, injectables or combination drugs available in the SDR, except those with very low outpatient utilization rates (Supplementary Table 2). For non-insulin medications, we defined cumulative daily dosage options using local formulary configurations30. For insulins, since exact dose titration often relies on factors beyond the EMR, including self-monitored capillary blood glucose readings, we limited dosage options to “maintain” or “increase” relative to the previous prescription.

Clinical knowledge-based guidelines

To facilitate meaningful treatment recommendations, we distilled established clinical practice standards published by the American Diabetes Association (ADA)2, the European Association for the Study of Diabetes3, and the Agency for Care Effectiveness Singapore38; drug label requirements from MIMS Singapore30 as of August 2021; and the experience of clinical experts in managing T2D into a compilation of knowledge-based diabetes management (KDM) guidelines. We included important medication contraindications (based on age, estimated glomerular filtration rate or eGFR, medical history); restrictions on drug combinations (based on previous visit prescription of sulfonylurea, meglitinide, premixed or rapid-acting insulin, DPP-4 inhibitor, and/or GLP-1 RA); renal dosing requirements (based on eGFR and previous prescription information); restrictions on introducing medications with higher risk of hypoglycemia (based on patient age, HbA1c and glycemic target); as well as drug prioritizations. For drug prioritizations, we specified the general ranking order based on multifactorial practice requirements as (1) Metformin, (2) SGLT2-i, (3) GLP-1 RA, (4) DPP-4 inhibitor, (5) long-acting insulin, and (6) others, with stronger prioritization of metformin in cases with no active metformin prescription and of SGLT2-i and GLP-1 RA based on age and cardiorenal comorbidities. In constructing the knowledge representation, we also incorporated key institutional considerations implicit in the SDR data. The KDM guidelines are summarized in Supplementary Table 2.

Drug mix and dose decision algorithm

Our decision algorithm, AIDA, employs a predict-then-optimize approach. AIDA uses an HbA1c prediction model to evaluate the glycemic control objective; and a heuristic to optimize this objective subject to the KDM guidelines specified above (Fig. 1). AIDA’s treatment options comprise combinations of generic drug names and their daily dosages, resulting in recommendations of the form “metformin: 1000 mg/day, sitagliptin: 50 mg/day, insulin glargine: increase”.

Prediction model

Given the visit medical profile and a plausible in-visit treatment recommendation, the modeling task is to predict the post-visit HbA1c. For the visit medical profile, we performed feature selection using the Least Absolute Shrinkage and Selection Operator (LASSO)39 regression method on the model development set. We applied LASSO with 4-fold cross-validation and grid search across hyperparameters controlling the regularization strength, and used the model with the highest R2 to select the non-zero features. For the in-visit treatment recommendation, we included the complete drug mix and dose regimen feature set. We trained the prediction model using eXtreme Gradient Boosting (XGBoost) for regression, a gradient-boosted regression tree-based ensemble machine learning algorithm40. We tuned the hyperparameters (learning rate and number of estimators) using 4-fold cross-validation and grid search across five learning rate settings {0.001, 0.005, 0.01, 0.05, 0.1} and three number of estimators settings {500, 1000, 1500}. We assigned the model with the highest R2 as AIDA’s prediction model. We further used the XGBoost regression model prediction to classify whether the post-visit HbA1c is within the individualized glycemic control target, so as to inform the decision algorithm.

Optimization details

AIDA’s search heuristic is designed to carefully consider dose effects and practical constraints on severity levels. First, to evaluate possible treatment intensifications (Fig. 1, steps 2–5), we leveraged the established dose response property that treatment effects should be increasing in dose (up to specified maximum ranges)41. Hence, for a given treatment regimen, if glycemic control is not predicted to be possible at a given dose, then it is also not likely to be possible at lower doses. Based on this property, AIDA uses maximum feasible doses to evaluate whether a given treatment regimen is likely to bring the patient into glycemic control. Then, once the optimal treatment regimen is identified, AIDA estimates the minimum dose required for glycemic control. Specifically, it searches the set of feasible doses in descending order, evaluates the corresponding post-visit HbA1c for each feasible dose, and returns the lowest dose seen such that glycemic control is possible. Further, to limit over-intensification for safety and practicability, we capped severity level changes to one if pre-visit HbA1c < max{8%, individualized HbA1c target + 1%} and two otherwise. Formal algorithm statements are provided in Supplementary Notes1 and 2.

Quantitative performance evaluations

To evaluate performance of AIDA’s HbA1c regression model, we computed RMSE, R2, and MAPE using observed (ground-truth) post-visit HbA1c measurements under standard of care (SoC) prescriptions. In addition, we categorized whether the predicted values were within the individualized HbA1c target and computed the F1 score.

Further, we compared post-visit HbA1c outcomes under AIDA recommendations, the SoC prescriptions and a no-change recommendation. For these comparisons, we considered visits where AIDA recommends a treatment intensification (increase dose, substitute or add medications). To ensure a fair comparison between AIDA and the SoC, we only allowed AIDA to recommend medications that were available at the time of any given visit. This required disallowing AIDA from recommending drugs approved after January 1, 2013 (i.e., SGLT2 inhibitors) for any visits preceding the drug’s earliest prescribed date in our dataset. Since outcomes under AIDA and no-change recommendations are not observable, we estimated them using weighted importance sampling (WIS)42, an off-policy evaluation technique widely adopted in the treatment optimization literature21,43–45. Weighted importance sampling approximates the expectation of a function of a random variable X with respect to a target distribution p by computing the weighted average of samples from another distribution q. More specifically, given samples of X, xi, i=1,…, n, from distribution q, the expectation of a function f under distribution p is approximated by the WIS estimator:1 WIS=∑i=1nwxifxi∑i=1nwxi,

where the weights w(xi) are given by2 wxi=pxiqxi.

In order to estimate the expected HbA1c outcome and reduction under AIDA (or no-change) recommendations, we require the probability of a given recommendation under the SoC (distribution q) and under AIDA (or no-change) (distribution p). To estimate distribution q, we trained multinomial logistic regression models on each test set to classify SoC treatment recommendations given patient visit medical profiles. For this task, we represented treatments at the drug class level18,20 and set the one-hot-encoded selected treatment as the output, and used the same input features as AIDA’s XGBoost model while excluding all features describing the current visit treatment regimen. These models perform reliably (Supplementary Table 7). To obtain the probability distribution p, we softened AIDA (or no-change) recommendations43 so as to recommend the suggested treatment regimen with 0.99 probability and any other treatment regimen uniformly at random with a total of 0.01 probability. To estimate the distributions of HbA1c outcomes and reductions under SoC, AIDA, and no-change recommendations, we used bootstrapping with 500 resamplings of 80% of the data. For each bootstrap sample, we computed the mean post-visit HbA1c outcomes under the SoC, AIDA, and no-change recommendations, respectively. We computed the differences between the mean pre-visit and post-visit HbA1c values to obtain the mean post-visit HbA1c reductions observed under the SoC, and estimated under the AIDA and no-change recommendations. We also computed differences between the post-visit HbA1c outcomes under AIDA and SoC to estimate the post-visit HbA1c benefits under AIDA, relative to SoC. We then computed 95% confidence intervals for all estimates across the 500 bootstrap samples.

For statistical testing, we hypothesized that the estimated mean post-visit HbA1c under AIDA would be lower than the mean pre-visit HbA1c. Further, we hypothesized that the estimated mean post-visit HbA1c reduction under AIDA would be better than that observed under the SoC prescription and that estimated under the no-change recommendation. We assessed significance levels using one-sided Student’s t-tests46. We repeated evaluations under five random 80/20 splits of data from the development sites to assess sensitivity to train/test data splitting; and for 17 subgroups based on demographic factors, body mass index (BMI), eGFR, and pre-visit HbA1c.

Comparators that omit drug-dose specifics

We compared AIDA to two competing methods that consider less granular treatment options.

The first comparator is a previously developed treatment advisory17 which defines treatment options as combinations of drug classes (e.g., “metformin, an injectable, and one other drug”). This comparator, which we term as Drug Class-combination Advisor (DCA), uses k-nearest neighbors to predict post-visit HbA1c under different drug class combinations and recommends the combination with lowest predicted post-visit HbA1c. DCA differs from AIDA in granularity of its medication representation as well as its visit medical profile feature representation and optimization objective (unlike AIDA’s objective to achieve individualized glycemic targets, DCA minimizes post-visit HbA1c). Our implementation followed the original publication17, except that we excluded ‘no regimen’ from the menu of treatment options and expanded the definition of injectables to include GLP-1 RA. More specifically, we used the following treatment options: (1) metformin monotherapy, (2) injectable monotherapy, (3) one non-metformin oral agent, (4) metformin and one other oral agent, (5) injectable and metformin, (6) injectable and one non-metformin agent, (7) two non-metformin oral agents, (8) metformin and two other oral agents, (9) injectable and metformin and one other agent, (10) injectable and two non-metformin agents, (11) three non-metformin oral agents or at least four agents.

The second comparator, termed Drug Mix Advisor (DMA), is an ablated version of AIDA, which we constructed to pinpoint the impact of including granular dose information on the recommendations. DMA’s treatment options are defined as a combination of generic drug names (e.g., “metformin, sitagliptin”). DMA is identical to AIDA in all aspects including representation of patient characteristics and medical profile, optimization objective, guidelines, prioritization structure, and search approach, except in granularity of its medication representation (previous visit prescription and in-visit recommendation). Specifically, unlike AIDA, DMA does not use dose information (i.e., the daily dosage features/options) either for HbA1c prediction or for optimization. This difference gave rise to drug class level differences between AIDA’s and DMA’s recommendations for 38.9–44.3% of visits eligible for treatment intensification in our three test sets. As DMA has a more granular medication representation than prior works17–21 which predominantly represent medications at the drug class or drug type level, it serves as a compelling comparator.

For fair comparisons, we tuned HbA1c prediction models used by both the comparators to achieve R2 as close to AIDA’s HbA1c model as possible. Further, we ensured that each comparator incorporated all the specified KDM guidelines that are permitted by its architecture and medication representation. More details on treatment options and knowledge-based guidelines for the comparators (in relation to AIDA) are in Supplementary Table 2.

Qualitative evaluations

To evaluate the quality of treatment recommendations, we conducted case reviews with a panel of specialists. We purposively sampled cases from the test set IV to represent a diverse set of patient visits with sufficient coverage for important clinical concepts and scenarios, as follows. We defined case profiles based on combinations of five primary criteria: (a) age, (b) care setting, (c) pre-visit HbA1c, (d) eGFR, and (e) regimen severity levels. An example of a case profile is ‘young adult seen at a primary care clinic with pre-visit HbA1c 7–8%, eGFR ≥ 60 ml/min/1.73 m2, and regimen of severity level 1’. We prioritized 60 case profiles with high prevalence in test set IV, while allowing for diversity. For each case profile, we randomly sampled 1 patient visit from the subset of test set IV visits that match the profile, while ensuring diversity and representation across secondary criteria covering (a) drug classes in active prescription, (b) number of comorbidities and (c) types of comorbidities. We also ensured that the selected cases represented the ratio of cases with differences between AIDA and the comparators in the overall test set. The characteristics of the 60 cases are provided in Supplementary Table 8, and generally follow relevant prevalence trends in the test set.

The evaluation panel comprised three senior clinicians with 17, 10, and 10 years of experience as practicing endocrinologists (Y.M.B., D.C., and P.C.L.). The evaluators first independently reviewed the cases, as follows. For each case, each evaluator, masked to SoC and algorithm recommendations, reviewed the case and noted his/her treatment recommendation. Thereafter, he/she reviewed treatment recommendations from AIDA, and responded to an open-ended questionnaire on the extent to which AIDA’s recommendation is sensible (i.e., consistent with established clinical practice norms) and precise (i.e., provides sufficient specifics to inform practical drug mix and dose decisions), and on how AIDA’s recommendation compares with the competing methods.

Two members of the research team (I.H.M., P.K.) independently analyzed the evaluators’ assessments. Specifically, they coded free-text comments from each evaluator into ordered categories defined as clinically sound (expected to pass muster with a sizeable number of medical professionals), acceptable (suboptimal but not wrong or unsafe, can be encountered in real-world settings), and not sensible (unsafe or not acceptable by clinical standards). The analysts also identified emergent themes47 for comparisons between AIDA and the competing methods. For cases where both analysts found the three evaluators to be in general agreement, we report the apparent consensus opinion. For cases where there were differing opinions, we conducted an adjudication session to resolve differences and obtain consensus opinion of the evaluation panel. In total, 28 cases had to be adjudicated for the question of whether AIDA sensible, 13 cases for the AIDA to DMA comparison, and 5 cases for the AIDA to DCA comparison. In 10 cases, consensus could not be reached, and the panel chose to report opinions by majority vote. Further, the research team discussed observations on comparisons between AIDA and the competing methods with the evaluators to obtain consensus on AIDA’s main advantages and disadvantages. For advantages, we identified illustrative cases to exemplify the themes. In cases with disadvantages or limitations, we identified corrective actions needed for future refinements. All reported results are based on the consensus opinion of the evaluation panel. A summary of cases and assessments is available upon request.

Guideline concordance analysis

Finally, we compared AIDA and the competing methods by the degree of concordance with the medication contraindication, drug combination restriction, and/or renal dosing guidelines specified within the treatment recommendation task (Supplementary Table 2). We first considered test set visits for which one or more of the above specified guidelines apply. Then, for each of these ‘relevant visits’, we assessed treatment recommendations from AIDA, DMA, and DCA. We designated a recommendation as concordant only if it definitively aligns to all of the above applicable guidelines. For each test set and treatment advisor, we quantified guideline concordance rates as the fraction of relevant test set visits with concordant recommendations.

Supplementary information

Supplementary Material

DECIDE-AI Checklist

Supplementary information

The online version contains supplementary material available at 10.1038/s41746-024-01230-5.

Acknowledgements

This research is supported by Agency for Science, Technology and Research (A*STAR), Singapore under its Industry Alignment Pre-Positioning Fund (Grant No. H19/01/a0/023 – Diabetes Clinic of the Future). The funders had no role in the study design, data collection, data analysis, data interpretation, or the writing of this manuscript. The authors would like to acknowledge helpful discussions with Guo Yang, Savitha Ramasamy, and Zhou Weizhuang, and support on compliance and governance from Nurensah Beevi at the Institute for Infocomm Research, A*STAR. We also thank the Singapore General Hospital Health Services Research Unit; and Luis Ng Chong Tin, Sonia Davila, Weng Khong Lim, and Patrick Tan at the SingHealth Duke-NUS Institute of Precision Medicine, Singapore for provisioning the computational resources and infrastructure for this work.

Author contributions

M.N., Y.M.B., and P.K. conceived the study and developed methodology; S.Y.C., N.N.M.S., and Y.M.B. contributed to data extraction and curation; M.N., Y.E.C., F.G., Y.M.B., D.C., P.C.L., I.H.M., and P.K. performed formal analyses; M.N., Y.E.C., and P.K. generated visualizations; M.N. and P.K. wrote the original draft of the manuscript with inputs from Y.E.C., I.H.M., and Y.M.B. All authors had full access to all the data in the study; and Y.M.B. and P.K have verified the data. All authors contributed to manuscript review and editing, and approved the final manuscript.

Data availability

The medical records dataset used in this study is not publicly available due to the need to protect patient privacy. Access to this dataset, i.e., the SingHealth Diabetes Registry, is possible on reasonable request to Dr. Yong Mong Bee at Singapore Health Services (bee.yong.mong@singhealth.com.sg), under restrictions subject to obtaining ethics approval from institutional review board(s) and an appropriate data use and/or research agreement.

Code availability

We have fully described our decision algorithm and analysis approaches in the manuscript so as to support the reported results. The underlying algorithm codebase is proprietary, and may be made available on reasonable request to Dr. Pavitra Krishnaswamy at the Institute for Infocomm Research, A*STAR (pavitrak@i2r.a-star.edu.sg) subject to licensing terms and conditions including restrictions of use. Source code for the analyses and evaluations can be found at https://github.com/health-intell/Drug-Mix-and-Dose-Decision-Algorithm-for-T2DM.

Competing interests

The authors declare no competing interests.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Melmed, S., Koenig, R., Rosen, C., Auchus, R. & Goldfine, A. Williams Textbook of Endocrinology - 14th Edition (Elsevier, 2019).
2. American Diabetes Association. 9. Pharmacologic approaches to glycemic treatment: standards of medical care in diabetes—2021. Diabetes Care 44, S111–S124 (2021).
3. Buse JB 2019 Update to: Management of hyperglycemia in type 2 diabetes, 2018. A consensus report by the American Diabetes Association (ADA) and the European Association for the Study of Diabetes (EASD) Diabetes Care 2019 43 487 493 31857443
Buse, J. B. et al. 2019 Update to: Management of hyperglycemia in type 2 diabetes, 2018. A consensus report by the American Diabetes Association (ADA) and the European Association for the Study of Diabetes (EASD). Diabetes Care 43, 487–493 (2019).31857443
4. Inzucchi SE Management of hyperglycemia in type 2 diabetes: a patient-centered approach: position statement of the American Diabetes Association (ADA) and the European Association for the Study of Diabetes (EASD) Diabetes Care 2012 35 1364 1379 22517736
Inzucchi, S. E. et al. Management of hyperglycemia in type 2 diabetes: a patient-centered approach: position statement of the American Diabetes Association (ADA) and the European Association for the Study of Diabetes (EASD). Diabetes Care 35, 1364–1379 (2012).22517736
5. Raz I Personalized management of hyperglycemia in type 2 diabetes: reflections from a Diabetes Care Editors’ Expert Forum Diabetes Care 2013 36 1779 1788 23704680
Raz, I. et al. Personalized management of hyperglycemia in type 2 diabetes: reflections from a Diabetes Care Editors’ Expert Forum. Diabetes Care 36, 1779–1788 (2013).23704680
6. Gimeno, J. A., Cánovas, G. & Durán, A. Factors associated with adherence to clinical practice guidelines for patients with type 2 diabetes mellitus: results of a Spanish Delphi Consensus. J. Diabetes Res. 9970859 (2021).
7. Phung OJ Scholle JM Talwar M Coleman CI Effect of noninsulin antidiabetic drugs added to metformin therapy on glycemic control, weight gain, and hypoglycemia in type 2 diabetes JAMA 2010 303 1410 1418 20388897
Phung, O. J., Scholle, J. M., Talwar, M. & Coleman, C. I. Effect of noninsulin antidiabetic drugs added to metformin therapy on glycemic control, weight gain, and hypoglycemia in type 2 diabetes. JAMA 303, 1410–1418 (2010).20388897
8. Greenfield S Comorbidity affects the relationship between glycemic control and cardiovascular outcomes in diabetes: a cohort study Ann. Intern. Med. 2009 151 854 860 20008761
Greenfield, S. et al. Comorbidity affects the relationship between glycemic control and cardiovascular outcomes in diabetes: a cohort study. Ann. Intern. Med. 151, 854–860 (2009).20008761
9. Turnbull FM Intensive glucose control and macrovascular outcomes in type 2 diabetes Diabetologia 2009 52 2288 2298 19655124
Turnbull, F. M. et al. Intensive glucose control and macrovascular outcomes in type 2 diabetes. Diabetologia 52, 2288–2298 (2009).19655124
10. Dennis JM Shields BM Henley WE Jones AG Hattersley AT Disease progression and treatment response in data-driven subgroups of type 2 diabetes compared with models based on simple clinical features: an analysis using clinical trial data Lancet Diabetes Endocrinol. 2019 7 442 451 31047901
Dennis, J. M., Shields, B. M., Henley, W. E., Jones, A. G. & Hattersley, A. T. Disease progression and treatment response in data-driven subgroups of type 2 diabetes compared with models based on simple clinical features: an analysis using clinical trial data. Lancet Diabetes Endocrinol. 7, 442–451 (2019).31047901
11. Nam S Chesla C Stotts NA Kroon L Janson SL Barriers to diabetes management: patient and provider factors Diabetes Res. Clin. Pr. 2011 93 1 9
Nam, S., Chesla, C., Stotts, N. A., Kroon, L. & Janson, S. L. Barriers to diabetes management: patient and provider factors. Diabetes Res. Clin. Pr. 93, 1–9 (2011).
12. Shah BR Hux JE Laupacis A Zinman B van Walraven C Clinical inertia in response to inadequate glycemic control: do specialists differ from primary care physicians? Diabetes Care 2005 28 600 606 15735195
Shah, B. R., Hux, J. E., Laupacis, A., Zinman, B. & van Walraven, C. Clinical inertia in response to inadequate glycemic control: do specialists differ from primary care physicians? Diabetes Care 28, 600–606 (2005).15735195
13. Zafar A Stone MA Davies MJ Khunti K Acknowledging and allocating responsibility for clinical inertia in the management of Type 2 diabetes in primary care: a qualitative study Diabet. Med. 2015 32 407 413 25251768
Zafar, A., Stone, M. A., Davies, M. J. & Khunti, K. Acknowledging and allocating responsibility for clinical inertia in the management of Type 2 diabetes in primary care: a qualitative study. Diabet. Med. 32, 407–413 (2015).25251768
14. Rushforth B McCrorie C Glidewell L Midgley E Foy R Barriers to effective management of type 2 diabetes in primary care: qualitative systematic review Br. J. Gen. Pr. 2016 66 e114 e127
Rushforth, B., McCrorie, C., Glidewell, L., Midgley, E. & Foy, R. Barriers to effective management of type 2 diabetes in primary care: qualitative systematic review. Br. J. Gen. Pr. 66, e114–e127 (2016).
15. Khunti K Wolden ML Thorsted BL Andersen M Davies MJ Clinical inertia in people with type 2 diabetes: a retrospective cohort study of more than 80,000 people Diabetes Care 2013 36 3411 3417 23877982
Khunti, K., Wolden, M. L., Thorsted, B. L., Andersen, M. & Davies, M. J. Clinical inertia in people with type 2 diabetes: a retrospective cohort study of more than 80,000 people. Diabetes Care 36, 3411–3417 (2013).23877982
16. Blonde L Gaps and barriers in the control of blood glucose in people with type 2 diabetes Diab Vasc. Dis. Res. 2017 14 172 183 28467203
Blonde, L. et al. Gaps and barriers in the control of blood glucose in people with type 2 diabetes. Diab Vasc. Dis. Res. 14, 172–183 (2017).28467203
17. Bertsimas D Kallus N Weinstein AM Zhuo YD Personalized diabetes management using electronic medical records Diabetes Care 2017 40 210 217 27920019
Bertsimas, D., Kallus, N., Weinstein, A. M. & Zhuo, Y. D. Personalized diabetes management using electronic medical records. Diabetes Care 40, 210–217 (2017).27920019
18. Sun X Effective treatment recommendations for type 2 diabetes management using reinforcement learning: treatment recommendation model development and validation J. Med. Internet Res. 2021 23 e27858 34292166
Sun, X. et al. Effective treatment recommendations for type 2 diabetes management using reinforcement learning: treatment recommendation model development and validation. J. Med. Internet Res. 23, e27858 (2021).34292166
19. Oh S-H Lee SJ Noh J Mo J Optimal treatment recommendations for diabetes patients using the Markov decision process along with the South Korean electronic health records Sci. Rep. 2021 11 6920 33767324
Oh, S.-H., Lee, S. J., Noh, J. & Mo, J. Optimal treatment recommendations for diabetes patients using the Markov decision process along with the South Korean electronic health records. Sci. Rep. 11, 6920 (2021).33767324
20. Zheng H Ryzhov IO Xie W Zhong J Personalized multimorbidity management for patients with type 2 diabetes using reinforcement learning of electronic health records Drugs 2021 81 471 482 33570745
Zheng, H., Ryzhov, I. O., Xie, W. & Zhong, J. Personalized multimorbidity management for patients with type 2 diabetes using reinforcement learning of electronic health records. Drugs 81, 471–482 (2021).33570745
21. Nambiar, M. et al. Deep offline reinforcement learning for real-world treatment optimization applications. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4673–4684 (Association for Computing Machinery, 2023).
22. Scirica BM Heart failure, saxagliptin, and diabetes mellitus: observations from the SAVOR-TIMI 53 randomized trial Circulation 2014 130 1579 1588 25189213
Scirica, B. M. et al. Heart failure, saxagliptin, and diabetes mellitus: observations from the SAVOR-TIMI 53 randomized trial. Circulation 130, 1579–1588 (2014).25189213
23. McGuire DK Linagliptin effects on heart failure and related outcomes in individuals with type 2 diabetes mellitus at high cardiovascular and renal risk in CARMELINA Circulation 2019 139 351 361 30586723
McGuire, D. K. et al. Linagliptin effects on heart failure and related outcomes in individuals with type 2 diabetes mellitus at high cardiovascular and renal risk in CARMELINA. Circulation 139, 351–361 (2019).30586723
24. Loke YK Kwok CS Singh S Comparative cardiovascular effects of thiazolidinediones: systematic review and meta-analysis of observational studies BMJ 2011 342 d1309 21415101
Loke, Y. K., Kwok, C. S. & Singh, S. Comparative cardiovascular effects of thiazolidinediones: systematic review and meta-analysis of observational studies. BMJ 342, d1309 (2011).21415101
25. Roberts DM Sevastos J Carland JE Stocker SL Lea-Henry TN Clinical pharmacokinetics in kidney disease: application to rational design of dosing regimens Clin. J. Am. Soc. Nephrol. 2018 13 1254 1263 30042221
Roberts, D. M., Sevastos, J., Carland, J. E., Stocker, S. L. & Lea-Henry, T. N. Clinical pharmacokinetics in kidney disease: application to rational design of dosing regimens. Clin. J. Am. Soc. Nephrol. 13, 1254–1263 (2018).30042221
26. Kanto K Effects of dosage and dosing frequency on the efficacy and safety of high-dose metformin in Japanese patients with type 2 diabetes mellitus J. Diabetes Investig. 2017 9 587 593 28963752
Kanto, K. et al. Effects of dosage and dosing frequency on the efficacy and safety of high-dose metformin in Japanese patients with type 2 diabetes mellitus. J. Diabetes Investig. 9, 587–593 (2017).28963752
27. Chubb B Once-daily oral semaglutide versus injectable GLP-1 RAs in people with type 2 diabetes inadequately controlled on basal insulin: systematic review and network meta-analysis Diabetes Ther. 2021 12 1325 1339 33723769
Chubb, B. et al. Once-daily oral semaglutide versus injectable GLP-1 RAs in people with type 2 diabetes inadequately controlled on basal insulin: systematic review and network meta-analysis. Diabetes Ther. 12, 1325–1339 (2021).33723769
28. Simpson SH Majumdar SR Tsuyuki RT Eurich DT Johnson JA Dose-response relation between sulfonylurea drugs and mortality in type 2 diabetes mellitus: a population-based cohort study CMAJ 2006 174 169 174 16415461
Simpson, S. H., Majumdar, S. R., Tsuyuki, R. T., Eurich, D. T. & Johnson, J. A. Dose-response relation between sulfonylurea drugs and mortality in type 2 diabetes mellitus: a population-based cohort study. CMAJ 174, 169–174 (2006).16415461
29. Lim, D. Y. Z., Chia, S. Y., Abdul Kadir, H., Mohamed Salim, N. N. & Bee, Y. M. Establishment of the SingHealth Diabetes Registry. Clin. Epidemiol. 13, 215–223 (2021).
30. MIMS Singapore. ‘antidiabetic agents’ Drug Search. https://www.mims.com/singapore/drug/search?q=antidiabetic+agents&code=11b (2021).
31. Saheb Kashaf M McGill ET Berger ZD Shared decision-making and outcomes in type 2 diabetes: a systematic review and meta-analysis Patient Educ. Couns. 2017 100 2159 2171 28693922
Saheb Kashaf, M., McGill, E. T. & Berger, Z. D. Shared decision-making and outcomes in type 2 diabetes: a systematic review and meta-analysis. Patient Educ. Couns. 100, 2159–2171 (2017).28693922
32. Tamhane S Rodriguez-Gutierrez R Hargraves I Montori VM Shared decision-making in diabetes care Curr. Diab Rep. 2015 15 112 26458383
Tamhane, S., Rodriguez-Gutierrez, R., Hargraves, I. & Montori, V. M. Shared decision-making in diabetes care. Curr. Diab Rep. 15, 112 (2015).26458383
33. SPRINT Research Group A randomized trial of intensive versus standard blood-pressure control N. Engl. J. Med. 2015 373 2103 2116 26551272
SPRINT Research Group et al. A randomized trial of intensive versus standard blood-pressure control. N. Engl. J. Med. 373, 2103–2116 (2015).26551272
34. Hong S-J Treat-to-target or high-intensity statin in patients with coronary artery disease: a randomized clinical trial JAMA 2023 329 1078 1087 36877807
Hong, S.-J. et al. Treat-to-target or high-intensity statin in patients with coronary artery disease: a randomized clinical trial. JAMA 329, 1078–1087 (2023).36877807
35. Raghavan S Matlock D Diabetes mellitus treatment deintensification: when well-controlled diabetes mellitus becomes overcontrolled Circ: Cardiovasc. Qual. Outcomes 2017 10 e003706 28416532
Raghavan, S. & Matlock, D. Diabetes mellitus treatment deintensification: when well-controlled diabetes mellitus becomes overcontrolled. Circ: Cardiovasc. Qual. Outcomes 10, e003706 (2017).28416532
36. Arnab, R. Chapter 7 - Stratified sampling. in Survey Sampling Theory and Applications (ed. Arnab, R.) 213–256 10.1016/B978-0-12-811848-1.00007-8 (Academic Press, 2017).
37. Subramanian S Hirsch IB Personalized diabetes management: moving from algorithmic to individualized therapy Diabetes Spectr. 2014 27 87 91 26246764
Subramanian, S. & Hirsch, I. B. Personalized diabetes management: moving from algorithmic to individualized therapy. Diabetes Spectr. 27, 87–91 (2014).26246764
38. Agency for Care Effectiveness, Ministry of Health, Republic of Singapore. Appropriate care guide (ACG): oral glucose-lowering agents in type 2 diabetes mellitus—an update. https://www.ace-hta.gov.sg/healthcare-professionals/ace-clinical-guidances-(acgs)/details/t2dm-personalising-medications (2017).
39. Tibshirani R Regression shrinkage and selection via the Lasso J. R. Stat. Soc. Ser. B 1996 58 267 288
Tibshirani, R. Regression shrinkage and selection via the Lasso. J. R. Stat. Soc. Ser. B 58, 267–288 (1996).
40. Chen, T. & Guestrin, C. XGBoost: a scalable tree boosting system. in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785–794 (Association for Computing Machinery, 2016).
41. Koomen JV Stevens J Monster-Simons MH Heerspink HJL Mol PGM A review of the dose justification of phase 3 trials to regulatory authorities for drugs intended for the treatment of type 2 diabetes in Europe Front. Pharmacol. 2021 12 626766 33995023
Koomen, J. V., Stevens, J., Monster-Simons, M. H., Heerspink, H. J. L. & Mol, P. G. M. A review of the dose justification of phase 3 trials to regulatory authorities for drugs intended for the treatment of type 2 diabetes in Europe. Front. Pharmacol. 12, 626766 (2021).33995023
42. Hesterberg T Weighted average importance sampling and defensive mixture distributions Technometrics 1995 37 185 194
Hesterberg, T. Weighted average importance sampling and defensive mixture distributions. Technometrics 37, 185–194 (1995).
43. Komorowski M Celi LA Badawi O Gordon AC Faisal AA The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care Nat. Med. 2018 24 1716 1720 30349085
Komorowski, M., Celi, L. A., Badawi, O., Gordon, A. C. & Faisal, A. A. The Artificial Intelligence Clinician learns optimal treatment strategies for sepsis in intensive care. Nat. Med. 24, 1716–1720 (2018).30349085
44. Roggeveen L Transatlantic transferability of a new reinforcement learning model for optimizing haemodynamic treatment for critically ill patients with sepsis Artif. Intell. Med. 2021 112 102003 33581824
Roggeveen, L. et al. Transatlantic transferability of a new reinforcement learning model for optimizing haemodynamic treatment for critically ill patients with sepsis. Artif. Intell. Med. 112, 102003 (2021).33581824
45. Raghu, A., Komorowski, M., Celi, L.A., Szolovits, P. & Ghassemi, M. Continuous state-space models for optimal sepsis treatment: A deep reinforcement learning approach. In Proceedings of the 2nd Machine Learning for Healthcare Conference. Vol. 68 (eds Doshi-Velez, F. et al) 147–163 (PMLR, 2017).
46. Moore, D. S. The Basic Practice of Statistics (W.H. Freeman, 2004).
47. Pope, C. & Mays, N. Qualitative Research in Health Care, 4th Edition (Wiley-Blackwelltes, 2020).
