
==== Front
JCO Clin Cancer Inform
JCO Clin Cancer Inform
cci
CCI
JCO Clinical Cancer Informatics
2473-4276
Wolters Kluwer Health

39008783
CCI.24.00078
10.1200/CCI.24.00078
00075
ORIGINAL REPORTS
Drug Therapy
External Validation and Update of the Risk Prediction Model for Denosumab-Induced Hypocalcemia Developed From a Hospital-Based Administrative Database
https://orcid.org/0000-0002-6942-590X
Ikegami Keisuke BS 1
https://orcid.org/0000-0001-5706-613X
Imai Shungo PhD 1
Yasumuro Osamu PhD 2
https://orcid.org/0000-0003-3846-0435
Tsuchiya Masami PhD 1 3
Henmi Naomi BS 3
Suzuki Mariko BS 3
Hayashi Katsuhisa PhD 3
Miura Chisato BS 3
Abe Haruna BS 3
https://orcid.org/0000-0002-4572-1333
Kizaki Hayato MSc 1
https://orcid.org/0000-0002-1897-9689
Funakoshi Ryohkan PhD 2
Sato Yasunori PhD 4
https://orcid.org/0000-0002-4596-5418
Hori Satoko PhD 1
1 Keio University Faculty of Pharmacy/Graduate School of Pharmaceutical Sciences, Tokyo, Japan
2 Department of Pharmacy, Kameda General Hospital, Chiba, Japan
3 Department of Pharmacy, Miyagi Cancer Center, Miyagi, Japan
4 Department of Biostatistics, Keio University School of Medicine, Tokyo, Japan
Satoko Hori, PhD; e-mail: satokoh@keio.jp.
2024
15 7 2024
15 7 2024
8 e24000783 4 2024
23 4 2024
10 5 2024
© 2024 by American Society of Clinical Oncology
2024
American Society of Clinical Oncology
https://creativecommons.org/licenses/by-nc-nd/4.0/ Creative Commons Attribution Non-Commercial No Derivatives 4.0 License: https://creativecommons.org/licenses/by-nc-nd/4.0/

PURPOSE

Denosumab is used to treat patients with bone metastasis from solid tumors, but sometimes causes severe hypocalcemia, so careful clinical management is important. This study aims to externally validate our previously developed risk prediction model for denosumab-induced hypocalcemia by using data from two facilities with different characteristics in Japan and to develop an updated model with improved performance and generalizability.

METHODS

In the external validation, retrospective data of Kameda General Hospital (KGH) and Miyagi Cancer Center (MCC) between June 2013 and June 2022 were used and receiver operating characteristic (ROC)-AUC was mainly evaluated. A scoring-based updated model was developed using the same data set from a hospital-based administrative database as previously employed. Selection of variables related to prediction of hypocalcemia was based on the results of external validation.

RESULTS

For the external validation, data from 235 KGH patients and 224 MCC patients were collected. ROC-AUC values in the original model were 0.879 and 0.774, respectively. The updated model consisting of clinical laboratory tests (calcium, albumin, and alkaline phosphatase) afforded similar ROC-AUC values in the two facilities (KGH, 0.837; MCC, 0.856).

CONCLUSION

We developed an updated risk prediction model for denosumab-induced hypocalcemia with small interfacility differences. Our results indicate the importance of using data from plural facilities with different characteristics in the external validation of generalized prediction models and may be generally relevant to the clinical application of risk prediction models. Our findings are expected to contribute to improved management of bone metastasis treatment.

New risk model for denosumab-induced hypocalcemia outperforms original, validated across different facilities.

OPEN-ACCESSTRUE
==== Body
pmcINTRODUCTION

Denosumab is an anti-receptor activator for nuclear factor-κB ligand monoclonal antibody that is administered to prevent skeletal-related events in patients with bone metastasis from solid tumors. However, denosumab-induced hypocalcemia is a well-known adverse event1 and can have severe outcomes,2-5 including tetany, seizures, and cardiac dysrhythmias.6 The incidence of hypocalcemia can exceed 30%,3-5,7 and severe cases (grade 3/4) account for approximately 10%.3 Correction of calcium level may lead to discontinuation of cancer treatment, and detection of hypocalcemia may be delayed since outpatient treatment is given every 4 weeks,8 so careful clinical management is critical. No predictive methods have been established to quantitatively predict hypocalcemia risk associated with bone-modifying agents, and current practice relies on preadministration laboratory testing and supplementation of calcium and/or vitamin D. Therefore, evaluation of key risk factors and development of clinically applicable risk prediction models for denosumab-induced hypocalcemia are important to support medical decision making in individual patients.9 In the development of these models, the use of medical big data is pivotal for generalizability, and models with high external validity can be obtained.10 Nevertheless, model performance for external data tends to be inferior to that for internal data,11 and therefore, external validation is necessary before a model is employed in clinical practice.12,13 However, in fact, only a few prediction models have been externally validated among those so far developed.13-15 In addition, previous external validation studies16-21 have used medical big data or data from a single facility. In the case of clinical prediction models, which are mainly developed by using patient demographics and baseline characteristics as predictors, the model performance may differ when they are applied in different facilities because of variations in factors such as therapeutic options or patient populations. It has also been reported that a single validation is insufficient to fully capture the model's performance.14 Therefore, it may be important to use data sets from multiple facilities with different characteristics for external validation. However, to our knowledge, there has been no report of external validation using this approach.

CONTEXT

Key Objective

We aimed to validate a previously built model for predicting the risk of denosumab-induced hypocalcemia before treatment initiation using clinical data from two facilities in Japan and to assess its generalizability.

Knowledge Generated

Validation showed interfacility differences in model performance, so we created an updated denosumab-induced hypocalcemia risk prediction model, which performed better than the original model. Validation targeting multiple facilities is important to improve generalizability.

Relevance (J.L. Warner)

Denosumab is widely used in the treatment of bone metastases, and carries the risk of hypocalcemia. This improved risk prediction model could find immediate clinical applicability although widespread generalizability remains to be fully established.*

*Relevance section written by JCO Clinical Cancer Informatics Editor-in-Chief Jeremy L. Warner, MD, MS, FAMIA, FASCO.

If the model performance is externally inadequate, the model should be updated. Furthermore, if there are substantial interfacility differences in performance, model updating is still required to improve generalizability. This can be done by deleting variables or adding new variables.22 We therefore hypothesized that if there were substantial interfacility differences in performance, it would be possible to develop an updated model with smaller differences in performance by extracting variables that might have contributed to the differences on the basis of the results of the external validation and excluding them from the model variables.

To test this idea, we focused on a risk prediction model for denosumab-induced hypocalcemia in patients with bone metastases from solid tumors that was developed in our previous study.23 Although our model (original model) developed from a hospital-based administrative database has clinically acceptable performance and small over-fitting, the model includes variables, such as diagnosis of breast and gastric cancer, which are expected to be related to the patient distribution in each facility.

Primarily, this study aimed to conduct an external validation of the original model using medical data from two facilities, which have different geographical characteristics and cancer specialties, and second to update the risk prediction original model for denosumab-induced hypocalcemia23 to achieve higher performance and generalizability (Fig 1).

FIG 1. Study diagram. KGH, Kameda General Hospital; MCC, Miyagi Cancer Center. aIkegami et al.23

METHODS

Study Design and Patients

In the external validation, retrospective data from electronic medical records of Kameda General Hospital (KGH; Chiba, Japan) and Miyagi Cancer Center (MCC; Miyagi, Japan) were used. We chose these facilities on the basis that KGH is a tertiary care hospital with 865 beds (noncancer patients are also eligible) and MCC is a designated cancer center with 383 beds. Thus, the two facilities have different patient population characteristics, though treatment protocols are basically common in the two because of the universal health insurance scheme in Japan. Patients with bone metastases who initially received denosumab 120 mg (once every 4 weeks) between June 2013 and June 2022 were included. Patients with a diagnosis of multiple myeloma or giant cell tumor of bone were excluded. We also excluded patients for whom calcium values pre-denosumab and post-denosumab (within 28 days) were not available and those who did not receive two tablets per day of natural vitamin D/calcium (DENOTAS chewable combination tablet: 305 mg of calcium, 200 IU of cholecalciferol, and 15 mg of magnesium per tablet) in accordance with the provisions of the package insert in Japan. After excluding patients with grade 1 hypocalcemia and patients whose baseline data of albumin or alkaline phosphatase (ALP) were missing, the remaining patients were divided into the hypocalcemia group and the nonhypocalcemia group.23

In the model update, patient data (derivation dataset) from the Medical Data Vision database used in the previous study23 was used to develop the model, and the same inclusion criteria were adopted.23

Data Collection

Patient background data were collected on the basis of factors identified in previous studies1,23-26 and clinical importance: sex, age, body weight, Eastern Cooperative Oncology Group performance status (ECOG-PS), clinical laboratory tests (calcium [mg/dL], albumin [g/dL], sodium [mEq/L], potassium [mEq/L], chlorine [mEq/L], phosphorus [mg/dL], serum creatinine [mg/dL], estimated glomerular filtration rate [eGFR; mL/min/1.73 m2], AST [U/L], ALT [U/L], ALP [U/L], total bilirubin [mg/dL]), disease data (osteoporosis, prostate, breast, gastric, or lung cancer), and drug treatment (denosumab, cholecalciferol/calcium supplement, proton pump inhibitor [PPI], zoledronic acid). The number of cases of hypocalcemia by grade was also investigated.

Age, body weight, ECOG-PS, and clinical laboratory test data at baseline were collected. Baseline was defined as the most recent day before the first dose of denosumab. Osteoporosis data were collected for diagnoses before the first denosumab dose. Cancer data were collected as primary cancer of bone metastasis. All diagnoses were collected from electronic medical record information. The method of ALP measurement was changed after 2020 in Japan. The conventional measurement method (Japan Society of Clinical Chemistry [JSCC]) gives values approximately three times higher than the changed method (International Federation of Clinical Chemistry [IFCC]). Therefore, in this study, we adapted the following equations for ALP conversion recommended by the JSCC.ALP (JSCC)=ALP(IFCC)×2.84 (Unit:U/L)

ALP (IFCC)=ALP(JSCC)×0.35 (Unit:U/L)

PPI combination was defined as continuous prescription for 28 days (or until the onset of hypocalcemia) after the first denosumab dose. Continuous administration was determined on the basis of the date of prescription and the number of administration days. If the total deviation was more than 3 days, administration was judged not to be combined.23 The definition of pretreatment with zoledronic acid was administration within 60 days before the first denosumab dose. The formula for calculating eGFR values was the same as in previous studies.23

Risk Prediction Model (original model)

The risk prediction model that we developed in the previous study23 was as follows:Probability=1 / (1+exp (− Risk score)),

Risk score=11.469+0.418 (Female) – 2.049 (Corrected Calcium) – 0.671 (Albumin)+1.276 (Log (ALP(JSCC))) – 1.745 (Osteoporosis) – 1.288 (Breast cancer)+0.675 (Gastric cancer)+0.725 (PPI combination) – 1.437 (Pretreatment with zoledronic acid)

Calcium, albumin, and ALP input values were used as continuous values. Other variables were input as categorical. The original model was developed with a main focus on model performance.

Definition of Hypocalcemia

The development of grade ≥2 hypocalcemia within 28 days after the first dose of denosumab was evaluated. Grade assessment was based on Common Terminology Criteria for Adverse Events version 5.0, and grade ≥2 hypocalcemia was defined as <8.0 mg/dL. Correction of calcium values for albumin levels followed the previous study.23

Statistical Analysis

This study basically followed the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis reporting guidelines.27

Continuous variables were represented as median (IQR) and categorical variables were shown as numbers (%). The Fisher exact test and Mann-Whitney U test were employed for categorical and continuous variables, respectively.

In the external validation of the original model, model performance was evaluated from three viewpoints. (1) Discriminant performance was evaluated in terms of sensitivity, specificity, positive predictive value, negative predictive value, and receiver operating characteristic (ROC)-AUC. The cutoff value for risk stratification was 8.28%.23 (2) Predictive performance was evaluated by obtaining a calibration slope. A loss-based calibration was used that provides graphical objectivity regarding model fit.28 (3) Clinical utility was finally evaluated by conducting decision curve analysis.29 In this study, net benefit of threshold <35% was evaluated by comparing the benefit of assuming every patient is at high or low risk with that of using the original model.

In the model update, the scoring system was selected in terms of clinical usefulness. Selection of variables was conducted considering the results of external validation of the original model, and several scoring models were developed. Both statistical and clinical aspects were considered when variable selection was necessary in model development. Clinical laboratory tests were categorized on the basis of the criterion values and on the cutoff values from univariate logistic regression analysis. The score was assigned by rounding the β coefficient to the nearest integer. The model fit was evaluated on the basis of akaike information criterion. Missing values were replaced by the multiple imputation method on the basis of the fully conditional specification algorithm, which does not rely on the assumption of multivariate normality and specifies the multivariate imputation model per variable.30 Regardless of the missing rate, which was estimated to be <10% for the relevant variables, all model parameters and outcome were used for imputation.31 A total of 200 imputed data sets were created. The validation of updated models was conducted by using the same data sets of the two facilities, and models were evaluated in terms of discriminant performance. Sensitivity and specificity were determined by defining the cutoff value that maximizes the Youden index (sensitivity + specificity – 1). The statistically and clinically optimal model was selected as the updated model. Over-fitting for AUC was assessed by internal validation using 200 bootstrap samples from the derivation data set. The choices of internal validation method and evaluation item were based on the data set size and our emphasis on discriminant performance. Cochran-Armitage trend analysis was also performed to evaluate whether patients predicted as high risk by the original model could be further stratified by the updated model. Statistical analyses were performed with SAS software version 9.4 for Windows (SAS Institute, Cary, NC). Statistical significance was set at P < .05 (two-tailed).

Ethics Approval

This study was approved by the ethics committee for research involving human patients of the Keio University Faculty of Pharmacy (approval no. 230407-3).

Informed Consent

Informed consent was not applicable because of the retrospective analysis of the anonymized data.

RESULTS

Patient Characteristics at the Two Facilities

A total of 235 patients of KGH and 224 patients of MCC were included. The extraction flow of included patients is shown in Data Supplement (Fig S1), and patient characteristics are summarized in Table 1. Patient backgrounds, especially the distribution of cancer types, differed between the two facilities. The variables of sex, diagnosis of osteoporosis, and pretreatment with zoledronic acid also differed substantially in distribution from the derivation data set.23

TABLE 1. Summary of Clinical Characteristics

Variable	Kameda General Hospital (n = 235)	Miyagi Cancer Center (n = 224)	
Nonhypocalcemia (n = 211)	Hypocalcemia (n = 24)	P	Nonhypocalcemia (n = 202)	Hypocalcemia (n = 22)	P	
Female, No. (%)	116 (55.0)	9 (37.5)	.131	80 (39.6)	10 (45.5)	.650	
Age, years, median (IQR)	70 (60-75)	66 (55-71)	.054	68 (61-74)	63 (58-70)	.044	
Body weight, kg, median (IQR)	55.9 (46.5-64.7)	54.7 (46.1-65.5)	.840	NA	NA	NA	
ECOG-PS (0, 1, 2, 3, 4)	NA	NA	NA	19, 89, 36, 54, 5	0, 10, 2, 10, 0	NA	
Clinical laboratory tests, median (IQR)							
 Corrected calcium (8.8-10.1 mg/dL)	9.5 (9.2-9.8)	9.1 (8.8-9.4)	<.001	9.6 (9.3-10.0)	9.2 (8.8-9.7)	.005	
 Albumin (4.1-5.1 g/dL)	3.9 (3.4-4.2)	3.7 (3.4-4.0)	.125	3.6 (3.2-4.0)	3.7 (3.1-4.0)	.921	
 Sodium (138-145 mEq/L)	140 (138-142)	139 (137-141)	.034	140 (138-141)	139 (136-142)	.525	
 Potassium (3.6-4.8 mEq/L)	4.2 (3.9-4.6)	4.2 (4.0-4.3)	.591	4.3 (4.0-4.6)	4.1 (3.8-4.5)	.153	
 Chlorine (101-108 mEq/L)	103 (101-106)	103 (101-105)	.589	103 (101-105)	104 (102-106)	.459	
 Phosphorus (2.7-4.6 mg/dL)	3.4 (3.1-3.8)	3.4 (3.0-4.0)	.909	3.4 (2.9-3.9)	3.5 (2.6-4.0)	1.000	
 Serum creatinine (male: 0.65-1.07 mg/dL, female: 0.46-0.79 mg/dL)							
  Male	0.80 (0.69-0.99)	0.82 (0.67-1.07)	.632	0.80 (0.69-0.94)	0.66 (0.57-0.96)	.162	
  Female	0.61 (0.51-0.70)	0.63 (0.58-0.68)	.576	0.65 (0.54-0.73)	0.59 (0.53-0.71)	.594	
 eGFR (≥60 mL/min/1.73 m2)	74.1 (61.4-91.8)	75.3 (66.4-88.0)	.999	71.5 (60.4-85.9)	81.5 (59.9-99.0)	.186	
 AST (13-30 U/L)	22 (18-29)	22 (17-30)	.962	23 (18-33)	27 (24-37)	.016	
 ALT (male: 10-42 U/L, female: 7-23 U/L)							
  Male	19 (15-28)	17 (15-32)	.893	20 (13-33)	15 (10-19)	.072	
  Female	15 (12-27)	24 (11-28)	.419	15 (11-26)	17 (12-26)	.842	
 ALP (JSCC, 106-322 U/L)	275 (213-387)	794 (363-1,568)	<.001	327 (242-487)	1,065 (600-1,625)	<.001	
 Total bilirubin (0.4-1.5 mg/dL)	0.50 (0.30-0.60)	0.45 (0.40-0.60)	.917	0.43 (0.33-0.57)	0.41 (0.34-0.50)	.642	
Osteoporosis, No.(%)	34 (16.1)	4 (16.7)	1.000	11 (5.4)	2 (9.1)	.372	
Primary cancer, No.(%)							
 Prostate cancer	34 (16.1)	3 (12.5)	.776	19 (9.4)	4 (18.2)	.256	
 Breast cancer	76 (36.0)	2 (8.3)	.005	29 (14.4)	1 (4.5)	.324	
 Gastric cancer	10 (4.7)	4 (16.7)	.042	4 (2.0)	7 (31.8)	<.001	
 Lung cancer	58 (27.5)	4 (16.7)	.332	96 (47.5)	4 (18.2)	.012	
 Others	33 (15.6)	11 (45.8)	NA	54 (26.7)	6 (27.3)	NA	
PPI combination,a No.(%)	41 (19.4)	8 (33.3)	.118	55 (27.2)	8 (36.4)	.454	
Pretreatment with zoledronic acid,b No.(%)	10 (4.7)	2 (8.3)	.352	1 (0.5)	3 (13.6)	.003	
Hypocalcemia (grade 2/3/4)c	NA	23/1/0	NA	NA	17/4/1	NA	
NOTE. The ranges of clinical laboratory tests are based on shared reference range criteria in Japan (ver. 2022).

Abbreviations: ALP, alkaline phosphatase; ECOG-PS, Eastern Cooperative Oncology Group performance status; eGFR, estimated glomerular filtration rate; JSCC, Japan Society of Clinical Chemistry; NA, not available; PPI, proton pump inhibitor.

a Continuous 28 days of first denosumab dose (or until the day of hypocalcemia).

b Within 60 days before first denosumab dose.

c Hypocalcemia was defined based on the Common Terminology Criteria for Adverse Events version 5.0 as follows: grade 2, corrected calcium <8.0-7.0 mg/dL; grade 3, <7.0-6.0 mg/dL; grade 4, <6.0 mg/dL.

External Validation of the Original Model

ROC-AUC for discriminant performance was 0.879 for KGH and 0.774 for MCC (Fig 2). Specificity and positive predictive value were also relatively low for the population of MCC (Data Supplement, Tables S1 and S2). Predictive performance, especially around the cutoff value was lower (overestimation), and clinical utility was lost below the cutoff value of approximately 5% for the population of MCC (Data Supplement, Figs S2 and S3).

FIG 2. External validation of the original model (discriminant performance).

Model Update and Validation

We hypothesized that one of the factors contributing to the difference in model performance between facilities was the distribution of cancer types, and we developed nine updated models on the basis of the following strategies to minimize the difference. Strategy 1: Without splitting the data set used to develop the original model,23 develop models using only clinical laboratory test values. Strategy 2: Split the data set into two, one of patients with a diagnosis of gastric cancer and the other of patients without it and develop models for each data set. Strategy 3: Use the same approach as for strategy 2 with regard to breast cancer. Diagnosis of gastric cancer and breast cancer were factors included in the original model.

After considering three strategies, a scoring model consisting of three variables, calcium, albumin, and ALP (strategy 1), was adopted as the updated model (refer to the Data Supplement, Tables S3-S15; Table 2). On the basis of the updated model, a ROC curve from a logistic regression model using the sum of the scores for the derivation data set was obtained (Data Supplement, Fig S4). Classification ability of positive cases was improved (Data Supplement, Table S16). Bootstrap internal validation showed over-fitting of the model was slight (0.009 for AUC). ROC curves to the facility data sets (Fig 3) revealed that the interfacility difference in discriminant performance had become smaller. The relationship between scores and probability of hypocalcemia is shown in Figure 4. It was also found that patients predicted as high risk by the original model could be further stratified by the updated model (Data Supplement, Fig S5).

TABLE 2. Development of the Scoring Model 1

Variable	β Coefficient	P	Score	
Corrected calcium, mg/dL				
 <8.8	2.532	<.001	3	
 8.8 to <9.2	1.584	<.001	2	
 ≥9.2	Ref	Ref	0	
Albumin, g/dL				
 <3.5	0.748	.014	1	
 3.5 to <4.1	0.088	.772	0	
 ≥4.1	Ref	Ref	0	
ALP (IFCC), U/L				
 <113	Ref	Ref	0	
 113 to <195	0.679	.051	1	
 ≥195	2.469	<.001	2	
NOTE. Among the variables in the original model, we identified calcium, albumin and ALP as factors that are independent of cancer type. The score was assigned by rounding the β coefficient to the nearest integer.

Abbreviations: ALP, alkaline phosphatase; IFCC, International Federation of Clinical Chemistry; Ref, reference.

FIG 3. Validation of the updated model (discriminant performance).

FIG 4. The relationship between total scores and probability of hypocalcemia (≥grade 2). The solid line and the band indicate estimated value and 95% CI, respectively. Total score can be obtained from the sum of calcium, albumin, and ALP scores. The number of patients with hypocalcemia per score in the derivation data is also shown. ALP, alkaline phosphatase.

DISCUSSION

In this study, the original risk prediction model for denosumab-induced hypocalcemia that we previously developed from a hospital-based administrative database was externally validated using data from two facilities. However, differences in model performance were observed between the facilities. To solve this issue, we developed nine updated models and examined the differences in their performance. Finally, we successfully developed a scoring model that can be easily used in clinical practice and has small interfacility differences in performance.

The effort to reduce differences in model performance between facilities is crucial for ensuring the consistency and general applicability of a predictive model. For example, in the present case, there were large interfacility differences in the distributions of sex, diagnosis of osteoporosis, and pretreatment with zoledronic acid. Sex is a cancer type-dependent factor.32 Osteoporosis diagnosis may depend on the therapeutic options at facilities. There is a report that pretreatment with zoledronic acid could reduce the risk of hypocalcemia,26 but our previous study did not show a statistically significant difference.23 In addition, distributions of patients in both facilities in this study revealed no trend for pretreatment with zoledronic acid to reduce the risk of hypocalcemia. Therefore, we decided not to use these factors. We also considered PPI combination should not be included as a factor in the model because it is a future event from the prediction point. An analysis of patients with osteoporosis showed that creatinine levels were a predictor of denosumab-induced hypocalcemia.33 However, recent studies, including ours, have found that renal dysfunction is not a risk factor for hypocalcemia in patients with bone metastasis who are receiving calcium/vitamin D supplement.23,34 This implies there are more important factors than renal function that should be monitored under these conditions. Furthermore, there were differences in model performance among facilities. One of the causes could be the distribution of cancer types. The original model includes factors of gastric and breast cancer, but not lung cancer. Lung cancer is characterized by an osteolytic bone metastatic pattern,35,36 and patients with bone metastases from lung cancer at both facilities tended to be more frequent in the nonhypocalcemia group. Therefore, lung cancer could be a factor influencing blood calcium levels, and this may be one reason why the original model, which does not contain this factor, showed relatively poor performance for patients in MCC, where a large population of patients had lung cancer.

Considering that lung cancer affects calcium levels, the influence of different cancer types can be considered to be integrated in laboratory values such as calcium values. Therefore, from this perspective it seems reasonable to adopt strategy 1. We developed a scoring model focusing on clinical laboratory tests, which are objective data. Although in general, the categorization of continuous values inevitably reduces performance,37 the discriminant performance of the updated model was still acceptable for clinical application. Although the model performance for patients in KGH was slightly lower than that of the original model, we succeeded in obtaining a model with a ROC-AUC >0.8 for both facilities and also in reducing the interfacility difference. The scoring model 1 with a cutoff value of 2 provided a sensitivity of 0.8 or higher at each facility. Consequently, variable exclusion did not negatively affect the model performance and contributed to reducing the differences between facilities. Furthermore, because of the use of the multiple imputation method, the bias in the model estimates is considered small.30 Such a simple scoring model with high performance may be useful in clinical practice; it provides a simple risk assessment of grade ≥2 hypocalcemia for patients receiving the first dose of denosumab by using only three blood test values, thus offering the possibility of clinical care according to the level of risk. This allows for safe treatment of high-risk patients through close laboratory monitoring and frequent visits. This updated model could further stratify patients predicted to be high risk by the original model, and this represents a further advantage of the updated model.

As we focused on reducing interfacility differences in model performance, some potential predictors related to denosumab-induced hypocalcemia were excluded. However, it should be noted that we were unable to evaluate some factors, such as existing vitamin D deficiency, drugs such as steroids and diuretics, or altered intestinal calcium absorption related to chemotherapy-induced severe diarrhea. We also could not evaluate whether all patients with osteoporosis underwent specific examinations, such as a bone density test. It is important to note that the selection of variables for the updated model was based on the patient backgrounds of only two facilities. Therefore, although this study highlights the importance of using plural data sets to validate the model, additional external validation of the updated model using data from other facilities is still needed. Finally, the study population was taken from only two Japanese facilities, and the fact that the conditions for supplementation and the health care system are the same could be an extrinsic factor that affects on the model's generalizability. However, the use of the Japanese facility data set made it possible to include calcium/vitamin D supplement prescribing data, which we think is valuable.

Our results suggest the importance of evaluating model fit per facility. Therefore, we anticipate that the model's robustness could be further improved by conducting validation at multiple facilities with different characteristics. The availability of a highly generalizable model to predict hypocalcemia in patients treated with denosumab relatively accurately across a wide range of facilities with different characteristics would contribute to safe and efficient personalized patient care. We are planning to test this model in a real clinical setting in the near future.

In conclusion, on the basis of the results of the external validation of our generalized prediction model previously developed from a hospital-based administrative database, we identified factors that may cause differences in model performance among facilities and successfully developed an updated risk prediction model for denosumab-induced hypocalcemia with small interfacility differences. Our results indicate the importance of using data from plural facilities with different characteristics in the external validation of generalized prediction models and may be generally relevant to the clinical application of risk prediction models developed from medical big data. We anticipate that this simple, high-performance scoring model will contribute to improved management of bone metastasis treatment.

PRIOR PRESENTATION

SUPPORT

DATA SHARING STATEMENT

Data used for developing the original and updated model are from the Medical Data Vision database, which are anonymized and commercially available data. The data that support the findings of this study are available from the corresponding author on reasonable request.

AUTHOR CONTRIBUTIONS

Conception and design: Keisuke Ikegami, Shungo Imai, Osamu Yasumuro, Katsuhisa Hayashi, Hayato Kizaki, Ryohkan Funakoshi, Satoko Hori

Financial support: Keisuke Ikegami, Satoko Hori

Administrative support: Ryohkan Funakoshi

Provision of study materials or patients: Masami Tsuchiya, Katsuhisa Hayashi

Collection and assembly of data: Keisuke Ikegami, Osamu Yasumuro, Masami Tsuchiya, Naomi Henmi, Mariko Suzuki, Katsuhisa Hayashi, Chisato Miura, Haruna Abe, Satoko Hori

Data analysis and interpretation: Keisuke Ikegami, Osamu Yasumuro, Masami Tsuchiya, Ryohkan Funakoshi, Yasunori Sato, Satoko Hori

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).

Presented in part at the Federation of Asian Pharmaceutical Associations Congress, Taipei, Taiwan, October 24-28, 2023.

Supported by JST SPRING grant number JPMJSP2123, JST CREST grant number JPMJCR22N1 and JSPS KAKENHI grant number JP 24KJ1941, Japan.

Shungo Imai

Speakers' Bureau: Takeda

Yasunori Sato

Honoraria: Lilly Japan

Consulting or Advisory Role: Mochida Pharmaceutical Co Ltd

Satoko Hori

Employment: Johnson & Johnson/Janssen (I)

No other potential conflicts of interest were reported.
==== Refs
REFERENCES

1. Body J-J , Bone HG , de Boer RH , et al : Hypocalcaemia in patients with metastatic bone disease treated with denosumab. Eur J Cancer 51 :1812-1821, 2015 26093811
2. Dadana S , Gundepalli S , Kondapalli A : Severe refractory hypocalcemia caused by denosumab. Cureus 15 :e39866, 2023 37404446
3. Yerram P , Kansagra S , Abdelghany O : Incidence of hypocalcemia in patients receiving denosumab for prevention of skeletal-related events in bone metastasis. J Oncol Pharm Pract 23 :179-184, 2017 26830549
4. Manzaneque A , Chaguaceda C , Mensa M , et al : Use and safety of denosumab in cancer patients. Int J Clin Pharm 39 :522-526, 2017 28382583
5. Lechner B , DeAngelis C , Jamal N , et al : The effects of denosumab on calcium profiles in advanced cancer patients with bone metastases. Support Care Cancer 22 :1765-1771, 2014 24515277
6. Cooper MS , Gittoes NJL : Diagnosis and management of hypocalcaemia. BMJ 336 :1298-1302, 2008 18535072
7. von Moos R , Costa L , Gonzalez-Suarez E , et al : Management of bone health in solid tumours: From bisphosphonates to a monoclonal antibody. Cancer Treat Rev 76 :57-67, 2019 31136850
8. Milat F , Goh S , Gani LU , et al : Prolonged hypocalcemia following denosumab therapy in metastatic hormone refractory prostate cancer. Bone 55 :305-308, 2013 23685544
9. Wynants L , Van Calster B , Collins GS , et al : Prediction models for diagnosis and prognosis of covid-19: Systematic review and critical appraisal. BMJ 369 :m1328, 2020 32265220
10. Mehta N , Pandit A : Concurrence of big data analytics and healthcare: A systematic review. Int J Med Inform 114 :57-65, 2018 29673604
11. Siontis GCM , Tzoulaki I , Castaldi PJ , et al : External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination. J Clin Epidemiol 68 :25-34, 2015 25441703
12. Moons KGM , Kengne AP , Grobbee DE , et al : Risk prediction models: II. External validation, model updating, and impact assessment. Heart 98 :691-698, 2012 22397946
13. Ramspek CL , Jager KJ , Dekker FW , et al : External validation of prognostic models: What, why, how, when and where? Clin Kidney J 14 :49-58, 2021 33564405
14. Wessler BS , Nelson J , Park JG , et al : External validations of cardiovascular clinical prediction models: A large-scale review of the literature. Circ Cardiovasc Qual Outcomes 14 :E007858, 2021 34340529
15. Bellou V , Belbasis L , Konstantinidis AK , et al : Prognostic models for outcome prediction in patients with chronic obstructive pulmonary disease: Systematic review and critical appraisal. BMJ 367 :l5358, 2019 31585960
16. Ferguson T , Ravani P , Sood MM , et al : Development and external validation of a machine learning model for progression of CKD. Kidney Int Rep 7 :1772-1781, 2022 35967110
17. Archer L , Koshiaris C , Lay-Flurrie S , et al : Development and external validation of a risk prediction model for falls in patients with an indication for antihypertensive treatment: Retrospective cohort study. BMJ 379 :e070918, 2022 36347531
18. Eslami MH , Saadeddin Z , Farber A , et al : External validation of the Vascular Study Group of New England carotid endarterectomy risk predictive model using an independent U.S. national surgical database. J Vasc Surg 71 :1954-1963, 2020 31676184
19. Bradley J , Schelbert EB , Bonnett LJ , et al : Predicting hospitalisation for heart failure and death in patients with, or at risk of, heart failure before first hospitalisation: A retrospective model development and external validation study. Lancet Digit Health 4 :e445-e454, 2022 35562273
20. Liu Y , Liang Z , Yuan S , et al : Development and validation of a prediction model for tuberculous pleural effusion: A large cohort study and external validation. Respir Res 23 :134, 2022 35624515
21. Leto E , Lofaro D , Lucca LF , et al : External validation and calibration of the DecaPreT prediction model for decannulation in patients with acquired brain injury. Brain Sci 11 :799, 2021 34204352
22. Binuya MAE , Engelhardt EG , Schats W , et al : Methodological guidance for the evaluation and updating of clinical prediction models: A systematic review. BMC Med Res Methodol 22 :316, 2022 36510134
23. Ikegami K , Hashiguchi M , Kizaki H , et al : Development of risk prediction model for grade 2 or higher hypocalcemia in patients with bone metastasis treated with denosumab plus cholecalciferol (vitamin D3)/calcium supplement. J Clin Pharmacol 62 :1151-1159, 2022 35383950
24. Kinoshita Y , Arai M , Ito N , et al : High serum ALP level is associated with increased risk of denosumab-related hypocalcemia in patients with bone metastases from solid tumors. Endocr J 63 :479-484, 2016 26860123
25. Huynh ALH , Baker ST , Stewardson AJ , et al : Denosumab-associated hypocalcaemia: Incidence, severity and patient characteristics in a tertiary hospital setting. Pharmacoepidemiol Drug Saf 25 :1274-1278, 2016 27255807
26. Okada N , Kawazoe K , Teraoka K , et al : Identification of the risk factors associated with hypocalcemia induced by denosumab. Biol Pharm Bull 36 :1622-1626, 2013 23934346
27. Moons KGM , Altman DG , Reitsma JB , et al : Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): Explanation and elaboration. Ann Intern Med 162 :W1-W73, 2015 25560730
28. Austin PC , Steyerberg EW : Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 33 :517-535, 2014 24002997
29. Vickers AJ , Elkin EB : Decision curve analysis: A novel method for evaluating prediction models. Med Decis Making 26 :565-574, 2006 17099194
30. Liu Y , De A : Multiple imputation by fully conditional specification for dealing with missing data in a large epidemiologic study. Int J Stat Med Res 4 :287-295, 2015 27429686
31. Moons KGM , Donders RART , Stijnen T , et al : Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol 59 :1092-1101, 2006 16980150
32. Mattiuzzi C , Lippi G : Current cancer epidemiology. J Epidemiol Glob Health 9 :217-222, 2019 31854162
33. Tsvetov G , Amitai O , Shochat T , et al : Denosumab-induced hypocalcemia in patients with osteoporosis: Can you know who will get low? Osteoporos Int 31 :655-665, 2020 31838550
34. Saito Y , Takekuma Y , Komatsu Y , et al : Risk analysis of denosumab-induced hypocalcemia in bone metastasis treatment: Renal dysfunction is not a risk factor for its incidence in a strict denosumab administration management system with calcium/vitamin D supplementation. Biol Pharm Bull 44 :1819-1823, 2021 34853264
35. Wu S , Pan Y , Mao Y , et al : Current progress and mechanisms of bone metastasis in lung cancer: A narrative review. Transl Lung Cancer Res 10 :439-451, 2021 33569325
36. Knapp BJ , Devarakonda S , Govindan R : Bone metastases in non-small cell lung cancer: A narrative review. J Thorac Dis 14 :1696-1712, 2022 35693589
37. Subramanian V , Mascha EJ , Kattan MW : Developing a clinical prediction score: Comparing prediction accuracy of integer scores to statistical regression models. Anesth Analg 132 :1603-1613, 2021 33464759
