==== Front Arthritis Res Ther Arthritis Res Ther Arthritis Research & Therapy 1478-6354 1478-6362 BioMed Central London 2355 10.1186/s13075-020-02355-0 Research Article Derivation and internal validation of a multi-biomarker-based cardiovascular disease risk prediction score for rheumatoid arthritis patients http://orcid.org/0000-0002-8907-8976Curtis Jeffrey R. jrcurtis@uabmc.edu 1 Xie Fenglong 1 Crowson Cynthia S. 2 Sasso Eric H. 34 Hitraya Elena 34 Chin Cheryl L. 34 Bamford Richard D. 34 Ben-Shachar Rotem 4 Gutin Alexander 4 Flake Darl D. II4 Mabey Brent 4 Lanchbury Jerry S. 4 1 grid.265892.20000000106344187University of Alabama at Birmingham, Birmingham, AL USA 2 grid.66875.3a0000 0004 0459 167XMayo Clinic, Rochester, MN USA 3 grid.420032.70000 0004 0460 790XCrescendo Bioscience, South San Francisco, CA USA 4 grid.420032.70000 0004 0460 790XMyriad Genetics Laboratories, Salt Lake City, UT USA 4 12 2020 4 12 2020 2020 22 28213 8 2020 15 10 2020 © The Author(s) 2020Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.Background Rheumatoid arthritis (RA) patients have increased risk for cardiovascular disease (CVD). Accurate CVD risk prediction could improve care for RA patients. Our goal is to develop and validate a biomarker-based model for predicting CVD risk in RA patients. Methods Medicare claims data were linked to multi-biomarker disease activity (MBDA) test results to create an RA patient cohort with age ≥ 40 years that was split 2:1 for training and internal validation. Clinical and RA-related variables, MBDA score, and its 12 biomarkers were evaluated as predictors of a composite CVD outcome: myocardial infarction (MI), stroke, or fatal CVD within 3 years. Model building used Cox proportional hazard regression with backward elimination. The final MBDA-based CVD risk score was internally validated and compared to four clinical CVD risk prediction models. Results 30,751 RA patients (904 CVD events) were analyzed. Covariates in the final MBDA-based CVD risk score were age, diabetes, hypertension, tobacco use, history of CVD (excluding MI/stroke), MBDA score, leptin, MMP-3 and TNF-R1. In internal validation, the MBDA-based CVD risk score was a strong predictor of 3-year risk for a CVD event, with hazard ratio (95% CI) of 2.89 (2.46–3.41). The predicted 3-year CVD risk was low for 9.4% of patients, borderline for 10.2%, intermediate for 52.2%, and high for 28.2%. Model fit was good, with mean predicted versus observed 3-year CVD risks of 4.5% versus 4.4%. The MBDA-based CVD risk score significantly improved risk discrimination by the likelihood ratio test, compared to four clinical models. The risk score also improved prediction, reclassifying 42% of patients versus the simplest clinical model (age + sex), with a net reclassification index (NRI) (95% CI) of 0.19 (0.10–0.27); and 28% of patients versus the most comprehensive clinical model (age + sex + diabetes + hypertension + tobacco use + history of CVD + CRP), with an NRI of 0.07 (0.001–0.13). C-index was 0.715 versus 0.661 to 0.696 for the four clinical models. Conclusion A prognostic score has been developed to predict 3-year CVD risk for RA patients by using clinical data, three serum biomarkers and the MBDA score. In internal validation, it had good accuracy and outperformed clinical models with and without CRP. The MBDA-based CVD risk prediction score may improve RA patient care by offering a risk stratification tool that incorporates the effect of RA inflammation. Keywords BiomarkerCardiovascular riskMBDA scoreMulti-biomarkerMyocardial infarctionRheumatoid arthritisStrokeMyriad Genetics, Inc.issue-copyright-statement© The Author(s) 2020 ==== Body Background Cardiovascular disease (CVD) is the leading cause of mortality for patients with rheumatoid arthritis (RA), accounting for 30–40% of deaths [1]. Patients with RA have approximately 50% greater risk for cardiovascular disease (CVD) compared to the general population [2]. Traditional CVD risk factors such as diabetes, hypertension, and hyperlipidemia are important in RA patients and are not difficult to assess. However, the time constraints of a busy office practice often preclude making CVD risk stratification a routine part of RA patient care. Indeed, 79% of rheumatologists cite a lack of time as a major barrier [3]. Even so, rheumatologists are well positioned to help manage CVD risk in RA patients because 30% of CVD risk in RA patients is attributable to systemic inflammation and other RA-related factors [4, 5]. CVD risk predictors developed for the general population tend to underestimate CVD risk in RA patients [6–8]. European League Against Rheumatism (EULAR) guidelines recommend that CVD risk predicted by tools such as the Framingham Risk Score (FRS) or the American College of Cardiology and American Heart Association (ACC/AHA) pooled cohort risk equation [9] be multiplied by 1.5 to account for the effect of RA on CVD risk [6, 10]. A limitation of this approach is that it treats all RA patients the same, regardless of the level of disease activity. ACC/AHA guidelines recommend preventive strategies for all patients with high predicted risk of CVD. Current recommendations support managing hyperlipidemia by “treating to risk” rather than a targeted LDL [11–13]. It is well established that vascular inflammation has a central role in atherosclerosis and CVD, but evidence that reducing systemic inflammation has potential to lower CVD risk is more recent. Proof of principle comes from the CANTOS trial, which showed that canakinumab, an anti-IL-1β biologic drug, reduced the CVD event rate in non-RA patients with a high risk of CVD and elevated high-sensitivity C-reactive protein (CRP) [14]. Patients with greater reduction in inflammation, measured by CRP, benefited the most [15]. Synovial and systemic inflammation in RA patients contribute to CVD risk independently of traditional risk factors [4]. In observational studies, the risk for CVD events was greatest in RA patients with high disease activity [16–20] and effective RA treatment appeared to reduce the risk for atherosclerosis [21] and CVD events [22, 23]. Traditional CVD risk factors, such as diabetes, may be exacerbated by RA-related mechanisms [24, 25]. Thus, it may be possible to reduce the CVD risk elevation attributable to RA by treating RA inflammatory pathways. High sensitivity CRP has prognostic value for CVD events in non-RA populations, but its role for CVD risk prediction in RA patients is less clear because CRP may be a marker for systemic inflammation in RA rather than a surrogate for the extent of vascular involvement [26]. Moreover, CRP is not elevated in some RA patients with active disease [27]. CVD risk prediction models that combine measures of RA disease activity with traditional risk factors [19, 28, 29] are not yet the standard of care. Molecular markers of inflammation other than CRP have not been incorporated into validated CVD risk predictors for RA patients. Their inclusion would be novel and may have potential to improve CVD preventive care for RA patients by making CVD risk stratification more accurate and accessible. The multi-biomarker disease activity (MBDA) test assesses RA disease activity by measuring 12 serum protein biomarkers to provide a validated score on a scale of 1–100 that correlates with the Disease Activity Score in 28 joints with CRP (DAS28-CRP) [30]. In 2019, the American College of Rheumatology disease activity measures working group concluded that the MBDA score was one of 11 measures of RA disease activity that met the minimum standard for regular use [31]. The MBDA score is predictive of future radiographic damage, independently of other measures [32, 33]. In a large, cross-sectional observational study, the MBDA score was found to be associated with risk for CVD, suggesting that the MBDA score and at least some of its biomarkers detect inflammation that is relevant to cardiovascular pathology [16]. Building on this evidence, we now describe the development and internal validation of an RA-specific CVD risk prediction score that uses routine clinical assessments plus RA-related biomarkers to predict CVD risk. The goal of this approach was to improve preventive CVD care in RA patients by developing a prognostic score that uses biomarkers to incorporate the contribution of RA-related inflammation to individual CVD risk. The intended end result of this endeavor is to create a validated CVD risk score that will enable rheumatologists to risk stratify their RA patients efficiently in an office setting, with components associated with RA disease activity directly represented in the CV risk estimate. Methods Data source A retrospective RA cohort was created for this study by linking claims data in the Medicare database with data in the MBDA test commercial database (Vectra®, formerly Crescendo Bioscience, Inc., South San Francisco, CA, USA, currently Myriad Genetics Laboratories, Salt Lake City, UT, USA), using all fee-for-service Medicare data from 2006 to 2016 for all individuals who underwent MBDA testing. Data were linked on patient date of birth, sex, MBDA test date, MBDA testing codes (defined by Current Procedural Terminology codes 81479, 83520, 84999, 86140, and 81490, submitted by Crescendo Bioscience or Myriad Genetics Laboratories), and the National Provider Identifier of the treating rheumatologist. Data were linked deterministically, using established methods [16, 34]. The University of Alabama at Birmingham institutional review board approved the study. Participant and MBDA test eligibility criteria The patient cohort and MBDA test results included in this study were selected by applying a series of criteria to the patients and MBDA tests in the linked database described above (Supplemental Table 1). To be eligible for inclusion in the study, patients were required to (1) be ≥ 40 years old, (2) have at least one RA diagnosis code from a rheumatologist (ICD9 714.0; ICD10 M05.*, M06.*, excluding M06.4 and M06.1, with * representing any number of digits or characters), (3) have received an RA-specific treatment (TNF-inhibitor, abatacept, rituximab, anti-IL-6R, Janus kinase inhibitor, conventional synthetic disease-modifying anti-rheumatic drug including methotrexate, sulfasalazine, leflunomide and hydroxychloroquine) anytime up to and including the date of the first MBDA test, and (4) have at least one linked MBDA test result. The accuracy of this claim-based method of identifying RA patients exceeds 85% [35] and is likely made greater here by the linkage with data from MBDA testing, which is only for patients diagnosed with RA. The baseline period for a patient was defined as the interval preceding the date of the first MBDA test in the linked database. It included all available preceding Medicare data and was required to span at least 1 year, with patients being required to have had at least 365 days of continuous coverage with Medicare parts A (hospital coverage), B (outpatient coverage), and D (pharmacy coverage). Patients were excluded if they had any diagnosis code in the baseline period for malignancy (except non-melanoma skin cancer), myocardial infarction (MI), or stroke. MBDA test results (i.e., the MBDA score and 12 biomarker measurements) were used from the earliest MBDA test performed after the above requirements had been met, unless (1) it was performed within 14 days following any hospital discharge or (2) the patient had used anti-IL-6R treatment in the preceding 90 days (because tocilizumab treatment may affect the MBDA score in a way that might confound CVD risk prediction) [36]; in these cases, the next MBDA test meeting the above requirements was used and the baseline period was anchored to that test. The follow-up period for ascertaining CVD outcomes (see below) began on the date of the first qualifying MBDA test. The follow-up period ended at the earliest of (1) a CVD outcome, (2) diagnosis of malignancy, (3) non-CVD death, or (4) the end of study (December 31, 2016). CVD outcome The CVD outcome we used for the prognostic test was a composite, defined as the occurrence of hospitalized MI, stroke, or fatal CVD. This outcome definition is consistent with the outcome used in the guidelines of the ACC/AHA [9]. MI was defined as ICD-9 diagnosis code 410.x1 or ICD-10 diagnosis code I21.* from an inpatient hospitalization lasting ≥ 1 night or where the patient died. Stroke was identified using ICD-9 diagnosis codes 430.*, 431.*, 433.x1, 434.x1, 436.* or ICD-10 diagnosis codes I60.*, I61.*, I63.* or I67.89 from hospital discharge. This approach has been described previously [37–39]. Fatal CVD was identified using a validated algorithm that identifies fatal MIs and fatal strokes from Medicare data at a threshold yielding a positive predictive value > 80%, with greater accuracy than is obtained using hospital discharge diagnoses [40]. Biomarkers and other predictors MBDA score All biomarker data in this study came from the MBDA test, which measures the serum concentrations of 12 biomarkers and uses an algorithm to produce a disease activity score on a scale of 1 to 100. The MBDA score has been validated against DAS28-CRP in patients treated with a variety of RA therapies, with AUROC values of 0.77 and 0.70 observed in seropositive and seronegative RA patients, respectively [30, 41]. The MBDA score is used to assess and monitor inflammatory disease activity in RA patients and is complementary to clinical assessment. It is a stronger predictor of risk for radiographic progression than DAS28-CRP [32, 33]. The MBDA score is not intended for the diagnosis of RA but rather is for use in assessing disease activity in patients with already-diagnosed RA. The MBDA score has been available for use in clinical practice in the US since 2010. Its cost has been covered in the US by Medicare since 2013 and is also covered by some private insurers. The biomarkers in the MBDA test reflect the biology of RA and comprise cytokine-related proteins (IL-6, TNF-R1), acute phase reactants (CRP, serum amyloid A), an adhesion molecule (VCAM-1), a skeletal-related protein (bone glycoprotein 39 [YKL-40]), growth factors (EGF, VEGF-A), matrix metalloproteinases (MMP-1, MMP-3), and adipokines (leptin, resistin). All MBDA scores analyzed here were from tests that had been ordered by practitioners in the US as part of routine patient care. All MBDA testing was performed in a Clinical Laboratory Improvements Amendment-certified commercial laboratory in South San Francisco, CA (Crescendo Bioscience), where MBDA scores were calculated and stored with related data in a secure database. Prior to and independently of the present study, an algorithm was developed and validated to adjust the MBDA score for the effects of age, sex, and leptin (as a surrogate for adiposity) [42]. This adjustment acts on the original MBDA score without affecting the individual contributions of the 12 biomarkers. Thus, the original MBDA score is calculated as previously, then adjusted to produce a score that, like the original score, has a scale of 1–100 and RA disease activity categories of low (< 30), moderate (30–44), and high (> 44) [30, 42]. The adjusted MBDA score has been in routine use since December 2017. Original MBDA scores were converted to adjusted MBDA scores for this study. In the remainder of this report, the term “MBDA score” means the adjusted MBDA score. Variables considered for inclusion in model building Variables considered for use in model building that came from the MBDA database included the MBDA score and the serum concentrations of its 12 component biomarkers. This approach was non-redundant because the algorithm for the MBDA score is a non-linear combination of its component biomarkers, which were neither selected nor weighted for CVD prediction [30, 41]. Demographic and clinical predictors were obtained from the Medicare database and were considered for inclusion in model building based upon their expected association with CVD risk, informed by subject matter expertise and the medical literature. Other considerations were face validity, data quality in the Medicare database, and feasibility of collecting a variable accurately in clinical practice. These predictors included age, sex, race, tobacco use (past or present), history of CVD other than MI or stroke, diagnoses of and medications for diabetes, hypertension and hyperlipidemia, RA medications as described above, glucocorticoids, and non-steroidal anti-inflammatory drugs. A diagnosis was counted as present if any of its diagnostic codes was found for the patient. Diagnostic codes for the candidate predictors, i.e., the subset of variables that were included in the final model-building exercise, and the prevalences of CVD-related conditions, appear in Supplemental Table 2. Clinical measurements (e.g., blood pressure or lipid levels) were not available in either database and were not considered for inclusion in model building. Current use of CV-related medications (e.g., lipid-lowering therapies) and RA medications was initially considered and was evaluated as part of baseline data assessment. However, a decision was made to not include any medications as variables in model building for two reasons: (1) without being able to account for disease-related clinical measurements, the estimated effect of medications may be counterintuitive or inaccurate and (2) suboptimal medication adherence could result in meaningful misclassification of the CV risk associated with these treatments. Race was excluded because of uncertainties related to racial heterogeneity and the reporting of race. Statistical analysis A principled, pre-specified approach to model building and selection was conducted that followed Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines [43]. First, the cohort was randomly split 2:1 into separate datasets for training and testing (i.e., internal validation). Prior to model building, the independent association of the MBDA score with the CVD risk was evaluated in the training dataset with a multivariable analysis that included all non-biomarker candidate predictors [16]. Separately, the form of the relationship between MBDA score and CVD risk, on the logarithmic scale of hazard, was examined and found to be linear up to MBDA scores of approximately 60 and non-linear thereafter—a relationship that can be described with a hyperbolic tangent function (see below), which is commonly used in other fields, e.g., in models of neural networks [44]. Training: evaluation of variables and model building Model development was conducted in the training dataset, to achieve the goal of estimating individual risk for the composite CVD outcome as a function of the candidate predictors. Individual biomarker concentrations in ng/ml were natural log transformed. MBDA scores (integers on a scale of 1 to 100) were hyperbolic tangent-transformed, as f(x) = tanh(a ∗ x), where a is a constant parameter that was based on maximum likelihood estimation and updated in each step of model building. Age in years was treated as a continuous variable. A separate age-squared term was initially included to account for possible nonlinearity between age and the composite CVD outcome, but it added no additional value to model building and was dropped. Other candidate predictors were treated as binary variables. Association with 3-year CVD risk was assessed for each candidate variable with a hazard ratio (HR) and determined by univariable analysis in the training dataset. A 3-year time frame was chosen based on the availability of MBDA biomarker data from testing performed as part of routine care. Model building used Cox proportional hazards regression with backward elimination in the training dataset. In the first step, a model was fit by including every candidate predictor variable; in each subsequent step, the least significant variable (i.e., with the highest p value) was removed, and the model was refit with the remaining variables. This process was repeated until all remaining variables had p < 0.05. Clinical models developed for comparison Four prespecified models for predicting CVD risk were built in the training dataset for comparison with the MBDA-based model: (1) age + sex, (2) age + sex + CRP, (3) a clinical model (age + sex + tobacco use + diabetes + hypertension + history of CVD [excluding MI and stroke]), and (4) the clinical model + CRP. These models were chosen for the availability of their variables in routine clinical practice and in our linked database. Derivation of categories of 3-year risk for CVD events The thresholds for 3-year CVD risk categories that would be equivalent to the thresholds for 10-year risk categories of other CVD risk prediction equations were derived in a cohort with 10 years of longitudinal data. To create a dataset in which CVD event rates at 3 and 10 years could be bridged, a cohort of 533,139 Medicare RA patients with data available from 2006 to 2016 was selected with the same requirements as for the main cohort of this study but without requiring MBDA testing. An age + sex model was developed in this cohort to establish 10-year rates of CVD events, and 3-year cutpoints corresponding to the 10-year ACC/AHA risk thresholds of 5% (± 0.1%), 7.5% (± 0.1%), and 20% (± 0.1%) [11] were obtained by bootstrapping. The derived cutpoints were 1.3%, 1.8%, and 5.2%, defining 3-year CVD risk categories of low (0 to < 1.3%), borderline (≥ 1.3 to < 1.8%), intermediate (≥ 1.8 to < 5.2%), and high (≥ 5.2%) risk. Internal validation The primary analysis for establishing internal validation was to estimate the risk of a composite CVD event at 3 years (i.e., the probability of a patient having an MI, a stroke, or CVD death in the next 3 years), by using the MBDA-based CVD risk score as the only variable in a Cox proportional hazard regression model. HR (with 95% confidence interval [CI]; p value by partial likelihood ratio test [LRT]) was determined for the MBDA-based CVD risk score [45–47]. A risk curve was constructed to illustrate this relationship, using methods described in Supplementary Text. These and all other validation analyses were performed in the validation dataset. To assess accuracy of the MBDA-based CVD risk score, a secondary analysis for internal validation examined goodness of fit with plots that compared observed risk (based on Kaplan-Meier estimates with 95% CI) with predicted risk across CVD event-based deciles. P values were determined using the Greenwood-Nam-D’Agostino test [48], with higher (i.e., non-significant) p values indicating better fit. Goodness of fit was also assessed among patient subgroups, based on age, sex, diagnosis of diabetes, hypertension, tobacco use (past or present), and hyperlipidemia, as well as history of CVD, statin use, oral glucocorticoid use, initiation or change of a biologic agent during follow-up, and MBDA score category. Bonferroni correction was used to adjust for multiple testing. CVD event quintiles, rather than deciles, were used for patient subgroups with fewer than 110 CVD events to avoid data sparsity. In addition, Kaplan-Meier plots of CVD event-free status over time were constructed for patients grouped into CVD risk categories by the MBDA-based CVD risk score, using the Mantel-Haenszel test [45, 46]. Validation included comparisons of the predictive abilities of the MBDA-based CVD risk score and four clinical models described above. HR (95% CI) and p value (using the partial LRT) were calculated from Cox proportional hazards models in single-score (i.e., univariable) analyses of the MBDA-based CVD risk score and each of the four clinical models. To determine the incremental contribution of the MBDA-based model to each clinical model for predicting CVD risk (and vice versa), change in model deviance was determined using the likelihood ratio statistic in sequential (i.e., bivariable) analyses for each model pair. The MBDA-based CVD risk model was also compared to the four clinical models with reclassification tables and the Net Reclassification Index (NRI) [49, 50]. The five models were each evaluated for discrimination based on the C-index (similar to AUROC) for predicting risk at 3 years, with times weighted by the square inverse of the censoring distribution [51]. Statistical software SAS 9.4 was used for data preparation. R version 3.4 and R packages survival, nricens, and pec were used for evaluating model performance, calculating NRIs and C-indices, and generating plots [52]. Results Cohort selection 30,751 RA patients with 904 CVD events (480 MI, 362 stroke, 62 CVD death) were eligible for the total cohort (Supplemental Table 1). Total follow-up from the index date was 56,684 patient-years (PY) with median (interquartile range [IQR]) follow-up duration of 1.7 (0.8–2.7) years. The overall CVD event rate (95% CI) was 15.9 (14.9–17.0) events per 1000 PY. At baseline, the mean age was 69 years, 23% of patients were under age 65 years, 18% were men, and 8% were Black (Table 1). The prevalence of CVD-related comorbidities, such as diabetes (40%) and hypertension (79%), was high. Statin use was found in 42%. Sixty percent of patients were receiving methotrexate, 33% a TNF inhibitor (TNFi), and 15% a non-TNFi biologic. Median (IQR) CRP value was 4.5 (1.6–12.0) mg/L (or 1.5 [0.5–2.5] μg/ml natural log transformed). Median (IQR) MBDA score was 40 [32–49], which is in the moderate MBDA category (range, 30–44) (Table 1). Table 1 Patient characteristics at baseline* Predictors Complete cohort N = 30,751 Patients with CVD event N = 904 Patients with no CVD event N = 29,847 Age, mean (SD) 68.8 (9.6) 72.7 (9.3) 68.6 (9.6) Age group, %  < 65 years 23.4 14.6 23.6  65–74 years 50.9 42.8 51.1  > 74 years 25.8 42.6 25.3 Male, % 18.2 23.0 18.1 Black race, % 8.4 6.9 8.4 Comorbidities*, %  Diabetes 39.8 47.7 39.5  History of CVD 37.1 55.0 36.5  Hyperlipidemia 75.4 81.5 75.2  Hypertension 78.7 88.9 78.4  Obesity 12.1 8.7 12.2  Tobacco use (past or current) 24.5 27.7 24.3 Medications, %  ACEI 25.9 32.4 25.7  ARB 22.0 24.2 22.0  Beta-blockers 34.4 48.6 34.0  Statins 42.4 45.1 42.4 RA medications, %  Methotrexate 59.8 58.4 59.9  Other csDMARDs 44.7 44.5 44.7  TNFi biologics 32.8 29.5 32.9  Non-TNFi biologics 14.8 15.8 14.8  Abatacept 9.9 11.3 9.9  Rituximab 3.8 4.1 3.8  Tocilizimab 1.9 1.4 2.0  Tofacitinib 1.7 < 1.2 1.7  Oral glucocorticoids 57.5 62.7 57.4  NSAIDs 48.0 45.0 48.1 Biomarkers†, median [IQR]  CRP (ug/ml) 1.5 [0.5, 2.5] 1.9 [0.9, 2.9] 1.5 [0.5, 2.5]  EGF (pg/ml) 4.5 [3.7, 5.1] 4.4 [3.7, 5.0] 4.5 [3.7, 5.1]  IL-6 (pg/ml) 2.6 [2.1, 3.3] 2.8 [2.3, 3.6] 2.6 [2.1, 3.3]  Leptin (ng/ml) 3.2 [2.3, 3.9] 3.0 [2.1, 3.7] 3.2 [2.3, 3.9]  MMP-1 (ng/ml) 1.9 [1.4, 2.4] 1.9 [1.4, 2.4] 1.9 [1.4, 2.4]  MMP-3 (ng/ml) 3.3 [2.8, 3.8] 3.5 [3.0, 4.1] 3.3 [2.8, 3.8]  Resistin (ng/ml) 2.1 [1.8, 2.4] 2.1 [1.9, 2.5] 2.1 [1.8, 2.4]  SAA (ug/ml) 1.0 [0.3, 1.9] 1.3 [0.5, 2.5] 1.0 [0.3, 1.9]  TNF-R1 (ng/ml) 0.6 [0.3, 0.8] 0.8 [0.5, 1.1] 0.6 [0.3, 0.8]  VCAM1 (ug/ml) − 0.4 [− 0.5, − 0.2] − 0.3 [− 0.5, − 0.1] − 0.4 [− 0.5, − 0.2]  VEGF (pg/ml 5.5 [5.1, 5.9] 5.6 [5.2, 6.0] 5.5 [5.1, 5.9]  YKL-40 (ng/ml) 4.7 [4.2, 5.3] 5.1 [4.5, 5.6] 4.7 [4.2, 5.3] MBDA score, median [IQR] 40.0 [32.0, 49.0] 44.0 [36.0, 54.0] 40.0 [32.0, 48.0] *Based on diagnostic codes and administration and fill information in the baseline period (see Methods) †All biomarker concentrations were from the MBDA test at the end of the baseline period and are natural log transformed. MBDA score is the adjusted score CVD event is myocardial infarction, stroke, or CV death in 3 years from baseline ACEI angiotensin converting enzyme inhibitor, ARB angiotensin receptor blocker, csDMARD conventional synthetic DMARD, CVD cardiovascular disease, IQR interquartile range, MBDA multi-biomarker disease activity (adjusted), NSAID non-steroidal anti-inflammatory drug, SD standard deviation, TNFi tumor necrosis factor inhibitor Confirming the MBDA score as an independent predictor of CVD risk In the training dataset (N = 20,476 patients with 611 CVD events), the MBDA score, untransformed, was significantly prognostic of CVD events in a multivariable analysis with age, sex, diabetes, hypertension, tobacco use, CVD history, and hyperlipidemia, but with no individual biomarker variables (HR = 1.023; 95% CI 1.017–1.029). Training of the MBDA-based model In univariable analyses in the training dataset, all candidate predictors except EGF and MMP-1 were individually predictive of CVD events (Table 2). In the final MBDA-based model, derived from backward elimination, the variables of age, diabetes, history of CVD, hypertension, tobacco use, MBDA score, and three biomarkers (leptin, MMP-3, TNF-R1) were significant predictors in multivariable analyses; sex, hyperlipidemia, and nine biomarkers were not. HRs were significantly > 1.0 for all predictor variables in the final MBDA-based model except leptin, for which HR was 0.84, indicating a negative relationship between leptin concentration and CVD risk (Table 2). Table 2 Hazard ratios (HR) of predictor variables used in CVD risk models (training dataset, N = 20,476) Predictors Univariable models Multivariable models HR (CI) p value Age + Sex HR (CI) Age + Sex + CRP HR (CI) Clinical HR (CI) Clinical + CRP HR (CI) Final MBDA-based model HR (CI) p value Age 1.05 (1.04–1.06) 3.45 × 10−24 1.05 (1.04–1.06) 1.05 (1.04–1.06) 1.04 (1.03–1.05) 1.04 (1.03–1.05) 1.03 (1.02–1.04) 6.68 × 10−11 Male 1.39 (1.15–1.68) 9.29 × 10−4 1.43 (1.19–1.73) 1.43 (1.19–1.73) 1.31 (1.08–1.59) 1.32 (1.09–1.60) – – Comorbidities*  Diabetes 1.49 (1.27–1.74) 1.08 × 10−6 – – 1.31 (1.11–1.54) 1.29 (1.10–1.52) 1.31 (1.11–1.55) 0.0012  History of CVD 2.01 (1.71–2.35) 1.08 × 10−17 – – 1.47 (1.24–1.74) 1.44 (1.21–1.71) 1.40 (1.18–1.66) 9.85 × 10−5  Hyperlipidemia 1.36 (1.11–1.66) 0.0023 – – – – – –  Hypertension 2.52 (1.84–3.46) 5.49 × 10−11 – – 1.35 (1.04–1.75) 1.30 (1.00–1.69) 1.31 (1.01–1.71) 0.0405  Tobacco use 1.38 (1.16–1.65) 4.55 × 10−4 – – 1.42 (1.19–1.70) 1.35 (1.12–1.61) 1.31 (1.09–1.57) 0.0044 Molecular  MBDA score 1.02 (1.02–1.03) 4.98 × 10−14 – – – – – –  Tanh-MBDA‡ 15.35 (7.17–32.87) 3.70 × 10− 14 – – – – 4.99 (2.24–11.13) 4.22 × 10−5  CRP†, mg/L 1.20 (1.13–1.26) 7.04 × 10−11 – 1.22 (1.16–1.29) – 1.20 (1.13–1.26) – –  EGF†, ng/mL 0.92 (0.85–1.00) 0.0650 – – – – – –  IL-6†, ng/mL 1.29 (1.20–1.38) 1.92 × 10−11 – – – – – –  Leptin†, ng/mL 0.87 (0.81–0.93) 7.93 × 10−5 – – – – 0.84 (0.79–0.90) 2.99 × 10−6  MMP-1†, ng/mL 1.07 (0.96–1.19) 0.2497 – – – – – –  MMP-3†, ng/mL 1.49 (1.35–1.65) 4.43 × 10−14 – – – – 1.16 (1.03–1.30) 0.0139  Resistin†, ng/mL 1.58 (1.32–1.90) 7.32 × 10−7 – – – – – –  SAA†, ng/mL 1.19 (1.13–1.26) 9.38 × 10−10 – – – – – –  TNF-R1†, ng/mL 2.83 (2.39–3.35) 8.32 × 10−29 – – – – 1.77 (1.43–2.19) 3.33 × 10−11  VCAM-1†, ng/mL 3.04 (2.36–3.91) 1.13 × 10−16 – – – – – –  VEGF†, ng/mL 1.35 (1.19–1.54) 5.20 × 10−6 – – – – – –  YKL-40†, ng/mL 1.60 (1.46–1.76) 1.30 × 10−21 – – – – – – These predictors comprise the complete list of predictors tested by backwards elimination to build the final MBDA-based model; also shown is the adjusted MBDA score, untransformed. P values by the likelihood ratio test *Based on diagnostic codes during the baseline period (see the “Methods” section) †Natural log transformed ‡Hyperbolic tangent transformed (tanh [a × MBDA Score [adjusted]], where a = 1/33.0807) Clinical model includes age + sex + tobacco use + diabetes + hypertension + history of CVD (excluding MI and stroke) CI 95% confidence interval, CVD cardiovascular disease, MBDA multi-biomarker disease activity The equation for the final MBDA-based CVD risk score was: 0.0314×age+0.2691×tobaccouse+0.2732×diabetes+0.2694×hypertension+0.3378×history ofCVD−0.1711×InLeptin+0.1454×lnMMP3+0.5724×lnTNFR1+1.6076×tanhMBDA score/33.0807, where the age is in years, clinical variables are scored as 1 when present and zero when absent, Leptin, MMP-3, and TNF-R1 represent serum concentrations in ng/mL, the term “ln” means natural logarithm, and “tanh” means hyperbolic tangent transformation. The output of this algorithm is the MBDA-based CVD risk score. This score is used in a separate formula to calculate the predicted 3-year risk for a CVD event as a percentage value (see Supplemental Text). In the four multivariable clinical models that were generated for comparison—i.e., an age + sex model and an age + sex + diabetes + hypertension + history of CVD + tobacco use model, each one with and without CRP—all variables in each model were significant CVD predictors (Table 2). Internal validation of the MBDA-based model The MBDA-based CVD risk score was a strong predictor of 3-year risk for a CVD event in the validation dataset (N = 10,275 patients with 293 CVD events), with an HR (95% CI) of 2.89 (2.46–3.41, p = 4.67 × 10− 38). The relationship between the MBDA-based CVD risk score and predicted 3-year CVD risk is shown in Fig. 1a. The proportions of patients in the low, borderline, intermediate, and high categories of predicted 3-year CVD risk in the validation dataset were 9.4%, 10.2%, 52.2%, and 28.2%, respectively (Fig. 1b). Fig. 1 Characterization of the MBDA CVD risk score in the validation dataset (N = 10,275). a Relationship between MBDA-based CVD risk score and predicted 3-year risk of a CVD event, with 95% confidence interval. b Distribution of predicted 3-year risks. Dotted lines, horizontal in a and vertical in b, indicate thresholds at 1.3%, 1.8%, and 5.2% separating the categories of low, borderline, intermediate, and high risk, which contained 9.4%, 10.2%, 52.2%, and 28.2% of patients, respectively. CVD event is myocardial infarction, stroke, or CVD death. CVD cardiovascular disease, MBDA multi-biomarker disease activity Assessment of accuracy with goodness of fit The 3-year CVD risk predictions made by the MBDA-based model were similar to the observed CVD event rates across deciles based on observed CVD events (Fig. 2). The goodness of fit test statistic indicated good fit (p = 0.39). The confidence intervals for observed risk contained the average predicted risk for all but one decile group. Overall, the mean predicted 3-year CVD risk in the validation dataset was 4.5%, compared with the observed 3-year CVD risk of 4.4%. Subanalyses showed that the MBDA-based model performed well in subgroups of interest: males and females, with/without diagnosis of diabetes, with/without diagnosis of hypertension, with/without tobacco use, with/without history of CVD, with/without hyperlipidemia, taking/not taking statins, < 65 years old, < 75 years old, and patients who had or had not used oral glucocorticoids in the baseline period, or initiated or changed a biologic drug during the follow-up period, or had low, moderate, or high disease activity (MBDA score) (Supplemental Fig. 1). Fig. 2 Goodness of fit: Predicted CVD risk versus observed 3-year CVD event rates. The observed 3-year CVD event rate was determined for each event-based decile and is shown vs. the average predicted 3-year risk in each decile. Analysis used the validation dataset (N = 10,275). Observed event rates were determined as Kaplan-Meier (95% log-log CI) estimates. P = 0.39 by the Greenwood-Nam-D’Agostino test, indicating good fit. CVD event is myocardial infarction, stroke, or CV death. 3-year CVD risk categories (low, borderline, intermediate, high) were derived from the 10-year risk categories of the 2018 Guidelines of the American College of Cardiology/American Heart Association [8]. Threshold between low and borderline risk categories is 1.3% (not shown). CI confidence interval, CVD cardiovascular disease, MBDA multi-biomarker disease activity Loss of CVD outcome-free status by category of predicted risk A Kaplan-Meier plot depicting loss of CVD outcome-free status in the validation dataset showed statistically significant separation of the low, borderline, intermediate and high predicted CVD risk groups over time (p = 1.7 × 10−32) (Fig. 3). Fig. 3 Kaplan-Meier plot of CVD event-free survival. Occurrence of CVD events by Kaplan-Meier survival analysis is shown for patients in the validation dataset (N = 10,275) grouped by a 3-year CVD risk category predicted by the MBDA-based CVD risk score at baseline. P = 1.7 × 10−32 by the Mantel-Haenszel test. CVD event is myocardial infarction, stroke, or CVD death. See Fig. 2 for explanation of CVD risk categories. CVD cardiovascular disease, MBDA multi-biomarker disease activity Model evaluation and comparison by likelihood test When analyzed alone, each of the four clinical models made statistically significant contributions to the prediction of CVD risk in terms of the likelihood ratio, which represents how well the model fits the data (Fig. 4). However, these models made smaller contributions than the MBDA-based CVD risk score (Fig. 4). Moreover, the addition of these clinical models to the MBDA-based CVD risk score in paired analyses did not improve CVD risk prediction, as indicated by the respective increments in LRT statistic (0.4–3.0), which were small and non-significant (Table 3). In contrast, the MBDA-based CVD risk score provided additional information to improve the prediction of CVD risk when it was added to each clinical model, with the increments in LRT statistic being large (35.4–83.3) and statistically significant (all p < 3 × 10− 9) (Table 3). Fig. 4 Contribution to CVD risk prediction by MBDA-based CVD risk score and clinical models. Likelihood ratio test statistics are shown for univariable (i.e., single-score) analyses of a CVD risk prediction by the MBDA-based CVD risk score and four comparison models, using the validation dataset (N = 10,275) (see also Table 3). P values are by the likelihood ratio test. The clinical model includes age, sex, tobacco use, diabetes, hypertension, and history of CVD. CRP C-reactive protein, CVD cardiovascular disease, MBDA multi-biomarker disease activity Table 3 Contribution of MBDA-based CVD risk score and other models to prediction of 3-year CVD risk CVD risk prediction score Single-score analyses (univariable) Sequential analyses of paired scores (bivariable) A. Non-MBDA-based CVD Risk Score added to a base model of MBDA-based CVD Risk Score B. MBDA-based CVD Risk Score added to a non-MBDA-based CVD Risk Score base model HR (95% CI) LRT statistic p value HR (95% CI) Increment in LRT statistic p value HR (95% CI) Increment in LRT statistic p value Age + sex 3.44 (2.62–4.53) 82.2 1.22 × 10−19 1.34 (0.96–1.86) 3.0 0.084 2.62 (2.14–3.20) 83.3 7.15 × 10−20 Age + sex + CRP 2.97 (2.38–3.70) 95.8 1.29 × 10−22 1.13 (0.84–1.54) 0.7 0.412 2.71 (2.15–3.41) 67.4 2.20 × 10− 16 Clinical 3.08 (2.50–3.80) 115.1 7.34 × 10−27 1.24 (0.89–1.71) 1.7 0.197 2.56 (1.98–3.29) 49.0 2.51 × 10−12 Clinical + CRP 2.94 (2.43–3.55) 127.5 1.44 × 10−29 1.12 (0.79–1.60) 0.4 0.526 2.67 (1.96–3.63) 35.4 2.67 × 10−9 MBDA-based CVD risk score 2.89 (2.46–3.41) 162.5 4.67 × 10−38 – – – – – – In single-score (univariable) analyses, each of the five risk scores derived from training was analyzed as a single independent variable for predicting risk for a CVD event in the validation dataset (N = 10,275). In sequential (bivariable) analyses of paired scores, the risk scores of the MBDA-based model and each non-MBDA-based risk model were evaluated as the only two variables used to predict CVD risk: (A) with the MBDA-based risk score as the base model and (B) with the non-MBDA-based CVD risk score as the base model. The increment in LRT statistic represents the extent to which a second CVD risk score adds to the CVD risk prediction ability of a first CVD risk score (i.e., the base model). Non-MBDA-based CVD risk score refers to the first four scores in the first column. P values by the likelihood ratio test. Statistically significant HR values are bolded. Clinical model includes age + sex + tobacco use + diabetes + hypertension + history of CVD (excluding MI and stroke) CRP C-reactive protein, CVD cardiovascular disease, HR hazard ratio, LRT likelihood ratio test, MBDA multi-biomarker disease activity Reclassification Compared to the simplest of the clinical models, the age + sex model, the MBDA-based model reclassified the CVD risk for 42% of patients overall and as many as 75% of patients, depending on the age + sex model risk category (Table 4A). Compared to the most comprehensive clinical model, the clinical + CRP model, the MBDA-based model reclassified the CVD risk for 28% of patients overall and as many as 64% of patients, depending on the clinical + CRP model risk category (Table 4B). Reclassification results for the age + sex + CRP model and the clinical model (without CRP) were generally intermediate to those of the other two models (Supplemental Tables 3A and 3B). Table 4 Reclassification of patients by the MBDA-based CVD risk score versus: A, age + sex model and B, clinical + CRP model A. CVD risk predicted by age + sex model CVD risk predicted by MBDA-based CVD model Observed cumulative incidence Total patients (n) within category of age + sex model and % reclassified Low (< 1.3%) Borderline (1.3 to < 1.8%) Intermediate (1.8 to < 5.2%) High (≥ 5.2%) Low (< 1.3%) 2.6% 0.9% 0.9% < 0.1% 1.4% 460 (42.6%) Borderline (1.3 to < 1.8%) 1.7% 1.4% 2.5% 0.2% 1.3% 600 (75.3%) Intermediate (1.8 to < 5.2%) 4.9% 7.3% 37.1% 11.0% 3.7% 6185 (38.4%) High (≥ 5.2%) 0.1% 0.6% 11.8% 17.0% 7.8% 3030 (42.4%) Observed cumulative incidence 0.9% 1.7% 3.1% 9.9% – – B. CVD risk predicted by clinical + CRP model CVD risk predicted by MBDA-based CVD model Observed cumulative incidence Total patients (n) within category of clinical + CRP model and % reclassified Low (< 1.3%) Borderline (1.3 to < 1.8%) Intermediate (1.8 to 5.2%) High (≥ 5.2%) Low (< 1.3%) 4.4% 1.2% 0.5% 0.0% 1.3% 627 (28.2%) Borderline (1.3 to < 1.8%) 3.0% 3.0% 2.3% < 0.1% 1.3% 853 (64.4%) Intermediate (1.8 to < 5.2%) 2.0% 6.0% 41.5% 5.4% 3.1% 5644 (24.4%) High (≥ 5.2%) 0 < 0.1% 7.9% 22.7% 9.3% 3151 (26.0%) Observed cumulative incidence 0.9% 1.7% 3.1% 9.9% – – Values in the 16 cross-classification cells are percentages of the total validation dataset (N = 10,275). Boldfaced values represent patients who were reclassified, i.e., they were classified differently by the MBDA-based CVD model and the other model. Observed cumulative incidence values represent CVD event rates among patients in a row or column. Percentages of patients reclassified are of the total number of patients in that row. Clinical + CRP model includes age + sex + tobacco use + diabetes + hypertension + history of CVD (excluding MI and stroke) and CRP CVD cardiovascular disease, MBDA multi-biomarker disease activity NRI test statistics demonstrated that the MBDA-based model significantly improved classification versus all four clinical models, with NRI test statistics (95% CI) of 0.19 (0.10–0.27) versus the age + sex model, 0.16 (0.08–0.23) versus the age + sex + CRP model, 0.10 (0.04–0.17) versus the clinical model, and 0.07 (0.001–0.13) versus the clinical + CRP model. Discrimination The C-index (95% CI) for the prediction of CVD risk at 3 years by the MBDA-based CVD risk score in the validation dataset was 0.715 (0.683–0.747), which was numerically greater than the C-index for each clinical model. The difference was greatest versus the simplest clinical model and least versus the most comprehensive clinical model, with C-indices (95% CI) of 0.661 (0.628–0.695) for the age + sex model, 0.674 (0.642–0.707) for the age + sex + CRP model, 0.688 (0.656–0.721) for the clinical model, and 0.696 (0.664–0.729) for the clinical + CRP model. Relationship between individual biomarkers and MBDA-based CVD risk score Scatterplots derived from the validation dataset demonstrate the positive relationships between 3-year risk predicted by the MBDA-based CVD risk score and MBDA score (r = 0.438), MMP-3 (r = 0.437), and TNF-R1 (r = 0.632); and the negative relationship with leptin (r = − 0.179). For the MBDA score and for each biomarker, at most levels a range of CVD risks was observed, consistent with variation among the other variables of the MBDA-based CVD risk score (Fig. 5). Fig. 5 Relationship between predicted CVD risk and molecular variables. The predicted 3-year risk for a CVD event (myocardial infarction, stroke, or fatal CVD) is shown versus (a) the MBDA score and (b–d) serum concentrations (ng/ml, natural log transformed) of the three biomarker variables in the MBDA-based CVD risk score, using the validation dataset (N = 10,275). R values are Spearman correlation coefficients. CVD cardiovascular disease, MBDA multi-biomarker disease activity Discussion We have used a cohort of over 30,000 RA patients to derive and internally validate an MBDA-based CVD risk score for use in patients with RA. This score reflects the contribution of systemic inflammation to CVD risk by including the MBDA score and three individual biomarkers, while also incorporating age and four clinical risk factors. The MBDA-based risk score accurately predicted CVD risk in terms of goodness of fit analyses in the internal validation cohort and in clinically relevant subgroups, including patients who did or did not have prior CVD, who were already taking statins, or had different levels of RA disease activity. The MBDA-based risk score discriminated CVD risk better than clinical models, assigning some patients to higher or lower risk categories compared with clinical assessment alone. This test is unique because it uses biomarker-based measurements to incorporate the contribution of RA inflammation to CVD risk in a more personalized way than by multiplying by a fixed value, such as 1.5 [6]. The MBDA score is a measure of RA disease activity that is also predictive of risk for radiographic progression. It was shown here and previously to be associated with the CVD risk [16], even though it was not originally developed for that purpose. MMP-3 and TNF-R1 were included in the final MBDA-based CVD risk score because in model building, they were positively associated with CVD risk independently of the MBDA score and other variables, which is consistent with previous reports of their role in cardiovascular risk [53–55]. The other individual biomarker in the CVD risk score was leptin. In our cohort, patients with a CVD event had less obesity and a numerically lower median leptin concentration than patients without a CVD event (Table 1). Leptin had a negative coefficient in the multivariable CVD risk prediction model. These results are consistent with evidence that leptin correlates strongly with body mass index (BMI) and that BMI has been negatively associated with CVD risk in RA patients [56], even though it is positively associated with CVD risk in the general population [57]. Our findings may reflect a contribution of RA inflammation to both weight loss and mortality, rather than a biologically protective effect of obesity [58]. They may also be a reflection of index case bias, which can lower the effect estimate for a risk factor, such as leptin, if it is associated with both the sequela of a disease and the disease itself, as with CVD events and RA [59]. IL-6, CRP, and other MBDA biomarkers were not included in the MBDA-based CVD risk score despite being individually associated with the CVD risk because none added significant information to leptin, MMP-3, TNF-R1, and the MBDA score for predicting CVD risk. Clinical covariates that might have been expected in the final MBDA-based model, such as sex and hyperlipidemia, were associated with the CVD risk in univariable analyses but were not included because they made small incremental contributions to the multivariable model and did not survive the model building process. Sex was less significant as a univariable predictor of CVD risk than any of the variables that were included in the model (Table 2). It may have been excluded due to co-linearity with other variables, such as tobacco use, which is less common in women with RA than men with RA [4], and leptin, the levels of which tend to be greater in women [60]. It is unlikely that the MBDA score caused sex to be excluded from the model because adjustment of the MBDA score (for age, sex, and leptin) should have reduced its co-linearity with sex. The failure of hyperlipidemia to survive backward elimination may relate to it also having been a less significant univariable predictor of CVD risk than any of the predictors that survived. In addition, the “lipid paradox” [61] may make it difficult to interpret lipid values in RA patients, as they can be lower during active RA and increase with effective treatment. A practical consideration is that many RA patients have not had lipids tested recently, and co-management with primary care physicians may be needed to improve rates of screening for hyperlipidemia [62]. The cohort we used included patients with diabetes or a history of CVD and patients who were receiving statin treatment. Excluding such patients, as some CVD risk calculators do, would have greatly narrowed the utility of the score and reduced the power to see differences in the risk due to other variables. Instead, diabetes and a history of CVD were entered into model building as predictor variables and they were included in the score. Subanalyses demonstrated good fit between predicted and observed CVD events for patients with or without diabetes or a history of CVD. Statin use is not in the MBDA-based CVD risk score because we excluded drug-related variables from model building. However, the risk score demonstrated good fit in subanalyses of patients who were and were not receiving statins. The MBDA-based CVD risk score accounts for the level of inflammation, the treatment of which has potential to reduce CVD risk in RA patients [21–23]. The score may have utility for RA patients who are receiving statins because the statin dose may not yet have been optimized and because the non-statin treatment options for elevated CVD risk in RA patients may include DMARDs. Other RA-specific CVD risk prediction models have been created. The expanded risk scored for CVD in RA (ERS-RA) was derived from a large RA cohort in the USA [19] and has been externally validated [28]. It quantifies RA disease activity categorically with the clinical disease activity index (CDAI) and also includes the Health Assessment Questionnaire (HAQ). A Trans-Atlantic Cardiovascular Risk Consortium for Rheumatoid Arthritis (ATACC-RA) developed two predictors that include serum lipid levels and account for RA disease activity with the 28-joint Disease Activity Score with erythrocyte sedimentation rate (DAS28-ESR) or HAQ, respectively [29]. The MBDA-based CVD risk score requires no clinical measurements and no laboratory data except results from the MBDA test. Rheumatologist preference among these predictors may depend on convenience and on which RA disease activity measures they use most routinely [63, 64]. CVD risk prediction for RA patients could be facilitated in a practical way if a risk score were to be automatically calculated—within an electronic medical record or, in the case of the MBDA-based CVD risk score, when the MBDA score is calculated by the testing laboratory—and provided to the ordering rheumatologist. The large size of this study was made possible by linking administrative data from the Medicare database to a database of existing MBDA test results. The approach we used to capture CVD endpoint components in the Medicare database has a positive predictive value of approximately ≥ 93% for MI and 80–85% for stroke [37–39]. Fatal CVD events were identified using algorithms with positive predictive values ≥ 80% [40]. This study was restricted to patients ≥ 40 years old, to be aligned with the ACC/AHA guidelines [9]. A limitation of having used the Medicare cohort is that it contained predominantly older patients with high rates of CVD risk factors, and most of the 23% of patients < 65 years old were eligible for Medicare because they were disabled. In subanalyses of the patients who were < 65 years old and of patients who had or lacked each of the four clinical risk factors in the model, the MBDA-based CVD risk score had good fit with observed CVD events. In a previous report, CVD risk was relatively similar in younger disabled vs. younger non-disabled RA patients after accounting for the lower prevalence of CVD risk factors [65], suggesting that the MBDA-based CVD risk score may be applicable to patients < 65 years old who are not disabled. However, further validation of the CVD risk score in younger RA patients is needed. Another limitation of our linked cohort is that clinical practice measurements, such as the blood pressure or lipid levels, were not available and the reasons for ordering MBDA tests were not known. Nevertheless, the MBDA-based CVD risk score demonstrated good fit with observed CVD events in patients with hypertension, hyperlipidemia, history of CVD or statin use, and in patients grouped by level of biomarker-based disease activity or according to whether a biologic DMARD treatment had been initiated or changed during follow-up. Because we lacked clinical measurements, the MBDA-based CVD risk score could not be compared with CVD risk predictors that require them, such as the ACC/AHA Pooled Cohort Equation or the Framingham Risk Score. As an alternative, the MBDA-based CVD risk score was compared to four clinical models of increasing complexity, from an age + sex model to a model that included age, sex, four traditional clinical risk factors available in the Medicare database, and CRP. The MBDA-based CVD risk score showed better fit than all four models, based on LRT. It also demonstrated statistically significantly better NRI and a numerically greater C-index. Because likelihood has been considered the most powerful means for comparing CVD risk prediction tests [66], and C-indices can fail to reflect meaningful incremental contributions of CVD-related biomarkers [67], these results suggest that the MBDA-based CVD risk score may be at least comparable to existing CVD risk calculators and potentially more practical for routine use. Direct comparison with other RA-specific calculators and general population CVD risk calculators adjusted for RA would be of interest. The 3-year horizon used here for the composite CVD outcome reflects a constraint from the availability and uptake of the MBDA test for routine clinical practice in the US. Of more scientific relevance, however, is that RA is a dynamic disease and disease activity for many patients will fluctuate, such that a single measurement of disease activity may become less associated with true CVD risk over time. Thus, our shorter, 3-year time horizon may be preferable for predicting CVD risk in patients with RA, in that it is less subject to misclassification of RA disease activity than with the 10-year time horizon used by many existing CVD risk calculators. Indeed, the dynamic nature of RA disease activity and other factors that may be important to assessing CVD risk in RA patients is reflected in the ACC/AHA recommendation that, for adult patients with RA, “it can be useful to recheck lipid values and other major ASCVD (atherosclerotic CVD) risk factors 2 to 4 months after the patient’s inflammatory disease has been controlled [11].” Among all specialists, rheumatologists are likely in the best position to assess treatment response and systemic inflammatory burden in RA patients. The MBDA-based CVD risk score may assist rheumatologists by reminding them of the need for CVD risk management in RA patients—which some may wish to co-manage with a primary care physician or cardiologist—and of the unique role rheumatologists have in treating the inflammatory disease component of CVD risk [13]. Conclusions In conclusion, we have developed and internally validated an MBDA-based CVD risk score that predicts risk for MI, stroke, or fatal CVD in the next 3 years for RA patients. It is novel because it accounts for the contribution of RA inflammatory disease activity by including the MBDA score and three biomarkers that are independently associated with CVD. It performed better than prediction models that used only clinical data. The MBDA-based CVD risk prediction score provides rheumatologists with a feasible tool for assessing CVD risk to inform the management of traditional CVD risk factors and RA inflammation. Further validation with more extended time frames and more heterogeneous cohorts of RA patients will be helpful to assure its robustness as a prediction model. Supplementary information Additional file 1: Supplemental Figure 1. Goodness of fit in patient subgroups (validation dataset, total N=10,275). Supplemental Table 1. Cohort Derivation. Supplemental Table 2. A, Diagnostic codes for candidate variables used to build the MBDA-based CVD risk score and B, Frequencies of CVD-related conditions comprising the History of CVD variable. Supplemental Table 3. Reclassification of patients based on CVD risk predicted by the MBDA-based CVD risk score versus: A, the Age + Sex + CRP model and B, the Clinical model. Supplemental Text. Conversion of the MBDA-based CVD risk score into 3-year percentage risk of a CVD event. Abbreviations ACC/AHAAmerican College of Cardiology and American Heart Association CDAIClinical Disease Activity Index CIConfidence interval CRPC-reactive protein CVDCardiovascular disease DAS28-CRPDisease Activity Score in 28 joints with CRP DAS28-ESRDisease Activity Score in 28 joints with erythrocyte sedimentation rate EULAREuropean League Against Rheumatism FRSFramingham Risk Score LRTLikelihood ratio test MBDAMulti-biomarker disease activity MIMyocardial infarction NRINet Reclassification Index PYPatient-years RARheumatoid arthritis TNFiTNF inhibitor TRIPODTransparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary information Supplementary information accompanies this paper at 10.1186/s13075-020-02355-0. Acknowledgements The authors thank Brooke Hullinger, JD, for her assistance in preparing the figures and tables and editing the manuscript. Authors’ contributions JC and FX had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study conception and design: JC, FX, CSC, ES, RB-S, AG, DF, BM, and JL. Acquisition of data: JC, FX, AG, DF, and BM. Analysis and interpretation of data: JC, FX, CSC, ES, EH, CLC, RB, RB-S, AG, DF, BM, and JL. All authors were involved drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be published. Authors’ information Eric Sasso, M.D., is an Affiliate Professor of Medicine (Rheumatology) at the University of Washington, Seattle, WA., USA. Funding This work was supported by the Myriad Genetics, Inc. Availability of data and materials The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. Ethics approval and consent to participate The University of Alabama at Birmingham institutional review board approved the study. Consent for publication Not applicable Competing interests JC received grants and personal fees from the Abbvie, Amgen, BMS, Corrona, Eli Lilly, Jannsen, Myriad Genetics, Inc., Pfizer, Regeneron, Roche, and UCB during the conduct of the study. FX and CSC received research funding from the Myriad Genetics, Inc., during the conduct of the study. ES, EH, CLC, RB, RB-S, AG, DF, BM, and JL are employed by the Myriad Genetics, Inc., and receive salaries and stock options as compensation. ==== Refs References 1. DeMizio DJ Geraldino-Pardilla LB Autoimmunity and inflammation link to cardiovascular disease risk in rheumatoid arthritis Rheumatol Ther 2020 7 1 19 33 10.1007/s40744-019-00189-0 31853784 2. Aviña-Zubieta JA Choi HK Sadatsafavi M Etminan M Esdaile JM Lacaille D Risk of cardiovascular mortality in patients with rheumatoid arthritis: a meta-analysis of observational studies Arthritis Rheum 2008 59 12 1690 1697 10.1002/art.24092 19035419 3. Ladak K Hashim J Clifford-Rashotte M Tandon V Matsos M Patel A Cardiovascular risk management in rheumatoid arthritis: a large gap to close Musculoskeletal Care 2018 16 1 152 157 10.1002/msc.1196 28417529 4. Crowson CS Rollefstad S Ikdahl E Kitas GD van Riel P Gabriel SE Impact of risk factors associated with cardiovascular outcomes in patients with rheumatoid arthritis Ann Rheum Dis 2018 77 1 48 54 10.1136/annrheumdis-2017-211735 28877868 5. Solomon DH Kremer J Curtis JR Hochberg MC Reed G Tsao P Explaining the cardiovascular risk associated with rheumatoid arthritis: traditional risk factors versus markers of rheumatoid arthritis severity Ann Rheum Dis 2010 69 1920 1925 10.1136/ard.2009.122226 20444756 6. Agca R Heslinga SC Rollefstad S Heslinga M McInnes IB Peters MJ EULAR recommendations for cardiovascular disease risk management in patients with rheumatoid arthritis and other forms of inflammatory joint disorders: 2015/2016 update Ann Rheum Dis 2017 76 1 17 28 10.1136/annrheumdis-2016-209775 27697765 7. Arts EE Popa C Den Broeder AA Semb AG Toms T Kitas GD Performance of four current risk algorithms in predicting cardiovascular events in patients with early rheumatoid arthritis Ann Rheum Dis 2015 74 4 668 674 10.1136/annrheumdis-2013-204024 24389293 8. Alemao E Cawston H Bourhis F Al M Rutten-van Molken M Liao KP Comparison of cardiovascular risk algorithms in patients with vs without rheumatoid arthritis and the role of C-reactive protein in predicting cardiovascular outcomes in rheumatoid arthritis Rheumatology (Oxford) 2017 56 5 777 786 28087832 9. Stone NJ, Robinson JG, Lichtenstein AH, Bairey Merz CN, Blum CB, Eckel RH, et al. 2013 ACC/AHA guideline on the treatment of blood cholesterol to reduce atherosclerotic cardiovascular risk in adults: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2014;63(25 Pt B):2889-2934. 10. Crowson CS Liao KP Davis JM 3rd Solomon DH Matteson EL Knutson KL Rheumatoid arthritis and cardiovascular disease Am Heart J 2013 166 4 622 628 10.1016/j.ahj.2013.07.010 24093840 11. Grundy SM Stone NJ Bailey AL Beam C Birtcher KK Blumenthal RS 2018 AHA/ACC/AACVPR/AAPA/ABC/ACPM/ADA/AGS/APhA/ASPC/NLA/PCNA Guideline on the Management of Blood Cholesterol: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines Circulation. 2019 139 25 e1082 ee143 30586774 12. Goff DC, Jr., Lloyd-Jones DM, Bennett G, Coady S, D'Agostino RB, Sr., Gibbons R, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2014;63(25 Pt B):2935-2959. 13. Arnett DK Blumenthal RS Albert MA Buroker AB Goldberger ZD Hahn EJ 2019 ACC/AHA guideline on the primary prevention of cardiovascular disease: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines Circulation. 2019 140 11 e596 e646 30879355 14. Ridker PM Everett BM Thuren T MacFadyen JG Chang WH Ballantyne C Antiinflammatory therapy with Canakinumab for atherosclerotic disease N Engl J Med 2017 377 12 1119 1131 10.1056/NEJMoa1707914 28845751 15. Ridker PM MacFadyen JG Everett BM Libby P Thuren T Glynn RJ Relationship of C-reactive protein reduction to cardiovascular event reduction following treatment with canakinumab: a secondary analysis from the CANTOS randomised controlled trial Lancet. 2018 391 10118 319 328 10.1016/S0140-6736(17)32814-3 29146124 16. Curtis JR Xie F Chen L Saag KG Yun H Muntner P Biomarker-related risk for myocardial infarction and serious infections in patients with rheumatoid arthritis: a population-based study Ann Rheum Dis 2018 77 3 386 392 10.1136/annrheumdis-2017-211727 29269418 17. Arts EE Fransen J den Broeder AA Popa CD van Riel PL The effect of disease duration and disease activity on the risk of cardiovascular disease in rheumatoid arthritis patients Ann Rheum Dis 2015 74 6 998 1003 10.1136/annrheumdis-2013-204531 24458537 18. Mantel A Holmqvist M Nyberg F Tornling G Frisell T Alfredsson L Risk factors for the rapid increase in risk of acute coronary events in patients with new-onset rheumatoid arthritis: a nested case-control study Arthritis Rheumatol. 2015 67 11 2845 2854 10.1002/art.39267 26138387 19. Solomon DH Greenberg J Curtis JR Liu M Farkouh ME Tsao P Derivation and internal validation of an expanded cardiovascular risk prediction score for rheumatoid arthritis: a consortium of rheumatology researchers of North America Registry Study Arthritis Rheumatol 2015 67 8 1995 2003 10.1002/art.39195 25989470 20. Solomon DH Reed GW Kremer JM Curtis JR Farkouh ME Harrold LR Disease activity in rheumatoid arthritis and the risk of cardiovascular events Arthritis Rheumatol. 2015 67 6 1449 1455 10.1002/art.39098 25776112 21. Karpouzas GA, Ormseth SR, Hernandez E, Budoff MJ. Biologics may prevent cardiovascular events in rheumatoid arthritis by inhibiting coronary plaque formation and stabilizing high-risk lesions. Arthritis Rheumatol.n/a(n/a). 22. Micha R, Imamura F, Wyler von Ballmoos M, Solomon DH, Hernan MA, Ridker PM, et al. Systematic review and meta-analysis of methotrexate use and risk of cardiovascular disease. Am J Cardiol 2011;108(9):1362–1370. 23. Roubille C Richer V Starnino T McCourt C McFarlane A Fleming P The effects of tumour necrosis factor inhibitors, methotrexate, non-steroidal anti-inflammatory drugs and corticosteroids on cardiovascular events in rheumatoid arthritis, psoriasis and psoriatic arthritis: a systematic review and meta-analysis Ann Rheum Dis 2015 74 3 480 489 10.1136/annrheumdis-2014-206624 25561362 24. England BR Thiele GM Anderson DR Mikuls TR Increased cardiovascular risk in rheumatoid arthritis: mechanisms and implications BMJ. 2018 361 k1036 10.1136/bmj.k1036 29685876 25. Su CC Chen Ie C Young FN Lian IB Risk of diabetes in patients with rheumatoid arthritis: a 12-year retrospective cohort study J Rheumatol 2013 40 9 1513 1518 10.3899/jrheum.121259 23818711 26. Kozera L Andrews J Morgan AW Cardiovascular risk and rheumatoid arthritis--the next step: differentiating true soluble biomarkers of cardiovascular risk from surrogate measures of inflammation Rheumatology (Oxford) 2011 50 11 1944 1954 10.1093/rheumatology/ker232 21752870 27. Kay J Morgacheva O Messing SP Kremer JM Greenberg JD Reed GW Clinical disease activity and acute phase reactant levels are discordant among patients with active rheumatoid arthritis: acute phase reactant levels contribute separately to predicting outcome at one year Arthritis Res Therapy 2014 16 1 R40 10.1186/ar4469 28. Ljung L Ueda P Liao KP Greenberg JD Etzel CJ Solomon DH Performance of the expanded cardiovascular risk prediction score for rheumatoid arthritis in a geographically distant national register-based cohort: an external validation RMD Open 2018 4 2 e000771 10.1136/rmdopen-2018-000771 30622736 29. Crowson CS Rollefstad S Kitas GD van Riel PL Gabriel SE Semb AG Challenges of developing a cardiovascular risk calculator for patients with rheumatoid arthritis PLoS One 2017 12 3 e0174656 10.1371/journal.pone.0174656 28334012 30. Curtis JR van der Helm-van Mil AH Knevel R Huizinga TW Haney DJ Shen Y Validation of a novel multibiomarker test to assess rheumatoid arthritis disease activity Arthritis Care Res (Hoboken) 2012 64 12 1794 1803 10.1002/acr.21767 22736476 31. England BR Tiong BK Bergman MJ Curtis JR Kazi S Mikuls TR 2019 Update of the American College of Rheumatology recommended rheumatoid arthritis disease activity measures Arthritis Care Res (Hoboken). 2019 71 12 1540 1555 10.1002/acr.24042 31709779 32. Curtis JR Brahe CH Ostergaard M Lund Hetland M Hambardzumyan K Saevarsdottir S Predicting risk for radiographic damage in rheumatoid arthritis: comparative analysis of the multi-biomarker disease activity score and conventional measures of disease activity in multiple studies Curr Med Res Opin 2019 35 9 1483 1493 10.1080/03007995.2019.1585064 30777458 33. van der Helm-van Mil AH, Knevel R, Cavet G, Huizinga TW, Haney DJ. An evaluation of molecular and clinical remission in rheumatoid arthritis by assessing radiographic progression. Rheumatology (Oxford). 2013;52(5):839–46. 34. Curtis JR Chen L Bharat A Delzell E Greenberg JD Harrold L Linkage of a de-identified United States rheumatoid arthritis registry with administrative data to facilitate comparative effectiveness research Arthritis Care Res (Hoboken). 2014 66 12 1790 1798 10.1002/acr.22377 24905637 35. Kim SY Servi A Polinski JM Mogun H Weinblatt ME Katz JN Validation of rheumatoid arthritis diagnoses in health care utilization data Arthritis Res Ther 2011 13 1 R32 10.1186/ar3260 21345216 36. Reiss WG Devenport JN Low JM Wu G Sasso EH Interpreting the multi-biomarker disease activity score in the context of tocilizumab treatment for patients with rheumatoid arthritis Rheumatol Int 2016 36 2 295 300 10.1007/s00296-015-3285-2 26026604 37. McCormick N Lacaille D Bhole V Avina-Zubieta JA Validity of myocardial infarction diagnoses in administrative databases: a systematic review PLoS One 2014 9 3 e92286 10.1371/journal.pone.0092286 24682186 38. Birman-Deych E Waterman AD Yan Y Nilasena DS Radford MJ Gage BF Accuracy of ICD-9-CM codes for identifying cardiovascular and stroke risk factors Med Care 2005 43 5 480 485 10.1097/01.mlr.0000160417.39497.a9 15838413 39. Tirschwell DL Longstreth WT Jr Validating administrative data in stroke research Stroke. 2002 33 10 2465 2470 10.1161/01.STR.0000032240.28636.BD 12364739 40. Xie F Colantonio LD Curtis JR Kilgore ML Levitan EB Monda KL Development of algorithms for identifying fatal cardiovascular disease in Medicare claims Pharmacoepidemiol Drug Saf 2018 27 7 740 750 10.1002/pds.4421 29537120 41. Centola M Cavet G Shen Y Ramanujan S Knowlton N Swan KA Development of a multi-biomarker disease activity test for rheumatoid arthritis PLoS One 2013 8 4 e60635 10.1371/journal.pone.0060635 23585841 42. Curtis JR Flake DD Weinblatt ME Shadick NA Ostergaard M Hetland ML Adjustment of the multi-biomarker disease activity score to account for age, sex and adiposity in patients with rheumatoid arthritis Rheumatology (Oxford) 2019 58 5 874 883 10.1093/rheumatology/key367 30590790 43. Collins GS Reitsma JB Altman DG Moons KG Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): the TRIPOD statement Br J Surg 2015 102 3 148 158 10.1002/bjs.9736 25627261 44. Karlik B Olgak A Performance analysis of various activation functions in generalized MLP architectures of neural networks Int J Artificial Intelligence Expert Systems 2010 4 111 122 45. Therneau TM, Grambsch PM. Modeling survival data: extending the Cox model. Gail M, Samet JM, editors: Springer; 2000. 46. Therneau TM, Lumley T, Atkinson E, Cynthia C. A package for survival analysis in R. R package version 3.1–11.: https://CRAN.R-project.org/package=survival. Accessed 25 Mar 2020; 2020. 47. Breslow N Discussion on professor Cox’s paper J R Stat Soc Ser B Methodol 1972 34 2 216 217 48. Demler OV Paynter NP Cook NR Tests of calibration and goodness-of-fit in the survival setting Stat Med 2015 34 10 1659 1680 10.1002/sim.6428 25684707 49. Pencina MJ D'Agostino RB Sr Steyerberg EW Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers Stat Med 2011 30 1 11 21 10.1002/sim.4085 21204120 50. Leening MJ Vedder MM Witteman JC Pencina MJ Steyerberg EW Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide Ann Intern Med 2014 160 2 122 131 10.7326/M13-1522 24592497 51. Uno H Cai T Pencina MJ D'Agostino RB Wei LJ On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data Stat Med 2011 30 10 1105 1117 10.1002/sim.4154 21484848 52. Mogensen UB Ishwaran H Gerds TA Evaluating random forests for survival analysis using prediction error curves J Stat Softw 2012 50 11 1 23 10.18637/jss.v050.i11 25317082 53. Guizani I Zidi W Zayani Y Boudiche S Hadj-Taieb S Sanhaji H Matrix metalloproteinase-3 predicts clinical cardiovascular outcomes in patients with coronary artery disease: a 5 years cohort study Mol Biol Rep 2019 46 5 4699 4707 10.1007/s11033-019-04914-4 31218540 54. Li X Zhang F Zhou H Hu Y Guo D Fang X Interplay of TNF-alpha, soluble TNF receptors and oxidative stress in coronary chronic total occlusion of the oldest patients with coronary heart disease Cytokine. 2020 125 154836 10.1016/j.cyto.2019.154836 31494339 55. Valgimigli M Ceconi C Malagutti P Merli E Soukhomovskaia O Francolini G Tumor necrosis factor-ALPHA receptor 1 is a major predictor of mortality and new-onset heart failure in patients with acute myocardial infarction: the Cytokine-Activation and Long-Term Prognosis in Myocardial Infarction (C-ALPHA) study Circulation. 2005 111 7 863 870 10.1161/01.CIR.0000155614.35441.69 15699251 56. Escalante A Haas RW del Rincon I Paradoxical effect of body mass index on survival in rheumatoid arthritis: role of comorbidity and systemic inflammation Arch Intern Med 2005 165 14 1624 1629 10.1001/archinte.165.14.1624 16043681 57. Khan SS Ning H Wilkins JT Allen N Carnethon M Berry JD Association of body mass index with lifetime risk of cardiovascular disease and compression of morbidity JAMA Cardiol 2018 3 4 280 287 10.1001/jamacardio.2018.0022 29490333 58. Baker JF Billig E Michaud K Ibrahim S Caplan L Cannon GW Weight loss, the obesity paradox, and the risk of death in rheumatoid arthritis Arthritis Rheumatol. 2015 67 7 1711 1717 10.1002/art.39136 25940140 59. Choi HK Nguyen U-S Niu J Danaei G Zhang Y Selection bias in rheumatic disease research Nat Rev Rheumatol 2014 10 7 403 412 10.1038/nrrheum.2014.36 24686510 60. Saad MF Damani S Gingerich RL Riad-Gabriel MG Khan A Boyadjian R Sexual dimorphism in plasma leptin concentration* J Clin Endocrinol Metabolism 1997 82 2 579 584 61. Myasoedova E Crowson CS Kremers HM Roger VL Fitz-Gibbon PD Therneau TM Lipid paradox in rheumatoid arthritis: the impact of serum lipid measures and systemic inflammation on the risk of cardiovascular disease Ann Rheum Dis 2011 70 3 482 487 10.1136/ard.2010.135871 21216812 62. Navarro-Millan I Yang S Chen L Yun H Jagpal A Bartels CM Screening of hyperlipidemia among patients with rheumatoid arthritis in the United States Arthritis Care Res (Hoboken). 2019 71 12 1593 1599 10.1002/acr.23810 30414353 63. Yun H Chen L Xie F Patel H Boytsov N Zhang X Do patients with moderate or high disease activity escalate rheumatoid arthritis therapy according to treat-to-target principles? Results from the rheumatology informatics system for effectiveness registry of the American College of Rheumatology Arthritis Care Res (Hoboken). 2020 72 2 166 175 10.1002/acr.24083 31566905 64. Curtis JR Xie F Yang S Danila MI Owensby JK Chen L Uptake and clinical utility of multibiomarker disease activity testing in the United States J Rheumatol 2019 46 3 237 244 10.3899/jrheum.180071 30442830 65. Xie F, Crowson C, Navarro-Millan I, Safford M, Curtis JR. Comparing the generalizability of cardiovascular risk in different rheumatoid arthritis cohorts [abstract]. 2019 ACR/ARP annual meeting; November 10, 2019; Atlanta: Arthritis Rheumatol; 2019. 66. Cook NR Quantifying the added value of new biomarkers: how and how not Diagnostic and Prognostic Research 2018 2 1 14 10.1186/s41512-018-0037-2 31093563 67. Cook NR Use and misuse of the receiver operating characteristic curve in risk prediction Circulation. 2007 115 7 928 935 10.1161/CIRCULATIONAHA.106.672402 17309939