==== Front
Bioinformatics
Bioinformatics
bioinformatics
Bioinformatics
1367-4803
1367-4811
Oxford University Press

37387152
10.1093/bioinformatics/btad204
btad204
Biomedical Informatics
AcademicSubjects/SCI01060
COmic: convolutional kernel networks for interpretable end-to-end learning on (multi-)omics data
Ditz Jonas C Methods in Medical Informatics, Department of Computer Science, University of Tübingen, Tübingen 72076, Germany

Reuter Bernhard Methods in Medical Informatics, Department of Computer Science, University of Tübingen, Tübingen 72076, Germany

Pfeifer Nico Methods in Medical Informatics, Department of Computer Science, University of Tübingen, Tübingen 72076, Germany

Corresponding author. Methods in Medical Informatics, Department of Computer Science, University of Tübingen, Sand 14, Tübingen 72076, Germany. E-mail: bernhard.reuter@uni-tuebingen.de (B.R.), nico.pfeifer@uni-tuebingen.de (N.P.)
6 2023
30 6 2023
30 6 2023
39 Suppl 1 ISMB/ECCB 2023 Proceedings i76i85
© The Author(s) 2023. Published by Oxford University Press.
2023
https://creativecommons.org/licenses/by/4.0/ This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Motivation

The size of available omics datasets is steadily increasing with technological advancement in recent years. While this increase in sample size can be used to improve the performance of relevant prediction tasks in healthcare, models that are optimized for large datasets usually operate as black boxes. In high-stakes scenarios, like healthcare, using a black-box model poses safety and security issues. Without an explanation about molecular factors and phenotypes that affected the prediction, healthcare providers are left with no choice but to blindly trust the models. We propose a new type of artificial neural network, named Convolutional Omics Kernel Network (COmic). By combining convolutional kernel networks with pathway-induced kernels, our method enables robust and interpretable end-to-end learning on omics datasets ranging in size from a few hundred to several hundreds of thousands of samples. Furthermore, COmic can be easily adapted to utilize multiomics data.

Results

We evaluated the performance capabilities of COmic on six different breast cancer cohorts. Additionally, we trained COmic models on multiomics data using the METABRIC cohort. Our models performed either better or similar to competitors on both tasks. We show how the use of pathway-induced Laplacian kernels opens the black-box nature of neural networks and results in intrinsically interpretable models that eliminate the need for post hoc explanation models.

Availability and implementation

Datasets, labels, and pathway-induced graph Laplacians used for the single-omics tasks can be downloaded at https://ibm.ent.box.com/s/ac2ilhyn7xjj27r0xiwtom4crccuobst/folder/48027287036. While datasets and graph Laplacians for the METABRIC cohort can be downloaded from the above mentioned repository, the labels have to be downloaded from cBioPortal at https://www.cbioportal.org/study/clinicalData?id=brca\_metabric. COmic source code as well as all scripts necessary to reproduce the experiments and analysis are publicly available at https://github.com/jditz/comics.

Deutsche Forschungsgemeinschaft 10.13039/501100001659 German Research Foundation 10.13039/501100001659 390727645 German Federal Ministry of Education and Research Training Center Machine Learning, Tübingen 01-S17054 German Federal Ministry of Education and Research Tübingen AI Center 01IS18039A
==== Body
pmc1 Introduction

In recent years, artificial neural networks (ANNs) show promising performance when employed to learn correlations between data points and outcome variables. They combine feature extraction and prediction training in a single end-to-end learning scheme lowering the necessary amount of labor put into feature engineering and can be used on very large datasets with relative ease. With the advent of big data and high-throughput data generation techniques in computational biology and healthcare, resulting in an increased number of data points available for the training of prediction models, the use of ANNs in these fields has vastly increased. In computational biology, ANNs showed promising performance capabilities when applied to prediction tasks in regulatory genomics (Alipanahi et al. 2015; Zhou and Troyanskaya 2015; Kelley et al. 2016) and in biological image analysis (Ronneberger et al. 2015; Ferrari et al. 2017; Pärnamaa and Parts 2017). Furthermore, several authors showed the potential of ANNs in healthcare scenarios such as diagnosis (De Fauw et al. 2018; Arieno et al. 2019), drug discovery (Fleming 2018; Álvarez-Machancoses and Fernández-Martínez 2019), epidemiology (Hay et al. 2013), personalized medicine (Miotto et al. 2016), and operational efficiency (Nelson et al. 2019). However, utilizing ANNs for prediction tasks usually comes with two shortcomings: First, a large amount of data is needed to robustly train a deep neural network and, second, neural network models operate as black-boxes. While the first problem can be tackled by shrinking the complexity of the neural network, which increases the stability of the model but often leads to a decrease in performance, the second shortcoming is most often addressed using post hoc interpretation models. This technique involves solving a secondary task that utilizes a pretrained prediction model such that the computed solution provides a humanly understandable interpretation for the results computed by the prediction model. Commonly used methods include Shapley additive explanation (Lundberg and Lee 2017), counterfactual explanation using generative models (Stepin et al. 2021), and saliency methods like Layer-wise Relevance Propagation (Montavon et al. 2019), Deep Taylor Decomposition (Montavon et al. 2017), GuidedBP (Springenberg et al. 2014), or DeepLIFT (Shrikumar et al. 2017). Using post hoc interpretation methods can provide additional information to improve understanding and advance scientific knowledge in low-risk scenarios but they have several properties that render their use in high-risk scenarios potentially problematic. Most post hoc interpretation methods are unfaithful to the computations of the original model (Rudin 2019). Furthermore, many saliency methods ignore information provided by deeper layers of ANNs (Sixt et al. 2020). Recent work showed that post hoc methods are limited in adversarial contexts (Bordt et al. 2022) and can be exploited to provide seemingly plausible but misleading explanations (Lipton 2018). In healthcare, decisions that are made based on wrong or misleading explanations have the potential to cause harm to patients.

Kernel methods can provide both robustness on small datasets and interpretation capabilities within the domain of the data. These methods utilize the kernel trick to solve a prediction task by implicitly projecting data into the reproducing kernel Hilbert space (RKHS) of a kernel function and solve the classification or regression problem within the RKHS. While the use of a kernel functions does not always guarantee interpretation capabilities, there are several kernel functions for biological data that result in interpretable models, e.g. the oligo kernel for sequences (Meinicke et al. 2004) or the pathway-induced kernel for omics data (Manica et al. 2019). Combining kernel functions with ANNs is a promising direction to increase the robustness of ANN models on small datasets and several efforts in that direction have been published in recent years (Cho and Saul 2009; Bo et al. 2011; Mairal et al. 2014; Mairal 2016). Chen et al. showed the feasibility of kernel networks for biological sequences by using a relaxation of the mismatch kernel (Eskin et al. 2003) to build convolutional and recurrent neural network architectures (Chen et al. 2019a,b). Furthermore, they showed how to use convolutional kernel neural networks on graph-structured data like protein structures (Chen et al. 2020). While these models showed promising results and increased robustness, the choice of the kernel function resulted in models that are not intrinsically interpretable. However, we recently showed that a carefully chosen kernel function results in intrinsically interpretable kernel networks for biological sequence data (Ditz et al. 2021). With this work, we introduce Convolutional Omics Kernel Networks (COmic), a neural network architecure that allows for intrinsically interpretable end-to-end learning on (multi-)omics data. This is achieved by using a kernel function based on graph Laplacians of biological networks to project input samples into a subspace of the corresponding RKHS with a variant of the Nyström method. Using max-pooling combined with strictly linear layers for classification results in COmic models that provide global interpretation, while attention layers can be used to create COmic models that provide local interpretation. In this manuscript, we use the definition most commonly found in the interpretable machine learning literature for global and local interpretation (Molnar 2020). In simple words, “global interpretation” can be used to answer the question “How does the trained model make predictions?”, while “local interpretation” can be used to answer the question “Why did the model make a certain prediction for a specific input?”.

We show the performance and interpretation capabilities of COmic models on six different breast cancer microarray cohorts. These cohorts contain microarray gene expression data from patients with breast cancer and were stratified on the occurrence of a relapse within 5 years. We compare our proposed method to 15 previously published approaches including several methods based on support vector machines (SVMs) like network-based SVMs (Zhu et al. 2009), recursive feature elimination SVMs (Guyon et al. 2002), and graph diffusion kernels for SVMs (Rapaport et al. 2007; Gao et al. 2009) as well as classification by average pathway expression (Guo et al. 2005), classification by significant hub genes (Taylor et al. 2009), classification by pathway activity (Lee et al. 2008), and pathway-induced multiple kernel learning (PIMKL, Manica et al. 2019). We show how the projection into a subspace of the RKHS of pathway-induced kernels in combination with linear and attention layers leads to global and local interpretations, respectively. Furthermore, we use the METABRIC cohort (Curtis et al. 2012) to show how COmic models can be used on multiomics data. On the METABRIC breast cancer cohort, we predicted disease-free survival using gene expression (mRNA) and copy number alteration (CNA) data.

This work is structured as follows. We first introduce COmic by describing the pathway-induced kernel and define the necessary network architecture to build a COmic model. Afterwards, we show how to achieve a globally interpretable COmic model using strictly linear layers and a locally interpretable COmic model using attention layers. We evaluate COmic models on six breast cancer cohorts and show how COmic models can be utilized for multiomics data. With this manuscript, we introduce a new kernel network architecture that can be both robustly trained on small-scale (multi-)omics datasets and easily utilized for prediction tasks on (multi-)omics datasets with several hundreds of thousands of data points. Furthermore, our method results in intrinsically interpretable models offering global and local interpretations of prediction results.

2 Convolutional omics kernel networks

In the following section, we describe the theoretical background of convolutional kernel networks for prediction tasks on omics-based datasets.

2.1 Pathway-induced kernel functions

The foundation of pathway-induced kernel functions is so-called graph Laplacian matrices. To define these matrices we first assume G=(V,E) to be an undirected graph with vertices V={v1,…,vn} and edges E. Furthermore, G is assumed to be a weighted graph with weight matrix W∈Rn×n, where wij=wji≥0 describes the weight of the edge between vertices vi and vj. The degree of each vertex vi∈V is defined as di=∑j=1nwij. The diagonal matrix with the degrees d1,…,dn on the diagonal is called the degree matrix D. The unnormalized graph Laplacian L∈Rn×n is defined as (Von Luxburg 2007):

(1) L:=D−W.

Since an unnormalized graph Laplacian has undesirable mathematical properties in case of very broadly distributed degrees within G (Von Luxburg 2007), we use a normalized graph Laplacian instead, which is defined as:

(2) Lsym:=D−12LD−12.

Similar to previous manuscripts (see e.g. Chen et al. 2011; Manica et al. 2019; ), we use molecular interaction networks as graphs underlying the normalized graph Laplacians. Using known interaction networks allows to define a kernel function that computes the similarity of molecular measures (gene expression, DNA methylation, etc.) under the assumed interactions defined by the network. Given two molecular measures xi∈Rn and xj∈Rn, we define the kernel function as where LMIN is the normalized graph Laplacian [as defined in Equation (2)] of a molecular interaction network.

(3) KMIN(xi,xj)=xiTLMINxj,

Manica et al. (2019) proposed to use pathway-specific sub-networks instead of whole interaction networks for computing normalized graph Laplacians. This method allows for a more tailored induction of prior knowledge into a prediction task. The authors call this approach pathway-induced (PI) kernel functions. Here, the similarity between two molecular measures is not computed using a single graph Laplacian but with a set of p graph Laplacians L={LPI1,…,LPIp} each defined over a pathway-specific sub-network of the molecular interaction network. Therefore the pathway-induced kernel for two molecular measures xi∈Rn and xj∈Rn is not a single function but a set of functions defined as with where LPIr∈Rd×d is the symmetric graph Laplacian of the rth pathway-specific subnetwork with d nodes and xi,PIr∈Rd and xj,PIr∈Rd are vectors containing only the signal values of the molecular measures that correspond to nodes within the pathway-specific subnetwork described by LPIr. Manica et al. used multiple kernel learning (MKL) to combine the set of pathway-induced kernel functions into a single learning framework. In contrast to prior work, we are using a variant of the Nyström method to formulate an explicit parametrization of an orthogonal projection onto a finite-dimensional subspace of a pathway-induced kernel’s RKHS. This enables us to define pathway-induced kernel layers that can be incorporated into ANNs. By using a neural network architecture as the basis for COmic, the resulting models can be tailored to specific datasets (single- or multiomics) as well as the desired form of interpretation. We will show that in the following sections.

(4) KPI(xi,xj)={KPI1(xi,PI1,xj,PI1),…,KPIp(xi,PIp,xj,PIp)},

(5) KPIr(xi,PIr,xj,PIr)=xi,PIrTLPIrxj,PIr,

2.2 Convolutional kernel layer projects onto a finite-dimensional RKHS-subspace

Convolutional kernel networks make use of a variant of the Nyström method to project input samples into a finite-dimensional subspace of the RKHS H of a kernel function. To achieve this, a set of q anchor points z1,…,zq is used to define a q-dimensional subspace E of H. The anchor points lie in the input space of the kernel function and the RKHS subspace is defined as where ϕ(zi) denotes the image of the ith anchor point under the kernel function. The orthogonal projection of input points onto E admits an explicit parametrization that utilizes the kernel trick to avoid explicitly calculating the images ϕ(zi) (Williams and Seeger 2001; Zhang et al. 2008; Mairal 2016). For an input x, i.e. a molecular measure in case of omics data, the explicit parametrization ψ(x)∈Rq is defined as where KZZ=(K(zi,zj))i=1,…,q;j=1,…,q is the gram matrix formed by the anchor points, KZZ−12 denotes the (pseudo-)inverse square root of the Gram matrix, and KZ(x)=[K(x,z1),…,K(x,zp)]T. As shown in Fig. 1, each pathway-induced kernel function has to be modeled with a separate orthogonal projection. This means that a COmic model utilizing p pathway-induced kernel functions maps each input onto p representations ψPI1,…,ψPIp∈Rq. These representations are then used to solve the prediction task for the input. In the next section, we show two different approaches to combine the representations leading to globally or locally interpretable models, respectively.

(6) E=Span(ϕz1,…,ϕzq),

(7) ψ(x)=KZZ−12KZ(x),

Figure 1. Schematic of the proposed interpretable COmic models. Top: Pooling-based COmic model. The kernel embedding of each involved pathway are reduced to a single dimension by using a 1D max pooling operation. The output of each pooling layer is concatenated and prediction is performed using a strictly linear fully-connected layer. The pooling-based COmic models are globally interpretable similar to PIMKL models by utilizing the weights of the fully-connected layer. Bottom: Attention-based COmic model. The kernel embeddings of each involved pathway are transformed into a bag of instances of a multiple instance learning problem. Attention weights for each instance are calculated using each pathway’s kernel embedding and a matrix multiplication between the bag of instances and the attention weights is performed. The output is used for prediction with a strictly linear fully-connected layer. The attention-based COmic models can be locally interpreted by utilizing the attention weights.

The anchor points can be initialized using k-means on all input samples with the number of clusters set to the number of anchor points. Afterwards, the anchor points are optimized with the end-to-end learning scheme used to train the whole network. For all experiments described in this manuscript, anchor points were initialized using k-means++ (Vassilvitskii and Arthur 2006).

2.3 Globally and locally interpretable COmic models

Globally interpretable COmic models are based on multiple kernel learning (MKL). A simple approach to MKL is finding an optimal linear combination of all utilized kernels. This approach learns a weight for each kernel and, therefore, can be used to determine the influence each kernel has on the prediction outcome. Since kernels are directly associated with pathways in PIMKL, Manica et al. (2019) show that MKL weights can be used to determine the importance of different pathways for a prediction. We can embed a similar weighted sum of pathway-induced kernels into the architecture of COmic models. Each kernel embedding produced by the PI-kernel layer described in Section 2.2 is passed into a 1D max pooling layer. This results in a single activation Ar=max(ψPIr) for each pathway-induced kernel, where max(ψPIr) denotes the maximum value in vector ψPIr. This activation is high, if the input is similar to one of the learned anchor points, and low otherwise. By concatenating all activations and passing them into a strictly linear fully-connected layer, the model learns a single weight for each pathway-induced kernel and the prediction is calculated as a weighted sum of all kernels, i.e. where y^ is the overall prediction, wr∈R is the weight and Ar∈R is the activation of the rth pathway-induced kernel. We call this architecture pooling-based COmic model. In contrast to the MKL approach, the weights can become negative. This enhances the interpretation capabilities of pooling-based COmic models, since we cannot only infer if a pathway is important for the prediction task but also with which class each pathway is associated by looking at the sign of the weight. The top part of Fig. 1 shows a schematic of a pooling-based COmic model.

(8) y^=∑r=1pwrAr,

Locally interpretable COmic models are based on multiple instance learning (MIL). In MIL, each sample is represented as a bag of instances with a single label per bag (Dietterich et al. 1997; Maron and Lozano-Pérez 1997; Oquab et al. 2014). There are two general approaches to solve an MIL problem: the instance-level approach and the embedding-level approach. In the instance-level approach, an instance-level classifier predicts a score for each of the instances in the bag. Afterwards, scores are aggregated by MIL pooling to compute the prediction for the bag. In the embedding-level approach, a low-dimensional embedding of each instance is computed and MIL pooling is used on the embedded instances to create a bag representation. This representation is used by a bag-level classifier to provide the prediction. While it was shown that the embedding-level approach leads to better performances (Wang et al. 2018), the instance-level approach leads to interpretable models (Liu et al. 2012). Ilse et al. (2018) proposed an MIL-model based on neural networks that combines the strength of both approaches, called attention-based MIL. Their approach can be utilized for COmic models in the following way. The output of our proposed PI-kernel layer can be viewed as a bag of low-dimensional instances H={ψPI1,…,ψPIp}, where each ψPIr∈Rq is the projection onto a q-dimensional subspace of the RKHS of one pathway-induced kernel. Attention-based MIL pooling is then used to compute the bag representation, i.e. where w∈Rl×1 and V∈Rl×m are parameters of the attention layer. As noticed by Ilse et al., the tanh(⋅) non-linearity introduces a potential limitation due to the fact that it is roughly linear only for x∈[−1,1]. This limitation can be reduced by using a gating mechanism (Dauphin et al. 2017). In this case, the attention weights are calculated as

(9) ψ˜=∑r=1parψPIr,

(10) ar= exp {wTtanh(VψPIrT)}∑j=1p exp {wTtanh(VψPIjT)}.

(11) ar= exp {wT(tanh(VψPIrT)⊙sigm(UψPIrT))}∑j=1p exp {wT(tanh(VψPIjT)⊙sigm(UψPIjT)}.

Again, w∈Rl×1, V∈Rl×m, and U∈Rl×m are parameters of the attention layer. In both cases, the training of all attention layer parameters is part of the end-to-end training routine for the whole network and, hence, does not introduce the need for additional measures. We call this architecture attention-based COmic model. Since the attention weights ap are input specific, they enhance a model with local interpretation capabilities. The bottom part of Fig. 1 shows a schematic of an attention-based COmic model.

3 Experiments on cancer benchmark data

To assess the performance and interpretation capabilities of our COmic models we use publicly available cancer benchmarks. The evaluation involves tasks on single-omics data as well as multiomics data.

3.1 Single-omics prediction on breast cancer benchmark cohorts

We trained COmic models on six different public breast cancer Affymetrix HGU133A microarray datasets (GSE11121, GSE1456, GSE2034, GSE2990, GSE4922, and GSE7390) that were previously used to benchmark knowledge-based classification methods that use interaction network priors. The task was to predict for each patient if distant metastasis-free survival (DMFS) or relapse-free survival (RFS) exceeded 5 years. On GSE11121 and GSE4922, the endpoint was DMFS, while RFS was considered for all other cohorts. Details about the datasets can be found in Table 1. Both, pooling-based and attention-based COmic models, used 50 different pathways to build kernel layers with 30 anchor points each. We used the Laplacians derived from a merge between KEGG pathways and Pathway Commons that were publicly released by Manica et al. (2019) (see original manuscript and corresponding Supplementary Material for details). Furthermore, we used gated attention together with an attention dimension of 128 for the attention-based COmic models. Networks were trained for 200 epochs with the Adam optimizer (Kingma and Ba 2014) using the class-balanced loss function (Cui et al. 2019). The batch size was set to 32. All models presented in this work were trained on a single NVIDIA GeForce GTX 1080 Ti. We used the area under the receiver operating characteristic (auROC) as our performance measure to be comparable to previously published results on the benchmarks. Competitors’ performances shown in Fig. 2A are taken from (Manica et al. 2019), for the PIMKL model, and (Cun and Fröhlich 2012), for all other competitors.

Figure 2. (A) Cross-validation performance of COmic models compared to previously published methods. The boxplots show the 10 mean auROC validation scores of a 10-times repeated 10-fold cross-validation over each of the 6 breast cancer cohorts. Performance of COmic models is shown in orange (last two boxes on the right side). The center line of each box indicates the median. The height of the boxes represents the interquartile range (IQR) with the upper and lower whiskers set to 1.5 times the IQR. Outliers are depicted by black diamonds. Notches represent the confidence interval (CI) around the median and were calculated using bootstrapping with 10 000 iterations. (B) Visualizing the global interpretation capabilities of a pooling-based COmic model. Each box represents one of the 50 pathways and was created using the pathway weights of the models trained on the six publicly available breast cancer cohorts: GSE11121, GSE1456, GSE2034, GSE2990, GSE4922, and GSE7390. The boxplots are defined as in (A) but without notches (CIs not shown). (C) Visualizing the local interpretation capabilities of an attention-based COmic model. Each heat-map shows the attention weights for each of the 50 pathways for three different patients. The model was trained on the GSE11121 cohort. Patient 1 was correctly classified to have a metastasis-free survival (DMFS) above 5 years. Patient 2 was correctly classified to have a DMFS below 5 years. Patient 3 was wrongly classified to have a DMFS above 5 years, while the DMFS of Patient 3 was actually below 5 years. More examples can be found in the supplement. (D) Mean training time of pooling-based and attention-based COmic models for differently sized datasets. The number of samples is 100, 1000, 10 000, and 100 000, respectively. Training was repeated five times per dataset and the stars represent the mean training time. The blue and green lines show the results for a fixed batch size of 32 samples per batch. The red and yellow lines show the results for an adaptive batch size of 1% of the dataset size (i.e. the batch size was 1 for the dataset with 100 samples and 1000 for the dataset with 100 000 samples). Each model was trained for 200 epochs.

Table 1. Total number of patients, number of patients in each class, and sources of the datasets used in single- and multiomics prediction experiments.a

Dataset	Patients	DMFS/RFS	Source	
		< 5 years/NO	≥ 5 years/YES		
GSE11121	181	28	153	Schmidt et al. (2008)	
GSE1456	153	34	119	Pawitan et al. (2005)	
GSE2034	275	93	182	Wang et al. (2005)	
GSE2990	158	42	116	Sotiriou et al. (2006)	
GSE4922	228	69	159	Ivshina et al. (2006)	
GSE7390	191	56	135	Desmedt et al. (2007)	
METABRIC	1980	803	1177	Curtis et al. (2012)	
a The first six rows contain information about the single-omics datasets used to train COmic models and compare the results to previously published methods. The last line contains information about the METABRIC multiomics dataset used in multiomics prediction experiments. The single-omics classes are DMFS/RFS below 5 years and above 5 years, while the multiomics classes are RFS NO and RFS YES.

As shown in Fig. 2A, COmic models either outperformed competitors or performed similar to previously published methods. Notably, the globally interpretable pooling-based COmic models were able to achieve a small improvement in terms of auROC compared to all other models. On the other hand, the locally interpretable attention-based models achieved a similar performance as the previously best-performing model, PIMKL. We derived exemplary visualizations to evaluate the interpretation capabilities of COmic models. Since the pooling-based variant learns a molecular signature by weighting each pathway, we assessed the stability of this signature across the six breast cancer benchmarks. Each box in Fig. 2B represents one of the 50 pathways used for the prediction task and are created from the six corresponding weights learned by the models trained on the different datasets. The pathway signature remains quite stable over the six different datasets and high (absolute) weights are associated with known cancer pathways like androgen response (Pietri et al. 2016), hedgehog signaling (Jamieson et al. 2020), notch signaling (Farnie and Clarke 2007), and MYC target (Xu et al. 2010). With the introduction of attention-based COmic models, we introduce models with the capability of providing local interpretations, i.e. visualizations that provide insights into the decision process for a specific sample. We show an exemplary visualization of attention weights for three different, randomly chosen patients in the GSE11121 dataset in Fig. 2C. For Patient 1, the DMFS was correctly predicted to exceed 5 years. Patient 2 was correctly classified to have a DMFS below 5 years and Patient 3 was wrongly classified to have a DMFS above 5 years, while the actual DMFS of Patient 3 was shorter than 5 years. The highest attention weights are associated with known cancer pathways. For Patient 2, the highest amount of attention is given to hedgehog signaling. Androgen response gets the highest attention for Patients 1 and 3. More examples can be found in the Supplementary Material.

One key advantage of ANNs over kernel methods is their applicability on datasets with a vast number of samples. In the following, we will investigate, if our kernel networks provide the same applicability to large-scale datasets. Thus, we created simulated omics datasets of four different sizes: 100 samples, 1000 samples, 10 000 samples, and 10 000 samples. We then repeatedly trained pooling-based and attention-based COmic models on each simulated dataset five times and calculated the mean training time. Figure 2D shows the results. Since the batch size is usually chosen based on the number of samples in the training set, we calculated the mean training time for two different batch sizes. The blue and green lines show the training times of models trained with a fixed batch size of 32 samples per batch. The red and yellow lines show the training times of models with an adaptive batch size of one percent of the total sample count, i.e. each batch included a single sample, in case of the smallest simulated dataset, and 1000 samples, in case of the largest simulated dataset. The results show that COmic models can be easily trained on datasets with several hundreds of thousands of samples with the training time being linear depended on the number of samples. Furthermore, choosing an appropriate batch size can improve the training time by more than 50% on large-scale datasets.

3.2 Multiomics prediction on the METABRIC benchmark cohort

Since our proposed kernel layer can be incorporated into any ANN, COmic models can be flexibly expanded to multiomics datasets. One possibility is to directly add the pathway kernels for the additional omics datatypes to the kernel layer, thereby increasing the number of graph Laplacians in the kernel layer. Another simple approach is to create sub-networks for each omics type, i.e. combine the output of pooling- or attention-based single-omics COmic models with a simple fully connected network. There are numerous other ways to expand COmic models to multiomics data and, since our proposed approach is knowledge-driven, the individual solution has to be selected with the context of the data in mind. Similar to the authors of PIMKL, we chose the METABRIC cohort to investigate the practicality of applying COmic models to multiomics datasets. The METABRIC breast cancer cohort contains gene expression (mRNA) and CNA data. We performed the same prediction task as in (Manica et al. 2019), i.e. using molecular measures to predict whether a patient had recurrent cancer.

While we used pooling-based models with the same hyperparameters as described above for the single-omics prediction experiments, the creation of a multiomics COmic model for the METABRIC cohort had to be done with caution. The issue with CNA data are that this datatype is tremendously sparse. To compensate for this sparseness, we build a network that used pooling-based kernel layers to compute an embedding for each datatype that is robust enough for sparse data and afterwards used a gated attention layer as described in Section 2.3 to make the prediction. The pooling-based kernel layers used the same hyperparameters as the pooling-based COmic models in the single-omics experiments. For the gated attention layer, we used an attention dimensionality of 4. Networks were trained for 200 epochs with the Adam optimizer (Kingma and Ba 2014) using the class-balanced loss function (Cui et al. 2019). The batch size was set to 32. We used the auROC as our performance measure to enable the comparison to previously published PIMKL results on the METABRIC cohort. Additionally, we investigated the computational efficiency of COmic with regard to an increasing number of omics modalities.

The results of our experiments on the METABRIC cohort can be found on the left side of Fig. 3. The shown PIMKL performance is taken from (Manica et al. 2019). As expected, neither PIMKL nor COmic achieved good performance on the sparse CNA data. COmic models slightly outperformed PIMKL on the single-omics prediction task using gene expression data. While PIMKL shows a slightly decreased performance on the multiomics prediction task, COmic seems to have the same performance on the multiomics data as on the gene expression data alone with an increase in variance. The runtime analysis (right side of Fig. 3) shows a linear dependency on the number of omics modalities.

Figure 3. Left: Cross-validation performance of COmic and PIMKL models on the METABRIC cohort. The boxplots show the 10 mean auROC validation scores of a 10-times repeated 10-fold cross-validation. Performances of COmic models are shown in orange (the right box for each datatype). The center line of each box indicates the median. The height of the boxes represents the interquartile range (IQR) with the upper and lower whiskers set to 1.5 times the IQR. Notches represent the confidence interval (CI) around the median and were calculated using bootstrapping with 10000 iterations. Right: Mean training time of multiomics COmic models. Artificial datasets with 2, 3, and 4 different omics modalities and 1000 data points per modality were investigated. Batch size and number of epochs were fixed as described for the METABRIC cohort. Training of models was repeated five times and mean times are indicated by stars.

4 Discussion

Kernel methods allow to induce prior knowledge into a prediction task resulting in increased robustness and the introduction of interpretation capabilities. In this work, we propose COmic, a method to incorporate pathway-induced kernel functions into convolutional kernel networks. We are able to create learning models that can be robustly trained on small-scale datasets and scale very well with the number of samples. Thus, they can be efficiently applied to large datasets with hundreds of thousands of samples. Furthermore, our models provide global and local interpretations of predictions made on molecular measures due to the pathway-induced kernel function.

We used six different breast cancer cohorts to compare the performance of COmic models to previously proposed methods that use prior knowledge for prediction tasks with molecular measures as input data. The results presented in Fig. 2A show that our method reaches state-of-the-art performance on classifying patients based on their DMFS/RFS from gene expression data: Compared to the considered competitors, COmic performs similar or even better. However, COmic models have the advantage that the time needed to train a model scales linearly with the number of samples (see Fig. 2D). This enables the use of COmic models on datasets with hundreds of thousands of samples. We provide evidence that our method can be readily applied on large datasets by training models on simulated single-omics data with sizes ranging from 100 to 100 000 samples. Although datasets and patient cohorts used in computational biology and medicine traditionally have smaller sample sizes, high-throughput methods and the nowadays more frequently used big data paradigm will result in increasing sample counts in biological and medical datasets. At this day, TCGA already contains data from more than 85 000 patients. While methods that can deal with large datasets are usually deployed as black-box models, our method provides increased insight into the decision-making process.

Using single-omics datasets strongly limits the decision process for diagnosis of a majority of diseases. Nowadays, it is well known that multiomics information has to be incorporated to get a complete image of the pathomechanism causing a certain disease. Therefore, methods that are limited to a single datatype face serious constraints if employed as a decision-support system or to deepen knowledge about a pathomechanism. Our proposed method does not face the limitation of only using single-omics data as we show in our experiment with the METABRIC multiomics cohort. The results show again that our method improves single-omics prediction as demonstrated by the performance on the gene expression data. The lower performance that both methods, PIMKL and COmic, show on the CNA data can be explained by the sparseness of CNA data. Sparse data poses serious problems for prediction models (Li et al. 2016) and both methods are not specifically designed for sparse data. However, we can show in Fig. 3 that COmic models are able to achieve slightly better performance than PIMKL models on the multiomics prediction task. This indicates that our approach could be advantageously used on multiomics data, while the flexibility of the architecture (as described in Section 3.2) enables researchers to tailor COmic models for specific datasets using domain expertise.

Computing an interpretation of a machine learning method, either with post hoc methods or through intrinsically interpretable models, is beneficial if and only if the interpretation serves a purpose. This purpose cannot be defined in general as it is highly dependent on the task, the data, and the user that is presented with the obtained interpretation. For the presented experiments, we investigated if the inherent interpretation capabilities of our COmic models are able to learn biological meaningful concepts directly from data. First, we considered the global interpretation capabilities of COmic. Here, COmic models assign a weight to each of the used pathways and the weights reflect the role that each pathway plays in classifying an input sample, i.e. a patient. We trained COmic models on six different single-omics breast cancer cohorts. Since we expect the biological processes in the cohorts to share high similarities, the weight signatures of all models should be similar if the COmic method is able to learn meaningful pathways from data. As shown in Fig. 2B, this assumption is indeed well fulfilled with all six models having similar weight signatures. Furthermore, pathways with a high weight assigned to them are mainly known cancer-related pathways. Therefore, COmic models are able to learn biological meaningful pathway weights. Although the previously published PIMKL method also has a global interpretation capability, our method is able to learn pathways that are important for both, the negative and the positive class, due to the fact that the learned weights can be positive or negative. PIMKL only learns positive weights.

While global interpretation is useful to gain insights into a dataset, local interpretation can be used to get insights into the decision a model makes for a specific input. Attention-based COmic models can provide this insight utilizing the attention weights that are computed for each input sample separately. These weights directly determine the influence that each pathway has on the decision made by the model. We can visualize these influence using a heatmap (as shown in Fig. 2C) to quickly see which pathways played an important role in the decision made. We randomly selected three samples from the GSE11121 dataset to evaluate if the attention weights are biological meaningful. Similar to the weight signatures of the globally interpretable COmic models, the attention weights of the locally interpretable COmic models highlighted known cancer-related pathways. Interestingly, the selected patient with a DMFS below 5 years has attention weights that are strongly focused on a single pathway. This is true for all correctly classified patients with a DMFS below 5 years (see Supplementary Material). On the other hand, patients with a DMFS below 5 years that were wrongly classified to have a DMFS above 5 years show attention weight patterns similar to those of patients with a DMFS above 5 years (see Patient 3 in Fig. 2 and additional examples in the Supplementary Material). This could indicate that the wrongly classified patients exhibit a different mechanism causing a DMFS below 5 years, compared to the correctly classified ones, which was not learned by the model. The local interpretation capabilities of COmic models can help to directly show possible directions to further investigate the data. Furthermore, the results of our experiments strongly suggest that both COmic model types are able to generate biologically meaningful interpretations. We chose heatmaps to visualize attention weights, since it appeared convenient for the considered prediction task on the studied dataset. However, different forms of explanations can be computed with attention weights, e.g. counterfactual explanation (Tran et al. 2021) and adversarial explanation (Kitada and Iyatomi 2021). The most suitable form of explanation is highly dependent on the application, target user group, and the goal aimed at by the explanation. Therefore, the chosen visualization should be understood as an example and not a general application recommendation.

COmic models have a few hyperparameters that can be optimized using appropriate methods like, e.g. grid search or random search. These hyperparameters include the number of anchor points, the attention type, the dimensionality of the attention layer’s parameters V and U, and the choice of pathways used for kernel layers. Furthermore, different initialization procedures for the anchor points can be explored, e.g. a parameter-free clustering that combines initializing anchor points with optimizing the number of anchor points for each pathway-induced kernel layer. We recommend to explore hyperparameter optimization when applying COmic models. However, minimizing energy consumption is a pressing concern that should be considered in every line of research nowadays. Therefore, we limited the computations performed for this work to the minimum required to support our claims. The hyperparameters for all models presented in this work were chosen by combining prior experience about kernel networks with domain expertise. Interestingly, this computation-free approach to hyperparameter selection already leads to competitive performance of our method on the considered prediction tasks.

5 Conclusion

The introduced convolutional omics kernel networks utilize prior knowledge by pathway-induced kernel functions to provide robust end-to-end learning on small- to large-scale molecular measure datasets. Furthermore, utilizing pathway-induced kernel functions makes our method intrinsically interpretable with the ability to provide global and local interpretations.

We show the competitive performance of our method on six different single-omics breast cancer cohorts while providing new interpretation capabilities that exceed the possibilities of previously proposed methods. Furthermore, we show that COmic models can be readily adapted to multiomics datasets.

On a larger scale, we show that incorporating a carefully crafted kernel function into an ANN allows to robustly train ANNs on small-scale datasets as they frequently occur in computational biology and medicine. On the other hand, our method enables scientist to utilize kernel functions for large datasets as they arise more frequently with the increasing use of high-throughput methods and big data.

Supplementary Material

btad204_Supplementary_Data Click here for additional data file.

Supplementary data

Supplementary data are available at Bioinformatics online.

Conflict of interest

None declared.

Funding

This work was supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy—EXC number 2064/1—Project number 390727645. This research was supported by the German Federal Ministry of Education and Research (BMBF) project “Training Center Machine Learning, Tübingen” with grant number 01-S17054. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A.
==== Refs
References

Alipanahi B , DelongA, WeirauchMT et al Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning. Nat Biotechnol 2015;33 :831–8.26213851
Álvarez-Machancoses Ó , Fernández-MartínezJL. Using artificial intelligence methods to speed up drug discovery. Expert Opin Drug Disc 2019;14 :769–77.
Arieno A , ChanA, DestounisSV. A review of the role of augmented intelligence in breast imaging: from automated breast density assessment to risk stratification. Am J Roentgenol 2019;212 :259–70.30422711
Bo L , LaiK, RenX et al Object recognition with hierarchical kernel descriptors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2011, 24:1729–1736.
Bordt S , FinckM, RaidlE et al Post-hoc explanations fail to achieve their purpose in adversarial contexts. In: 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT '22). Association for Computing Machinery, New York, NY, USA, pp. 891–905. 10.1145/3531146.3533153.
Chen D , JacobL, MairalJ. Biological sequence modeling with convolutional kernel networks. Bioinformatics 2019a;35 :3294–302.30753280
Chen D , JacobL, MairalJ. Convolutional kernel networks for graph-structured data. Int Conf Mach Learn 2020;37:1576–86.
Chen D , JacobL, MairalJ. Recurrent kernel networks. Adv Neural Inf Process Syst 2019b;32:13431–42.
Chen Z , LiuJ, NgH et al Identifying cancer biomarkers by network-constrained support vector machines. BMC Syst Biol 2011;5 :S1–20.
Cho Y , SaulL. Kernel methods for deep learning. Adv Neural Inf Process Syst 2009;22 :342–50.
Cui Y , JiaM, LinT-Y et al Class-balanced loss based on effective number of samples. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA. 2019, 9268–77.
Cun Y , FröhlichH. Prognostic gene signatures for patient stratification in breast cancer-accuracy, stability and interpretability of gene selection approaches using prior knowledge on protein-protein interactions. BMC Bioinformatics 2012;13 :1–13.22214541
Curtis C , ShahSP, ChinS-F et al ; METABRIC Group. The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups. Nature 2012;486 :346–52.22522925
Dauphin YN , FanA, AuliM et al Language modeling with gated convolutional networks. In: International Conference on Machine Learning, Sydney, Australia. 2017, 933–941.
De Fauw J , LedsamJR, Romera-ParedesB et al Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med 2018;24 :1342–50.30104768
Desmedt C , PietteF, LoiS et al ; TRANSBIG Consortium. Strong time dependence of the 76-gene prognostic signature for node-negative breast cancer patients in the transbig multicenter independent validation series. Clin Cancer Res 2007;13 :3207–14.17545524
Dietterich TG , LathropRH, Lozano-PérezT. Solving the multiple instance problem with axis-parallel rectangles. Artif Intell 1997;89 :31–71.
Ditz JC , ReuterB, PfeiferN. Convolutional motif kernel networks. arXiv, arXiv:2111.02272, 2021, preprint: not peer reviewed.
Eskin E , WestonJ, NobleWS et al Mismatch string kernels for SVM protein classification. Adv Neural Inf Process Syst 2003;15:1441–8.
Farnie G , ClarkeRB. Mammary stem cells and breast cancer—role of notch signalling. Stem Cell Rev 2007;3 :169–75.17873349
Ferrari A , LombardiS, SignoroniA. Bacterial colony counting with convolutional neural networks in digital microbiology imaging. Pattern Recognit 2017;61 :629–40.
Fleming N. How artificial intelligence is changing drug discovery. Nature 2018;557 :S55–S57.29849160
Gao C , DangX, ChenY et al Graph ranking for exploratory gene data analysis. BMC Bioinformatics 2009;10 :S19.
Guo Z , ZhangT, LiX et al Towards precise classification of cancers based on robust gene functional expression profiles. BMC Bioinformatics 2005;6 :58–12.15774002
Guyon I , WestonJ, BarnhillS et al Gene selection for cancer classification using support vector machines. Mach Learn 2002;46 :389–422.
Hay SI , GeorgeDB, MoyesCL et al Big data opportunities for global infectious disease surveillance. PLoS Med 2013;10 :e1001413.23565065
Ilse M , TomczakJ, WellingM. Attention-based deep multiple instance learning. In: International Conference on Machine Learning, Stockholm, Sweden, 2018, 2127–2136.
Ivshina AV , GeorgeJ, SenkoO et al Genetic reclassification of histologic grade delineates new clinical subtypes of breast cancer. Cancer Res 2006;66 :10292–301.17079448
Jamieson C , MartinelliG, PapayannidisC et al Hedgehog pathway inhibitors: a new therapeutic class for the treatment of acute myeloid leukemia hedgehog pathway inhibitors for acute myeloid leukemia. Blood Cancer Discov 2020;1 :134–45.34661144
Kelley DR , SnoekJ, RinnJL. Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks. Genome Res 2016;26 :990–9.27197224
Kingma DP , BaJ. Adam: a method for stochastic optimization. arXiv, arXiv:1412.6980, 2014, preprint: not peer reviewed.
Kitada S , IyatomiH. Attention meets perturbations: robust and interpretable attention with adversarial training. IEEE Access 2021;9 :92974–85.
Lee E , ChuangH-Y, KimJ-W et al Inferring pathway activity toward precise disease classification. PLoS Comput Biol 2008;4 :e1000217.18989396
Li X , LingCX, WangH. The convergence behavior of naive bayes on large sparse datasets. ACM Trans Knowl Discov Data 2016;11 :1–24.
Lipton ZC. The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery. Queue 2018;16 :31–57.
Liu G , WuJ, ZhouZ-H. Key instance detection in multi-instance learning. In: Asian Conference on Machine Learning, Singapore, Singapore, 2012, 253–268.
Lundberg SM , LeeS-I. A unified approach to interpreting model predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA. 2017, 4768–4777.
Mairal J. End-to-end kernel learning with supervised convolutional kernel networks. Adv Neural Inf Process Syst 2016;29:1399–407.
Mairal J , KoniuszP, HarchaouiZ et al Convolutional kernel networks. Adv Neural Inf Process Syst 2014;27:2627–35.
Manica M , CadowJ, MathisR et al Pimkl: pathway-induced multiple kernel learning. NPJ Syst Biol Appl 2019;5 :8.30854223
Maron O , Lozano-PérezT. A framework for multiple-instance learning. Adv Neural Inf Process Syst 1997;10.
Meinicke P , TechM, MorgensternB et al Oligo kernels for datamining on biological sequences: a case study on prokaryotic translation initiation sites. BMC Bioinformatics 2004;5 :169.15511290
Miotto R , LiL, KiddBA et al Deep patient: an unsupervised representation to predict the future of patients from the electronic health records. Sci Rep 2016;6 :1–10.28442746
Molnar C. Interpretable Machine Learning. Lulu. com., 2020.
Montavon G , BinderA, LapuschkinS et al Layer-wise relevance propagation: an overview. In: Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. 2019, 193–209.
Montavon G , LapuschkinS, BinderA et al Explaining nonlinear classification decisions with deep taylor decomposition. Pattern Recognit 2017;65 :211–22.
Nelson A , HerronD, ReesG et al Predicting scheduled hospital attendance with artificial intelligence. NPJ Digit Med 2019;2 :1–7.31304351
Oquab M , BottouL, LaptevI et al Is object localization for free?-weakly-supervised learning with convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, 685–694.
Pärnamaa T , PartsL. Accurate classification of protein subcellular localization from high-throughput microscopy images using deep learning. G3 (Bethesda) 2017;7 :1385–92.28391243
Pawitan Y , BjöhleJ, AmlerL et al Gene expression profiling spares early breast cancer patients from adjuvant therapy: derived and validated in two population-based cohorts. Breast Cancer Res 2005;7 :1–12.15642174
Pietri E , ConteducaV, AndreisD et al Androgen receptor signaling pathways as a target for breast cancer treatment. Endocr Relat Cancer 2016;23 :R485–98.27528625
Rapaport F , ZinovyevA, DutreixM et al Classification of microarray data using gene networks. BMC Bioinformatics 2007;8 :1–15.17199892
Ronneberger O , FischerP, BroxT. U-net: convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany. 2015, 234–241.
Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell 2019;1 :206–15.35603010
Schmidt M , BöhmD, von TörneC et al The humoral immune system has a key prognostic impact in node-negative breast cancer. Cancer Res 2008;68 :5405–13.18593943
Shrikumar A , GreensideP, KundajeA. Learning important features through propagating activation differences. In: International Conference on Machine Learning, Sydney, Australia. 2017, 3145–3153.
Sixt L , GranzM, LandgrafT. When explanations lie: Why many modified bp attributions fail. In: International Conference on Machine Learning, virtual conference. 2020, 9046–9057.
Sotiriou C , WirapatiP, LoiS et al Gene expression profiling in breast cancer: understanding the molecular basis of histologic grade to improve prognosis. J Natl Cancer Inst 2006;98 :262–72.16478745
Springenberg JT , DosovitskiyA, BroxT et al Striving for simplicity: the all convolutional net. arXiv, arXiv:1412.6806, 2014, preprint: not peer reviewed.
Stepin I , AlonsoJM, CatalaA et al A survey of contrastive and counterfactual explanation generation methods for explainable artificial intelligence. IEEE Access 2021;9 :11974–2001.
Taylor IW , LindingR, Warde-FarleyD et al Dynamic modularity in protein interaction networks predicts breast cancer outcome. Nat Biotechnol 2009;27 :199–204.19182785
Tran KH , GhazimatinA, Saha RoyR. Counterfactual explanations for neural recommenders. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, virtual conference. 2021, 1627–1631.
Vassilvitskii S , ArthurD. k-means++: The advantages of careful seeding. In: Proceedings of the Eighteenth Annual ACM-SIAM Symposium On Discrete Algorithms, New Orleans, Louisiana, USA. 2006, 1027–1035.
Von Luxburg U. A tutorial on spectral clustering. Stat Comput 2007;17 :395–416.
Wang X , YanY, TangP et al Revisiting multiple instance neural networks. Pattern Recognit 2018;74 :15–24.
Wang Y , KlijnJG, ZhangY et al Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer. Lancet 2005;365 :671–9.15721472
Williams C , SeegerM. Using the nyström method to speed up kernel machines. In: Proceedings of the 14th Annual Conference on Neural Information Processing Systems, (CONF), Vancouver, British Columbia, Canada. 2001, 682–688.
Xu J , ChenY, OlopadeOI. Myc and breast cancer. Genes Cancer 2010;1 :629–40.21779462
Zhang K , TsangIW, KwokJT. Improved nyström low-rank approximation and error analysis. In: Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland. 2008, 1232–1239.
Zhou J , TroyanskayaOG. Predicting effects of noncoding variants with deep learning–based sequence model. Nat Methods 2015;12 :931–4.26301843
Zhu Y , ShenX, PanW. Network-based support vector machine for classification of microarray samples. BMC Bioinformatics 2009;10 :S21.