
==== Front
bioRxiv
BIORXIV
bioRxiv
2692-8205
Cold Spring Harbor Laboratory

10.1101/2024.08.16.608287
preprint
1
Article
cytoKernel: Robust kernel embeddings for assessing differential expression of single cell data
Ghosh Tusharkanti http://orcid.org/0000-0002-7537-6374

Baxter Ryan M http://orcid.org/0000-0002-2235-6193

Seal Souvik http://orcid.org/0000-0003-3268-610X

Lui Victor G http://orcid.org/0000-0003-1553-1499

Rudra Pratyaydipta http://orcid.org/0000-0002-1089-7283

Vu Thao http://orcid.org/0000-0001-5252-0006

Hsieh Elena WY http://orcid.org/0000-0003-3969-6597

Ghosh Debashis http://orcid.org/0000-0001-6618-1316

19 8 2024
2024.08.16.608287https://creativecommons.org/licenses/by/4.0/ This work is licensed under a Creative Commons Attribution 4.0 International License, which allows reusers to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator. The license allows for commercial use.
http://biorxiv.org/lookup/doi/10.1101/2024.08.16.608287
nihpp-2024.08.16.608287.pdf
High-throughput sequencing of single-cell data can be used to rigorously evlauate cell specification and enable intricate variations between groups or conditions. Many popular existing methods for differential expression target differences in aggregate measurements (mean, median, sum) and limit their approaches to detect only global differential changes. We present a robust method for differential expression of single-cell data using a kernel-based score test, cytoKernel. cytoKernel is specifically designed to assess the differential expression of single cell RNA sequencing and high-dimensional flow or mass cytometry data using the full probability distribution pattern. cytoKernel is based on kernel embeddings which employs the probability distributions of the single cell data, by calculating the pairwise divergence/distance between distributions of subjects. It can detect both patterns involving aggregate changes, as well as more elusive variations that are often overlooked due to the multimodal characteristics of single cell data. We performed extensive benchmarks across both simulated and real data sets from mass cytometry data and single-cell RNA sequencing. The cytoKernel procedure effectively controls the False Discovery Rate (FDR) and shows favourable performance compared to existing methods. The method is able to identify more differential patterns than existing approaches. We apply cytoKernel to assess gene expression and protein marker expression differences from cell subpopulations in various publicly available single-cell RNAseq and mass cytometry data sets. The methods described in this paper are implemented in the open-source R package cytoKernel, which is freely available from Bioconductor at \url{http://bioconductor.org/packages/cytoKernel}.
==== Body
pmc
