  Tumor is heterogeneous - a tumor sample usually consists of a set of
subclones with distinct transcriptional profiles and potentially different
degrees of aggressiveness and responses to drugs. Understanding tumor
heterogeneity is therefore critical to precise cancer prognosis and treatment.
In this paper, we introduce BayCount, a Bayesian decomposition method to infer
tumor heterogeneity with highly over-dispersed RNA sequencing count data. Using
negative binomial factor analysis, BayCount takes into account both the
between-sample and gene-specific random effects on raw counts of sequencing
reads mapped to each gene. For posterior inference, we develop an efficient
compound Poisson based blocked Gibbs sampler. Through extensive simulation
studies and analysis of The Cancer Genome Atlas lung cancer and kidney cancer
RNA sequencing count data, we show that BayCount is able to accurately estimate
the number of subclones, the proportions of these subclones in each tumor
sample, and the gene expression profiles in each subclone. Our method
represents the first effort in characterizing tumor heterogeneity using RNA
sequencing count data that simultaneously removes the need of normalizing the
counts, achieves statistical robustness, and obtains biologically and
clinically meaningful insights.
