  Single-cell RNA sequencing (scRNA-seq) data is a potent tool for
comprehending the "language of life" and can provide insights into various
downstream biomedical tasks. Large-scale language models (LLMs) are starting to
be used for cell representation learning. However, current LLM-based cell
representation learning methods depend solely on the BERT architecture, causing
an anisotropic embedding space that leads to inefficient semantic
representation. Contrastive learning alleviates this problem by distributing
the embeddings uniformly. As a larger batch size in contrastive learning
results in better representation, the practical application of contrastive
learning in cell representation learning is hampered by the high dimensionality
of scRNA-seq data and the large parameter volume of LLMs. To address the batch
size limitation, we propose a novel divide-and-conquer contrastive learning
approach to decouple the batch size from the GPU memory size for cell
representation learning. Based on our divide-and-conquer contrastive learning
approach, we introduce Single-Cell Language Model CellLM, a large-scale cell
representation learning model to handle high-dimensional scRNA-seq data with
tens of thousands of genes. CellLM has over 50 million parameters trained with
2 million scRNA-seq data and makes the first attempt to learn cell language
models from both normal cells and cancer cells. CellLM achieves new
state-of-the-art (SOTA) results in all evaluated downstream tasks: including a
71.8 F_1-score for cell type annotation (a 3.0% absolute improvement over
scBERT), an average F_1-score of 88.9 for single-cell drug sensitivity
prediction in a few-shot scenario (an 8.3% absolute improvement), and a 93.4
Pearson's correlation for single-omics cell line drug sensitivity prediction (a
6.2% absolute improvement).
