  Contrastive, self-supervised learning (SSL) is used to train a model that
predicts cancer type from miRNA, mRNA or RPPA expression data. This model, a
pretrained FT-Transformer, is shown to outperform XGBoost and CatBoost,
standard benchmarks for tabular data, when labelled samples are scarce but the
number of unlabelled samples is high. This is despite the fact that the
datasets we use have $\mathcal{O}(10^{1})$ classes and
$\mathcal{O}(10^{2})-\mathcal{O}(10^{4})$ features. After demonstrating the
efficacy of our chosen method of self-supervised pretraining, we investigate
SSL for multi-modal models. A late-fusion model is proposed, where each omics
is passed through its own sub-network, the outputs of which are averaged and
passed to the pretraining or downstream objective function. Multi-modal
pretraining is shown to improve predictions from a single omics, and we argue
that this is useful for datasets with many unlabelled multi-modal samples, but
few labelled unimodal samples. Additionally, we show that pretraining each
omics-specific module individually is highly effective. This enables the
application of the proposed model in a variety of contexts where a large amount
of unlabelled data is available from each omics, but only a few labelled
samples.
