  Fully supervised models often require large amounts of labeled training data,
which tends to be costly and hard to acquire. In contrast, self-supervised
representation learning reduces the amount of labeled data needed for achieving
the same or even higher downstream performance. The goal is to pre-train deep
neural networks on a self-supervised task such that afterwards the networks are
able to extract meaningful features from raw input data. These features are
then used as inputs in downstream tasks, such as image classification.
Previously, autoencoders and Siamese networks such as SimSiam have been
successfully employed in those tasks. Yet, challenges remain, such as matching
characteristics of the features (e.g., level of detail) to the given task and
data set. In this paper, we present a new self-supervised method that combines
the benefits of Siamese architectures and denoising autoencoders. We show that
our model, called SidAE (Siamese denoising autoencoder), outperforms two
self-supervised baselines across multiple data sets, settings, and scenarios.
Crucially, this includes conditions in which only a small amount of labeled
data is available.
