  Recent unsupervised representation learning methods have shown to be
effective in a range of vision tasks by learning representations invariant to
data augmentations such as random cropping and color jittering. However, such
invariance could be harmful to downstream tasks if they rely on the
characteristics of the data augmentations, e.g., location- or color-sensitive.
This is not an issue just for unsupervised learning; we found that this occurs
even in supervised learning because it also learns to predict the same label
for all augmented samples of an instance. To avoid such failures and obtain
more generalizable representations, we suggest to optimize an auxiliary
self-supervised loss, coined AugSelf, that learns the difference of
augmentation parameters (e.g., cropping positions, color adjustment
intensities) between two randomly augmented samples. Our intuition is that
AugSelf encourages to preserve augmentation-aware information in learned
representations, which could be beneficial for their transferability.
Furthermore, AugSelf can easily be incorporated into recent state-of-the-art
representation learning methods with a negligible additional training cost.
Extensive experiments demonstrate that our simple idea consistently improves
the transferability of representations learned by supervised and unsupervised
methods in various transfer learning scenarios. The code is available at
https://github.com/hankook/AugSelf.
