  Unsupervised representation learning has recently received lots of interest
due to its powerful generalizability through effectively leveraging large-scale
unlabeled data. There are two prevalent approaches for this, contrastive
learning and generative pre-training, where the former learns representations
from instance-wise discrimination tasks and the latter learns them from
estimating the likelihood. These seemingly orthogonal approaches have their own
strengths and weaknesses. Contrastive learning tends to extract semantic
information and discards details irrelevant for classifying objects, making the
representations effective for discriminative tasks while degrading robustness
to out-of-distribution data. On the other hand, the generative pre-training
directly estimates the data distribution, so the representations tend to be
robust but not optimal for discriminative tasks. In this paper, we show that we
could achieve the best of both worlds by a hybrid training scheme.
Specifically, we demonstrated that a transformer-based encoder-decoder
architecture trained with both contrastive and generative losses can learn
highly discriminative and robust representations without hurting the generative
performance. We extensively validate our approach on various tasks.
