  Self-supervised learning (SSL), as a newly emerging unsupervised
representation learning paradigm, generally follows a two-stage learning
pipeline: 1) learning invariant and discriminative representations with
auto-annotation pretext(s), then 2) transferring the representations to assist
downstream task(s). Such two stages are usually implemented separately, making
the learned representation learned agnostic to the downstream tasks. Currently,
most works are devoted to exploring the first stage. Whereas, it is less
studied on how to learn downstream tasks with limited labeled data using the
already learned representations. Especially, it is crucial and challenging to
selectively utilize the complementary representations from diverse pretexts for
a downstream task. In this paper, we technically propose a novel solution by
leveraging the attention mechanism to adaptively squeeze suitable
representations for the tasks. Meanwhile, resorting to information theory, we
theoretically prove that gathering representation from diverse pretexts is more
effective than a single one. Extensive experiments validate that our scheme
significantly exceeds current popular pretext-matching based methods in
gathering knowledge and relieving negative transfer in downstream tasks.
