  What matters for contrastive learning? We argue that contrastive learning
heavily relies on informative features, or "hard" (positive or negative)
features. Early works include more informative features by applying complex
data augmentations and large batch size or memory bank, and recent works design
elaborate sampling approaches to explore informative features. The key
challenge toward exploring such features is that the source multi-view data is
generated by applying random data augmentations, making it infeasible to always
add useful information in the augmented data. Consequently, the informativeness
of features learned from such augmented data is limited. In response, we
propose to directly augment the features in latent space, thereby learning
discriminative representations without a large amount of input data. We perform
a meta learning technique to build the augmentation generator that updates its
network parameters by considering the performance of the encoder. However,
insufficient input data may lead the encoder to learn collapsed features and
therefore malfunction the augmentation generator. A new margin-injected
regularization is further added in the objective function to avoid the encoder
learning a degenerate mapping. To contrast all features in one gradient
back-propagation step, we adopt the proposed optimization-driven unified
contrastive loss instead of the conventional contrastive loss. Empirically, our
method achieves state-of-the-art results on several benchmark datasets.
