  We propose methods to strengthen the invariance properties of representations
obtained by contrastive learning. While existing approaches implicitly induce a
degree of invariance as representations are learned, we look to more directly
enforce invariance in the encoding process. To this end, we first introduce a
training objective for contrastive learning that uses a novel regularizer to
control how the representation changes under transformation. We show that
representations trained with this objective perform better on downstream tasks
and are more robust to the introduction of nuisance transformations at test
time. Second, we propose a change to how test time representations are
generated by introducing a feature averaging approach that combines encodings
from multiple transformations of the original input, finding that this leads to
across the board performance gains. Finally, we introduce the novel Spirograph
dataset to explore our ideas in the context of a differentiable generative
process with multiple downstream tasks, showing that our techniques for
learning invariance are highly beneficial.
