  Most data is automatically collected and only ever "seen" by algorithms. Yet,
data compressors preserve perceptual fidelity rather than just the information
needed by algorithms performing downstream tasks. In this paper, we
characterize the bit-rate required to ensure high performance on all predictive
tasks that are invariant under a set of transformations, such as data
augmentations. Based on our theory, we design unsupervised objectives for
training neural compressors. Using these objectives, we train a generic image
compressor that achieves substantial rate savings (more than $1000\times$ on
ImageNet) compared to JPEG on 8 datasets, without decreasing downstream
classification performance.
