  Contrastive learning pre-trains an image encoder using a large amount of
unlabeled data such that the image encoder can be used as a general-purpose
feature extractor for various downstream tasks. In this work, we propose
PoisonedEncoder, a data poisoning attack to contrastive learning. In
particular, an attacker injects carefully crafted poisoning inputs into the
unlabeled pre-training data, such that the downstream classifiers built based
on the poisoned encoder for multiple target downstream tasks simultaneously
classify attacker-chosen, arbitrary clean inputs as attacker-chosen, arbitrary
classes. We formulate our data poisoning attack as a bilevel optimization
problem, whose solution is the set of poisoning inputs; and we propose a
contrastive-learning-tailored method to approximately solve it. Our evaluation
on multiple datasets shows that PoisonedEncoder achieves high attack success
rates while maintaining the testing accuracy of the downstream classifiers
built upon the poisoned encoder for non-attacker-chosen inputs. We also
evaluate five defenses against PoisonedEncoder, including one pre-processing,
three in-processing, and one post-processing defenses. Our results show that
these defenses can decrease the attack success rate of PoisonedEncoder, but
they also sacrifice the utility of the encoder or require a large clean
pre-training dataset.
