  It has been reported that deep learning models are extremely vulnerable to
small but intentionally chosen perturbations of its input. In particular, a
deep network, despite its near-optimal accuracy on the clean images, often
mis-classifies an image with a worst-case but humanly imperceptible
perturbation (so-called adversarial examples). To tackle this problem, a great
amount of research has been done to study the training procedure of a network
to improve its robustness. However, most of the research so far has focused on
the case of supervised learning. With the increasing popularity of
self-supervised learning methods, it is also important to study and improve the
robustness of their resulting representation on the downstream tasks. In this
paper, we study the problem of robust representation learning with unlabeled
data in a task-agnostic manner. Specifically, we first derive an upper bound on
the adversarial loss of a prediction model (which is based on the learned
representation) on any downstream task, using its loss on the clean data and a
robustness regularizer. Moreover, the regularizer is task-independent, thus we
propose to minimize it directly during the representation learning phase to
make the downstream prediction model more robust. Extensive experiments show
that our method achieves preferable adversarial performance compared to
relevant baselines.
