  Large-scale vision models have become integral in many applications due to
their unprecedented performance and versatility across downstream tasks.
However, the robustness of these foundation models has primarily been explored
for a single task, namely image classification. The vulnerability of other
common vision tasks, such as semantic segmentation and depth estimation,
remains largely unknown. We present a comprehensive empirical evaluation of the
adversarial robustness of self-supervised vision encoders across multiple
downstream tasks. Our attacks operate in the encoder embedding space and at the
downstream task output level. In both cases, current state-of-the-art
adversarial fine-tuning techniques tested only for classification significantly
degrade clean and robust performance on other tasks. Since the purpose of a
foundation model is to cater to multiple applications at once, our findings
reveal the need to enhance encoder robustness more broadly. Our code is
available at ${github.com/layer6ai-labs/ssl-robustness}$.
