  Training foundation models on extensive datasets and then finetuning them on
specific tasks has emerged as the mainstream approach in artificial
intelligence. However, the model robustness, which is a critical aspect for
safety, is often optimized for each specific task rather than at the
pretraining stage. In this paper, we propose a method for pretraining
certifiably robust models that can be readily finetuned for adaptation to a
particular task. A key challenge is dealing with the compromise between
semantic learning and robustness. We address this with a simple yet highly
effective strategy based on significantly broadening the pretraining data
distribution, which is shown to greatly benefit finetuning for downstream
tasks. Through pretraining on a mixture of clean and various noisy images, we
find that surprisingly strong certified accuracy can be achieved even when
finetuning on only clean images. Furthermore, this strategy requires just a
single model to deal with various noise levels, thus substantially reducing
computational costs in relation to previous works that employ multiple models.
Despite using just one model, our method can still yield results that are on
par with, or even superior to, existing multi-model methods.
