  Randomized smoothing is the primary certified robustness method for accessing
the robustness of deep learning models to adversarial perturbations in the
l2-norm, by adding isotropic Gaussian noise to the input image and returning
the majority votes over the base classifier. Theoretically, it provides a
certified norm bound, ensuring predictions of adversarial examples are stable
within this bound. A notable constraint limiting widespread adoption is the
necessity to retrain base models entirely from scratch to attain a robust
version. This is because the base model fails to learn the noise-augmented data
distribution to give an accurate vote. One intuitive way to overcome this
challenge is to involve a custom-trained denoiser to eliminate the noise.
However, this approach is inefficient and sub-optimal. Inspired by recent large
model training procedures, we explore an alternative way named PEFTSmoothing to
adapt the base model to learn the Gaussian noise-augmented data with
Parameter-Efficient Fine-Tuning (PEFT) methods in both white-box and black-box
settings. Extensive results demonstrate the effectiveness and efficiency of
PEFTSmoothing, which allow us to certify over 98% accuracy for ViT on CIFAR-10,
20% higher than SoTA denoised smoothing, and over 61% accuracy on ImageNet
which is 30% higher than CNN-based denoiser and comparable to the
Diffusion-based denoiser.
