  While directly fine-tuning (FT) large-scale, pretrained models on
task-specific data is well-known to induce strong in-distribution task
performance, recent works have demonstrated that different adaptation
protocols, such as linear probing (LP) prior to FT, can improve
out-of-distribution generalization. However, the design space of such
adaptation protocols remains under-explored and the evaluation of such
protocols has primarily focused on distribution shifts. Therefore, in this
work, we evaluate common adaptation protocols across distributions shifts and
machine learning safety metrics (e.g., anomaly detection, calibration,
robustness to corruptions). We find that protocols induce disparate trade-offs
that were not apparent from prior evaluation. Further, we demonstrate that
appropriate pairing of data augmentation and protocol can substantially
mitigate this trade-off. Finally, we hypothesize and empirically see that using
hardness-promoting augmentations during LP and then FT with augmentations may
be particularly effective for trade-off mitigation.
