  Advances in the expressivity of pretrained models have increased interest in
the design of adaptation protocols which enable safe and effective transfer
learning. Going beyond conventional linear probing (LP) and fine tuning (FT)
strategies, protocols that can effectively control feature distortion, i.e.,
the failure to update features orthogonal to the in-distribution, have been
found to achieve improved out-of-distribution generalization (OOD). In order to
limit this distortion, the LP+FT protocol, which first learns a linear probe
and then uses this initialization for subsequent FT, was proposed. However, in
this paper, we find when adaptation protocols (LP, FT, LP+FT) are also
evaluated on a variety of safety objectives (e.g., calibration, robustness,
etc.), a complementary perspective to feature distortion is helpful to explain
protocol behavior. To this end, we study the susceptibility of protocols to
simplicity bias (SB), i.e. the well-known propensity of deep neural networks to
rely upon simple features, as SB has recently been shown to underlie several
problems in robust generalization. Using a synthetic dataset, we demonstrate
the susceptibility of existing protocols to SB. Given the strong effectiveness
of LP+FT, we then propose modified linear probes that help mitigate SB, and
lead to better initializations for subsequent FT. We verify the effectiveness
of the proposed LP+FT variants for decreasing SB in a controlled setting, and
their ability to improve OOD generalization and safety on three adaptation
datasets.
