  When evaluating the performance of a pre-trained model transferred to a
downstream task, it is imperative to assess not only the in-distribution (ID)
accuracy of the downstream model but also its capacity to generalize and
identify out-of-distribution (OOD) samples. In this paper, we unveil the hidden
costs associated with intrusive fine-tuning techniques. Specifically, we
demonstrate that commonly used fine-tuning methods not only distort the
representations necessary for generalizing to covariate-shifted OOD samples
(OOD generalization) but also distort the representations necessary for
detecting semantically-shifted OOD samples (OOD detection). To address these
challenges, we introduce a new model reprogramming approach for fine-tuning,
which we name Reprogrammer. Reprogrammer aims to improve the holistic
performance of the downstream model across ID, OOD generalization, and OOD
detection tasks. Our empirical evidence reveals that Reprogrammer is less
intrusive and yields superior downstream models. Furthermore, we demonstrate
that by appending an additional representation residual connection to
Reprogrammer, we can further preserve pre-training representations, resulting
in an even more safe and robust downstream model capable of excelling in many
ID classification, OOD generalization, and OOD detection settings.
