  Sampling biases can cause distribution shifts between train and test datasets
for supervised learning tasks, obscuring our ability to understand the
generalization capacity of a model. This is especially important considering
the wide adoption of pre-trained foundational neural networks -- whose behavior
remains poorly understood -- for transfer learning (TL) tasks. We present a
case study for TL on the Sentiment140 dataset and show that many pre-trained
foundation models encode different representations of Sentiment140's manually
curated test set $M$ from the automatically labeled training set $P$,
confirming that a distribution shift has occurred. We argue training on $P$ and
measuring performance on $M$ is a biased measure of generalization. Experiments
on pre-trained GPT-2 show that the features learnable from $P$ do not improve
(and in fact hamper) performance on $M$. Linear probes on pre-trained GPT-2's
representations are robust and may even outperform overall fine-tuning,
implying a fundamental importance for discerning distribution shift in
train/test splits for model interpretation.
