  While unbiased machine learning models are essential for many applications,
bias is a human-defined concept that can vary across tasks. Given only
input-label pairs, algorithms may lack sufficient information to distinguish
stable (causal) features from unstable (spurious) features. However, related
tasks often share similar biases -- an observation we may leverage to develop
stable classifiers in the transfer setting. In this work, we explicitly inform
the target classifier about unstable features in the source tasks.
Specifically, we derive a representation that encodes the unstable features by
contrasting different data environments in the source task. We achieve
robustness by clustering data of the target task according to this
representation and minimizing the worst-case risk across these clusters. We
evaluate our method on both text and image classifications. Empirical results
demonstrate that our algorithm is able to maintain robustness on the target
task for both synthetically generated environments and real-world environments.
Our code is available at https://github.com/YujiaBao/Tofu.
