  Regression models often fail to generalize effectively in regions
characterized by highly imbalanced label distributions. Previous methods for
deep imbalanced regression rely on gradient-based weight updates, which tend to
overfit in underrepresented regions. This paper proposes a paradigm shift
towards in-context learning as an effective alternative to conventional
in-weight learning methods, particularly for addressing imbalanced regression.
In-context learning refers to the ability of a model to condition itself, given
a prompt sequence composed of in-context samples (input-label pairs) alongside
a new query input to generate predictions, without requiring any parameter
updates. In this paper, we study the impact of the prompt sequence on the model
performance from both theoretical and empirical perspectives. We emphasize the
importance of localized context in reducing bias within regions of high
imbalance. Empirical evaluations across a variety of real-world datasets
demonstrate that in-context learning substantially outperforms existing
in-weight learning methods in scenarios with high levels of imbalance.
