  Learned reweighting (LRW) approaches to supervised learning use an
optimization criterion to assign weights for training instances, in order to
maximize performance on a representative validation dataset. We pose and
formalize the problem of optimized selection of the validation set used in LRW
training, to improve classifier generalization. In particular, we show that
using hard-to-classify instances in the validation set has both a theoretical
connection to, and strong empirical evidence of generalization. We provide an
efficient algorithm for training this meta-optimized model, as well as a simple
train-twice heuristic for careful comparative study. We demonstrate that LRW
with easy validation data performs consistently worse than LRW with hard
validation data, establishing the validity of our meta-optimization problem.
Our proposed algorithm outperforms a wide range of baselines on a range of
datasets and domain shift challenges (Imagenet-1K, CIFAR-100, Clothing-1M,
CAMELYON, WILDS, etc.), with ~1% gains using VIT-B on Imagenet. We also show
that using naturally hard examples for validation (Imagenet-R / Imagenet-A) in
LRW training for Imagenet improves performance on both clean and naturally hard
test instances by 1-2%. Secondary analyses show that using hard validation data
in an LRW framework improves margins on test data, hinting at the mechanism
underlying our empirical gains. We believe this work opens up new research
directions for the meta-optimization of meta-learning in a supervised learning
context.
