  The rise in internet usage has led to the generation of massive amounts of
data, resulting in the adoption of various supervised and semi-supervised
machine learning algorithms, which can effectively utilize the colossal amount
of data to train models. However, before deploying these models in the real
world, these must be strictly evaluated on performance measures like worst-case
recall and satisfy constraints such as fairness. We find that current
state-of-the-art empirical techniques offer sub-optimal performance on these
practical, non-decomposable performance objectives. On the other hand, the
theoretical techniques necessitate training a new model from scratch for each
performance objective. To bridge the gap, we propose SelMix, a selective
mixup-based inexpensive fine-tuning technique for pre-trained models, to
optimize for the desired objective. The core idea of our framework is to
determine a sampling distribution to perform a mixup of features between
samples from particular classes such that it optimizes the given objective. We
comprehensively evaluate our technique against the existing empirical and
theoretically principled methods on standard benchmark datasets for imbalanced
classification. We find that proposed SelMix fine-tuning significantly improves
the performance for various practical non-decomposable objectives across
benchmarks.
