  Recent studies have revealed that, beyond conventional accuracy, calibration
should also be considered for training modern deep neural networks. To address
miscalibration during learning, some methods have explored different penalty
functions as part of the learning objective, alongside a standard
classification loss, with a hyper-parameter controlling the relative
contribution of each term. Nevertheless, these methods share two major
drawbacks: 1) the scalar balancing weight is the same for all classes,
hindering the ability to address different intrinsic difficulties or imbalance
among classes; and 2) the balancing weight is usually fixed without an adaptive
strategy, which may prevent from reaching the best compromise between accuracy
and calibration, and requires hyper-parameter search for each application. We
propose Class Adaptive Label Smoothing (CALS) for calibrating deep networks,
which allows to learn class-wise multipliers during training, yielding a
powerful alternative to common label smoothing penalties. Our method builds on
a general Augmented Lagrangian approach, a well-established technique in
constrained optimization, but we introduce several modifications to tailor it
for large-scale, class-adaptive training. Comprehensive evaluation and multiple
comparisons on a variety of benchmarks, including standard and long-tailed
image classification, semantic segmentation, and text classification,
demonstrate the superiority of the proposed method. The code is available at
https://github.com/by-liu/CALS.
