  Real-world datasets exhibit imbalances of varying types and degrees. Several
techniques based on re-weighting and margin adjustment of loss are often used
to enhance the performance of neural networks, particularly on minority
classes. In this work, we analyze the class-imbalanced learning problem by
examining the loss landscape of neural networks trained with re-weighting and
margin-based techniques. Specifically, we examine the spectral density of
Hessian of class-wise loss, through which we observe that the network weights
converge to a saddle point in the loss landscapes of minority classes.
Following this observation, we also find that optimization methods designed to
escape from saddle points can be effectively used to improve generalization on
minority classes. We further theoretically and empirically demonstrate that
Sharpness-Aware Minimization (SAM), a recent technique that encourages
convergence to a flat minima, can be effectively used to escape saddle points
for minority classes. Using SAM results in a 6.2\% increase in accuracy on the
minority classes over the state-of-the-art Vector Scaling Loss, leading to an
overall average increase of 4\% across imbalanced datasets. The code is
available at: https://github.com/val-iisc/Saddle-LongTail.
