  The activation function plays a crucial role in model optimisation, yet the
optimal choice remains unclear. For example, the Sigmoid activation is the
de-facto activation in balanced classification tasks, however, in imbalanced
classification, it proves inappropriate due to bias towards frequent classes.
In this work, we delve deeper in this phenomenon by performing a comprehensive
statistical analysis in the classification and intermediate layers of both
balanced and imbalanced networks and we empirically show that aligning the
activation function with the data distribution, enhances the performance in
both balanced and imbalanced tasks. To this end, we propose the Adaptive
Parametric Activation (APA) function, a novel and versatile activation function
that unifies most common activation functions under a single formula. APA can
be applied in both intermediate layers and attention layers, significantly
outperforming the state-of-the-art on several imbalanced benchmarks such as
ImageNet-LT, iNaturalist2018, Places-LT, CIFAR100-LT and LVIS and balanced
benchmarks such as ImageNet1K, COCO and V3DET. The code is available at
https://github.com/kostas1515/AGLU.
