  In the architecture of deep learning models, inspired by biological neurons,
activation functions (AFs) play a pivotal role. They significantly influence
the performance of artificial neural networks. By modulating the non-linear
properties essential for learning complex patterns, AFs are fundamental in both
classification and regression tasks. This paper presents a comprehensive review
of various types of AFs, including fixed-shape, parametric, adaptive,
stochastic/probabilistic, non-standard, and ensemble/combining types. We begin
with a systematic taxonomy and detailed classification frameworks that
delineates the principal characteristics of AFs and organizes them based on
their structural and functional distinctions. Our in-depth analysis covers
primary groups such as sigmoid-based, ReLU-based, and ELU-based AFs, discussing
their theoretical foundations, mathematical formulations, and specific benefits
and limitations in different contexts. We also highlight key attributes of AFs
such as output range, monotonicity, and smoothness. Furthermore, we explore
miscellaneous AFs that do not conform to these categories but have shown unique
advantages in specialized applications. Non-standard AFs are also explored,
showcasing cutting-edge variations that challenge traditional paradigms and
offer enhanced adaptability and model performance. We examine strategies for
combining multiple AFs to leverage complementary properties. The paper
concludes with a comparative evaluation of 12 state-of-the-art AFs, using
rigorous statistical and experimental methodologies to assess their efficacy.
This analysis not only aids practitioners in selecting and designing the most
appropriate AFs for their specific deep learning tasks but also encourages
continued innovation in AF development within the machine learning community.
