  Activation functions are essential to deep learning networks. Popular and
versatile activation functions are mostly monotonic functions, some
non-monotonic activation functions are being explored and show promising
performance. But by introducing non-monotonicity, they also alter the positive
input, which is proved to be unnecessary by the success of ReLU and its
variants. In this paper, we double down on the non-monotonic activation
functions' development and propose the Saturated Gaussian Error Linear Units by
combining the characteristics of ReLU and non-monotonic activation functions.
We present three new activation functions built with our proposed method:
SGELU, SSiLU, and SMish, which are composed of the negative portion of GELU,
SiLU, and Mish, respectively, and ReLU's positive portion. The results of image
classification experiments on CIFAR-100 indicate that our proposed activation
functions are highly effective and outperform state-of-the-art baselines across
multiple deep learning architectures.
