  In this paper, we investigate the negative effect of activation functions on
forward and backward propagation and how to counteract this effect. First, We
examine how activation functions affect the forward and backward propagation of
neural networks and derive a general form for gradient variance that extends
the previous work in this area. We try to use mini-batch statistics to
dynamically update the normalization factor to ensure the normalization
property throughout the training process, rather than only accounting for the
state of the neural network after weight initialization. Second, we propose
ANAct, a method that normalizes activation functions to maintain consistent
gradient variance across layers and demonstrate its effectiveness through
experiments. We observe that the convergence rate is roughly related to the
normalization property. We compare ANAct with several common activation
functions on CNNs and residual networks and show that ANAct consistently
improves their performance. For instance, normalized Swish achieves 1.4\%
higher top-1 accuracy than vanilla Swish on ResNet50 with the Tiny ImageNet
dataset and more than 1.2\% higher with CIFAR-100.
