  Element-wise activation functions play a critical role in deep neural
networks via affecting the expressivity power and the learning dynamics.
Learning-based activation functions have recently gained increasing attention
and success. We propose a new perspective of learnable activation function
through formulating them with element-wise attention mechanism. In each network
layer, we devise an attention module which learns an element-wise, sign-based
attention map for the pre-activation feature map. The attention map scales an
element based on its sign. Adding the attention module with a rectified linear
unit (ReLU) results in an amplification of positive elements and a suppression
of negative ones, both with learned, data-adaptive parameters. We coin the
resulting activation function Attention-based Rectified Linear Unit (AReLU).
The attention module essentially learns an element-wise residue of the
activated part of the input, as ReLU can be viewed as an identity
transformation. This makes the network training more resistant to gradient
vanishing. The learned attentive activation leads to well-focused activation of
relevant regions of a feature map. Through extensive evaluations, we show that
AReLU significantly boosts the performance of most mainstream network
architectures with only two extra learnable parameters per layer introduced.
Notably, AReLU facilitates fast network training under small learning rates,
which makes it especially suited in the case of transfer learning and meta
learning. Our source code has been released (see
https://github.com/densechen/AReLU).
