  Activation functions are core components of all deep learning architectures.
Currently, the most popular activation functions are smooth ReLU variants like
GELU and SiLU. These are self-gated activation functions where the range of the
gating function is between zero and one. In this paper, we explore the
viability of using arctan as a gating mechanism. A self-gated activation
function that uses arctan as its gating function has a monotonically increasing
first derivative. To make this activation function competitive, it is necessary
to introduce a trainable parameter for every MLP block to expand the range of
the gating function beyond zero and one. We find that this technique also
improves existing self-gated activation functions. We conduct an empirical
evaluation of Expanded ArcTan Linear Unit (xATLU), Expanded GELU (xGELU), and
Expanded SiLU (xSiLU) and show that they outperform existing activation
functions within a transformer architecture. Additionally, expanded gating
ranges show promising results in improving first-order Gated Linear Units
(GLU).
