  We propose a new high-performance activation function, Moderate Adaptive
Linear Units (MoLU), for the deep neural network. The MoLU is a simple,
beautiful and powerful activation function that can be a good main activation
function among hundreds of activation functions. Because the MoLU is made up of
the elementary functions, not only it is a infinite diffeomorphism (i.e. smooth
and infinitely differentiable over whole domains), but also it decreases
training time.
