  Activation functions play a crucial role in neural networks because they are
the nonlinearities which have been attributed to the success story of deep
learning. One of the currently most popular activation functions is ReLU, but
several competitors have recently been proposed or 'discovered', including
LReLU functions and swish. While most works compare newly proposed activation
functions on few tasks (usually from image classification) and against few
competitors (usually ReLU), we perform the first large-scale comparison of 21
activation functions across eight different NLP tasks. We find that a largely
unknown activation function performs most stably across all tasks, the
so-called penalized tanh function. We also show that it can successfully
replace the sigmoid and tanh gates in LSTM cells, leading to a 2 percentage
point (pp) improvement over the standard choices on a challenging NLP task.
