  The performance of deep network learning strongly depends on the choice of
the non-linear activation function associated with each neuron. However,
deciding on the best activation is non-trivial, and the choice depends on the
architecture, hyper-parameters, and even on the dataset. Typically these
activations are fixed by hand before training. Here, we demonstrate how to
eliminate the reliance on first picking fixed activation functions by using
flexible parametric rational functions instead. The resulting Pad\'e Activation
Units (PAUs) can both approximate common activation functions and also learn
new ones while providing compact representations. Our empirical evidence shows
that end-to-end learning deep networks with PAUs can increase the predictive
performance. Moreover, PAUs pave the way to approximations with provable
robustness. https://github.com/ml-research/pau
