  Artificial neural networks typically have a fixed, non-linear activation
function at each neuron. We have designed a novel form of piecewise linear
activation function that is learned independently for each neuron using
gradient descent. With this adaptive activation function, we are able to
improve upon deep neural network architectures composed of static rectified
linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%),
CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs
boson decay modes.
