  We propose stochastic, non-parametric activation functions that are fully
learnable and individual to each neuron. Complexity and the risk of overfitting
are controlled by placing a Gaussian process prior over these functions. The
result is the Gaussian process neuron, a probabilistic unit that can be used as
the basic building block for probabilistic graphical models that resemble the
structure of neural networks. The proposed model can intrinsically handle
uncertainties in its inputs and self-estimate the confidence of its
predictions. Using variational Bayesian inference and the central limit
theorem, a fully deterministic loss function is derived, allowing it to be
trained as efficiently as a conventional neural network using mini-batch
gradient descent. The posterior distribution of activation functions is
inferred from the training data alongside the weights of the network.
  The proposed model favorably compares to deep Gaussian processes, both in
model complexity and efficiency of inference. It can be directly applied to
recurrent or convolutional network structures, allowing its use in audio and
image processing tasks.
  As an preliminary empirical evaluation we present experiments on regression
and classification tasks, in which our model achieves performance comparable to
or better than a Dropout regularized neural network with a fixed activation
function. Experiments are ongoing and results will be added as they become
available.
