  Activation functions (AF) are necessary components of neural networks that
allow approximation of functions, but AFs in current use are usually simple
monotonically increasing functions. In this paper, we propose trainable
compound AF (TCA) composed of a sum of shifted and scaled simple AFs. TCAs
increase the effectiveness of networks with fewer parameters compared to added
layers. TCAs have a special interpretation in generative networks because they
effectively estimate the marginal distributions of each dimension of the data
using a mixture distribution, reducing modality and making linear dimension
reduction more effective. When used in restricted Boltzmann machines (RBMs),
they result in a novel type of RBM with mixture-based stochastic units.
Improved performance is demonstrated in experiments using RBMs, deep belief
networks (DBN), projected belief networks (PBN), and variational auto-encoders
(VAE).
