  The vanishing gradient problem was a major obstacle for the success of deep
learning. In recent years it was gradually alleviated through multiple
different techniques. However the problem was not really overcome in a
fundamental way, since it is inherent to neural networks with activation
functions based on dot products. In a series of papers, we are going to analyze
alternative neural network structures which are not based on dot products. In
this first paper, we revisit neural networks built up of layers based on
distance measures and Gaussian activation functions. These kinds of networks
were only sparsely used in the past since they are hard to train when using
plain stochastic gradient descent methods. We show that by using Root Mean
Square Propagation (RMSProp) it is possible to efficiently learn multi-layer
neural networks. Furthermore we show that when appropriately initialized these
kinds of neural networks suffer much less from the vanishing and exploding
gradient problem than traditional neural networks even for deep networks.
