  Deep neural networks unlocked a vast range of new applications by solving
tasks of which many were previously deemed as reserved to higher human
intelligence. One of the developments enabling this success was a boost in
computing power provided by special purpose hardware, such as graphic or tensor
processing units. However, these do not leverage fundamental features of neural
networks like parallelism and analog state variables. Instead, they emulate
neural networks relying on binary computing, which results in unsustainable
energy consumption and comparatively low speed. Fully parallel and analogue
hardware promises to overcome these challenges, yet the impact of analogue
neuron noise and its propagation, i.e. accumulation, threatens rendering such
approaches inept. Here, we determine for the first time the propagation of
noise in deep neural networks comprising noisy nonlinear neurons in trained
fully connected layers. We study additive and multiplicative as well as
correlated and uncorrelated noise, and develop analytical methods that predict
the noise level in any layer of symmetric deep neural networks or deep neural
networks trained with back propagation. We find that noise accumulation is
generally bound, and adding additional network layers does not worsen the
signal to noise ratio beyond a limit. Most importantly, noise accumulation can
be suppressed entirely when neuron activation functions have a slope smaller
than unity. We therefore developed the framework for noise in fully connected
deep neural networks implemented in analog systems, and identify criteria
allowing engineers to design noise-resilient novel neural network hardware.
