  We study and analyze the fundamental aspects of noise propagation in
recurrent as well as deep, multi-layer networks. The main focus of our study
are neural networks in analogue hardware, yet the methodology provides insight
for networks in general. The system under study consists of noisy linear nodes,
and we investigate the signal-to-noise ratio at the network's outputs which is
the upper limit to such a system's computing accuracy. We consider additive and
multiplicative noise which can be purely local as well as correlated across
populations of neurons. This covers the chief internal-perturbations of
hardware networks and noise amplitudes were obtained from a physically
implemented recurrent neural network and therefore correspond to a real-world
system. Analytic solutions agree exceptionally well with numerical data,
enabling clear identification of the most critical components and aspects for
noise management. Focusing on linear nodes isolates the impact of network
connections and allows us to derive strategies for mitigating noise. Our work
is the starting point in addressing this aspect of analogue neural networks,
and our results identify notoriously sensitive points while simultaneously
highlighting the robustness of such computational systems.
