  The Backpropagation algorithm relies on the abstraction of using a neural
model that gets rid of the notion of time, since the input is mapped
instantaneously to the output. In this paper, we claim that this abstraction of
ignoring time, along with the abrupt input changes that occur when feeding the
training set, are in fact the reasons why, in some papers, Backprop biological
plausibility is regarded as an arguable issue. We show that as soon as a deep
feedforward network operates with neurons with time-delayed response, the
backprop weight update turns out to be the basic equation of a biologically
plausible diffusion process based on forward-backward waves. We also show that
such a process very well approximates the gradient for inputs that are not too
fast with respect to the depth of the network. These remarks somewhat disclose
the diffusion process behind the backprop equation and leads us to interpret
the corresponding algorithm as a degeneration of a more general diffusion
process that takes place also in neural networks with cyclic connections.
