  Initialising the synaptic weights of artificial neural networks (ANNs) with
orthogonal matrices is known to alleviate vanishing and exploding gradient
problems. A major objection against such initialisation schemes is that they
are deemed biologically implausible as they mandate factorization techniques
that are difficult to attribute to a neurobiological process. This paper
presents two initialisation schemes that allow a network to naturally evolve
its weights to form orthogonal matrices, provides theoretical analysis that
pre-training orthogonalisation always converges, and empirically confirms that
the proposed schemes outperform randomly initialised recurrent and feedforward
networks.
