  Weights initialization in deep neural networks have a strong impact on the
speed of converge of the learning map. Recent studies have shown that in the
case of random initializations, a chaos/order phase transition occur in the
space of variances of random weights and biases. Experiments then had shown
that large improvements can be made, in terms of the training speed, if a
neural network is initialized on values along the critical line of such phase
transition. In this contribution, we show evidence that the scaling property
exhibited by physical systems at criticality, is also present in untrained
feedforward networks with random weights initialization at the critical line.
Additionally, we suggest an additional data-resizing symmetry, which is
directly inherited from the scaling symmetry at criticality.
