  Deep neural networks are usually initialized with random weights, with
adequately selected initial variance to ensure stable signal propagation during
training. However, selecting the appropriate variance becomes challenging
especially as the number of layers grows. In this work, we replace random
weight initialization with a fully deterministic initialization scheme, viz.,
ZerO, which initializes the weights of networks with only zeros and ones (up to
a normalization factor), based on identity and Hadamard transforms. Through
both theoretical and empirical studies, we demonstrate that ZerO is able to
train networks without damaging their expressivity. Applying ZerO on ResNet
achieves state-of-the-art performance on various datasets, including ImageNet,
which suggests random weights may be unnecessary for network initialization. In
addition, ZerO has many benefits, such as training ultra deep networks (without
batch-normalization), exhibiting low-rank learning trajectories that result in
low-rank and sparse solutions, and improving training reproducibility.
