  The inductive bias of a neural network is largely determined by the
architecture and the training algorithm. To achieve good generalization, how to
effectively train a neural network is of great importance. We propose a novel
orthogonal over-parameterized training (OPT) framework that can provably
minimize the hyperspherical energy which characterizes the diversity of neurons
on a hypersphere. By maintaining the minimum hyperspherical energy during
training, OPT can greatly improve the empirical generalization. Specifically,
OPT fixes the randomly initialized weights of the neurons and learns an
orthogonal transformation that applies to these neurons. We consider multiple
ways to learn such an orthogonal transformation, including unrolling
orthogonalization algorithms, applying orthogonal parameterization, and
designing orthogonality-preserving gradient descent. For better scalability, we
propose the stochastic OPT which performs orthogonal transformation
stochastically for partial dimensions of neurons. Interestingly, OPT reveals
that learning a proper coordinate system for neurons is crucial to
generalization. We provide some insights on why OPT yields better
generalization. Extensive experiments validate the superiority of OPT over the
standard training.
