  We analyze the inductive bias of gradient descent for weight normalized
smooth homogeneous neural nets, when trained on exponential or cross-entropy
loss. We analyse both standard weight normalization (SWN) and exponential
weight normalization (EWN), and show that the gradient flow path with EWN is
equivalent to gradient flow on standard networks with an adaptive learning
rate. We extend these results to gradient descent, and establish asymptotic
relations between weights and gradients for both SWN and EWN. We also show that
EWN causes weights to be updated in a way that prefers asymptotic relative
sparsity. For EWN, we provide a finite-time convergence rate of the loss with
gradient flow and a tight asymptotic convergence rate with gradient descent. We
demonstrate our results for SWN and EWN on synthetic data sets. Experimental
results on simple datasets support our claim on sparse EWN solutions, even with
SGD. This demonstrates its potential applications in learning neural networks
amenable to pruning.
