  Recent works have highlighted scale invariance or symmetry that is present in
the weight space of a typical deep network and the adverse effect that it has
on the Euclidean gradient based stochastic gradient descent optimization. In
this work, we show that these and other commonly used deep networks, such as
those which use a max-pooling and sub-sampling layer, possess more complex
forms of symmetry arising from scaling based reparameterization of the network
weights. We then propose two symmetry-invariant gradient based weight updates
for stochastic gradient descent based learning. Our empirical evidence based on
the MNIST dataset shows that these updates improve the test performance without
sacrificing the computational efficiency of the weight updates. We also show
the results of training with one of the proposed weight updates on an image
segmentation problem.
