  Most of the recent successful applications of neural networks have been based
on training with gradient descent updates. However, for some small networks,
other mirror descent updates learn provably more efficiently when the target is
sparse. We present a general framework for casting a mirror descent update as a
gradient descent update on a different set of parameters. In some cases, the
mirror descent reparameterization can be described as training a modified
network with standard backpropagation. The reparameterization framework is
versatile and covers a wide range of mirror descent updates, even cases where
the domain is constrained. Our construction for the reparameterization argument
is done for the continuous versions of the updates. Finding general criteria
for the discrete versions to closely track their continuous counterparts
remains an interesting open problem.
