  Arguably the biggest challenge in applying neural networks is tuning the
hyperparameters, in particular the learning rate. The sensitivity to the
learning rate is due to the reliance on backpropagation to train the network.
In this paper we present the first application of Implicit Stochastic Gradient
Descent (ISGD) to train neural networks, a method known in convex optimization
to be unconditionally stable and robust to the learning rate. Our key
contribution is a novel layer-wise approximation of ISGD which makes its
updates tractable for neural networks. Experiments show that our method is more
robust to high learning rates and generally outperforms standard
backpropagation on a variety of tasks.
