  The Regularized Nonlinear Acceleration (RNA) algorithm is an acceleration
method capable of improving the rate of convergence of many optimization
schemes such as gradient descend, SAGA or SVRG. Until now, its analysis is
limited to convex problems, but empirical observations shows that RNA may be
extended to wider settings. In this paper, we investigate further the benefits
of RNA when applied to neural networks, in particular for the task of image
recognition on CIFAR10 and ImageNet. With very few modifications of exiting
frameworks, RNA improves slightly the optimization process of CNNs, after
training.
