  In this work, we present a new derivative-free optimization method and
investigate its use for training neural networks. Our method is motivated by
the Ensemble Kalman Filter (EnKF), which has been used successfully for solving
optimization problems that involve large-scale, highly nonlinear dynamical
systems. A key benefit of the EnKF method is that it requires only the
evaluation of the forward propagation but not its derivatives. Hence, in the
context of neural networks, it alleviates the need for back propagation and
reduces the memory consumption dramatically. However, the method is not a pure
"black-box" global optimization heuristic as it efficiently utilizes the
structure of typical learning problems. Promising first results of the EnKF for
training deep neural networks have been presented recently by Kovachki and
Stuart. We propose an important modification of the EnKF that enables us to
prove convergence of our method to the minimizer of a strongly convex function.
Our method also bears similarity with implicit filtering and we demonstrate its
potential for minimizing highly oscillatory functions using a simple example.
Further, we provide numerical examples that demonstrate the potential of our
method for training deep neural networks.
