  The Frank-Wolfe method and its extensions are well-suited for delivering
solutions with desirable structural properties, such as sparsity or low-rank
structure. We introduce a new variant of the Frank-Wolfe method that combines
Frank-Wolfe steps and steepest descent steps, as well as a novel modification
of the "Frank-Wolfe gap" to measure convergence in the non-convex case. We
further extend this method to incorporate in-face directions for preserving
structured solutions as well as block coordinate steps, and we demonstrate
computational guarantees in terms of the modified Frank-Wolfe gap for all of
these variants. We are particularly motivated by the application of this
methodology to the training of neural networks with sparse properties, and we
apply our block coordinate method to the problem of $\ell_1$ regularized neural
network training. We present the results of several numerical experiments on
both artificial and real datasets demonstrating significant improvements of our
method in training sparse neural networks.
