  With the motive of training all the parameters of a neural network, we study
why and when one can achieve this by iteratively creating, training, and
combining randomly selected subnetworks. Such scenarios have either implicitly
or explicitly emerged in the recent literature: see e.g., the Dropout family of
regularization techniques, or some distributed ML training protocols that
reduce communication/computation complexities, such as the Independent Subnet
Training protocol. While these methods are studied empirically and utilized in
practice, they often enjoy partial or no theoretical support, especially when
applied on neural network-based objectives.
  In this manuscript, our focus is on overparameterized single hidden layer
neural networks with ReLU activations in the lazy training regime. By carefully
analyzing $i)$ the subnetworks' neural tangent kernel, $ii)$ the surrogate
functions' gradient, and $iii)$ how we sample and combine the surrogate
functions, we prove linear convergence rate of the training error -- up to a
neighborhood around the optimal point -- for an overparameterized single-hidden
layer perceptron with a regression loss. Our analysis reveals a dependency of
the size of the neighborhood around the optimal point on the number of
surrogate models and the number of local training steps for each selected
subnetwork. Moreover, the considered framework generalizes and provides new
insights on dropout training, multi-sample dropout training, as well as
Independent Subnet Training; for each case, we provide convergence results as
corollaries of our main theorem.
