  In many applications where collecting data is expensive, for example
neuroscience or medical imaging, the sample size is typically small compared to
the feature dimension. It is challenging in this setting to train expressive,
non-linear models without overfitting. These datasets call for intelligent
regularization that exploits known structure, such as correlations between the
features arising from the measurement device. However, existing structured
regularizers need specially crafted solvers, which are difficult to apply to
complex models. We propose a new regularizer specifically designed to leverage
structure in the data in a way that can be applied efficiently to complex
models. Our approach relies on feature grouping, using a fast clustering
algorithm inside a stochastic gradient descent loop: given a family of feature
groupings that capture feature covariations, we randomly select these groups at
each iteration. We show that this approach amounts to enforcing a denoising
regularizer on the solution. The method is easy to implement in many model
architectures, such as fully connected neural networks, and has a linear
computational cost. We apply this regularizer to a real-world fMRI dataset and
the Olivetti Faces datasets. Experiments on both datasets demonstrate that the
proposed approach produces models that generalize better than those trained
with conventional regularizers, and also improves convergence speed.
