  This paper investigates efficient deep neural networks (DNNs) to replace
dense unstructured weight matrices with structured ones that possess desired
properties. The challenge arises because the optimal weight matrix structure in
popular neural network models is obscure in most cases and may vary from layer
to layer even in the same network. Prior structured matrices proposed for
efficient DNNs were mostly hand-crafted without a generalized framework to
systematically learn them. To address this issue, we propose a generalized and
differentiable framework to learn efficient structures of weight matrices by
gradient descent. We first define a new class of structured matrices that
covers a wide range of structured matrices in the literature by adjusting the
structural parameters. Then, the frequency-domain differentiable
parameterization scheme based on the Gaussian-Dirichlet kernel is adopted to
learn the structural parameters by proximal gradient descent. On the image and
language tasks, our method learns efficient DNNs with structured matrices,
achieving lower complexity and/or higher performance than prior approaches that
employ low-rank, block-sparse, or block-low-rank matrices.
