  Most deep neural networks (DNNs) consist fundamentally of convolutional
and/or fully connected layers, wherein the linear transform can be cast as the
product between a filter matrix and a data matrix obtained by arranging feature
tensors into columns. The lately proposed deformable butterfly (DeBut)
decomposes the filter matrix into generalized, butterflylike factors, thus
achieving network compression orthogonal to the traditional ways of pruning or
low-rank decomposition. This work reveals an intimate link between DeBut and a
systematic hierarchy of depthwise and pointwise convolutions, which explains
the empirically good performance of DeBut layers. By developing an automated
DeBut chain generator, we show for the first time the viability of homogenizing
a DNN into all DeBut layers, thus achieving an extreme sparsity and
compression. Various examples and hardware benchmarks verify the advantages of
All-DeBut networks. In particular, we show it is possible to compress a
PointNet to < 5% parameters with < 5% accuracy drop, a record not achievable by
other compression schemes.
