  Although state-of-the-art (SOTA) CNNs achieve outstanding performance on
various tasks, their high computation demand and massive number of parameters
make it difficult to deploy these SOTA CNNs onto resource-constrained devices.
Previous works on CNN acceleration utilize low-rank approximation of the
original convolution layers to reduce computation cost. However, these methods
are very difficult to conduct upon sparse models, which limits execution
speedup since redundancies within the CNN model are not fully exploited. We
argue that kernel granularity decomposition can be conducted with low-rank
assumption while exploiting the redundancy within the remaining compact
coefficients. Based on this observation, we propose PENNI, a CNN model
compression framework that is able to achieve model compactness and hardware
efficiency simultaneously by (1) implementing kernel sharing in convolution
layers via a small number of basis kernels and (2) alternately adjusting bases
and coefficients with sparse constraints. Experiments show that we can prune
97% parameters and 92% FLOPs on ResNet18 CIFAR10 with no accuracy loss, and
achieve 44% reduction in run-time memory consumption and a 53% reduction in
inference latency.
