  Tucker decomposition is one of the SOTA CNN model compression techniques.
However, unlike the FLOPs reduction, we observe very limited inference time
reduction with Tucker-compressed models using existing GPU software such as
cuDNN. To this end, we propose an efficient end-to-end framework that can
generate highly accurate and compact CNN models via Tucker decomposition and
optimized inference code on GPUs. Specifically, we propose an ADMM-based
training algorithm that can achieve highly accurate Tucker-format models. We
also develop a high-performance kernel for Tucker-format convolutions and
analytical performance models to guide the selection of execution parameters.
We further propose a co-design framework to determine the proper Tucker ranks
driven by practical inference time (rather than FLOPs). Our evaluation on five
modern CNNs with A100 demonstrates that our compressed models with our
optimized code achieve up to 2.21X speedup over cuDNN, 1.12X speedup over TVM,
and 3.27X over the original models using cuDNN with at most 0.05% accuracy
loss.
