  We present a novel deep learning architecture in which the convolution
operation leverages heterogeneous kernels. The proposed HetConv (Heterogeneous
Kernel-Based Convolution) reduces the computation (FLOPs) and the number of
parameters as compared to standard convolution operation while still
maintaining representational efficiency. To show the effectiveness of our
proposed convolution, we present extensive experimental results on the standard
convolutional neural network (CNN) architectures such as VGG \cite{vgg2014very}
and ResNet \cite{resnet}. We find that after replacing the standard
convolutional filters in these architectures with our proposed HetConv filters,
we achieve 3X to 8X FLOPs based improvement in speed while still maintaining
(and sometimes improving) the accuracy. We also compare our proposed
convolutions with group/depth wise convolutions and show that it achieves more
FLOPs reduction with significantly higher accuracy.
