  A lot of recent progress has been made in ultra low-bit quantization,
promising significant improvements in latency, memory footprint and energy
consumption on edge devices. Quantization methods such as Learned Step Size
Quantization can achieve model accuracy that is comparable to full-precision
floating-point baselines even with sub-byte quantization. However, it is
extremely challenging to deploy these ultra low-bit quantized models on
mainstream CPU devices because commodity SIMD (Single Instruction, Multiple
Data) hardware typically supports no less than 8-bit precision. To overcome
this limitation, we propose DeepGEMM, a lookup table based approach for the
execution of ultra low-precision convolutional neural networks on SIMD
hardware. The proposed method precomputes all possible products of weights and
activations, stores them in a lookup table, and efficiently accesses them at
inference time to avoid costly multiply-accumulate operations. Our 2-bit
implementation outperforms corresponding 8-bit integer kernels in the QNNPACK
framework by up to 1.74x on x86 platforms.
