  To adopt convolutional neural networks (CNN) for a range of
resource-constrained targets, it is necessary to compress the CNN models by
performing quantization, whereby precision representation is converted to a
lower bit representation. To overcome problems such as sensitivity of the
training dataset, high computational requirements, and large time consumption,
post-training quantization methods that do not require retraining have been
proposed. In addition, to compensate for the accuracy drop without retraining,
previous studies on post-training quantization have proposed several
complementary methods: calibration, schemes, clipping, granularity, and
mixed-precision. To generate a quantized model with minimal error, it is
necessary to study all possible combinations of the methods because each of
them is complementary and the CNN models have different characteristics.
However, an exhaustive or a heuristic search is either too time-consuming or
suboptimal. To overcome this challenge, we propose an auto-tuner known as
Quantune, which builds a gradient tree boosting model to accelerate the search
for the configurations of quantization and reduce the quantization error. We
evaluate and compare Quantune with the random, grid, and genetic algorithms.
The experimental results show that Quantune reduces the search time for
quantization by approximately 36.5x with an accuracy loss of 0.07 ~ 0.65%
across six CNN models, including the fragile ones (MobileNet, SqueezeNet, and
ShuffleNet). To support multiple targets and adopt continuously evolving
quantization works, Quantune is implemented on a full-fledged compiler for deep
learning as an open-sourced project.
