  Light-weight super-resolution (SR) models have received considerable
attention for their serviceability in mobile devices. Many efforts employ
network quantization to compress SR models. However, these methods suffer from
severe performance degradation when quantizing the SR models to ultra-low
precision (e.g., 2-bit and 3-bit) with the low-cost layer-wise quantizer. In
this paper, we identify that the performance drop comes from the contradiction
between the layer-wise symmetric quantizer and the highly asymmetric activation
distribution in SR models. This discrepancy leads to either a waste on the
quantization levels or detail loss in reconstructed images. Therefore, we
propose a novel activation quantizer, referred to as Dynamic Dual Trainable
Bounds (DDTB), to accommodate the asymmetry of the activations. Specifically,
DDTB innovates in: 1) A layer-wise quantizer with trainable upper and lower
bounds to tackle the highly asymmetric activations. 2) A dynamic gate
controller to adaptively adjust the upper and lower bounds at runtime to
overcome the drastically varying activation ranges over different samples.To
reduce the extra overhead, the dynamic gate controller is quantized to 2-bit
and applied to only part of the SR networks according to the introduced dynamic
intensity. Extensive experiments demonstrate that our DDTB exhibits significant
performance improvements in ultra-low precision. For example, our DDTB achieves
a 0.70dB PSNR increase on Urban100 benchmark when quantizing EDSR to 2-bit and
scaling up output images to x4. Code is at
\url{https://github.com/zysxmu/DDTB}.
