  Existing deep learning models separate JPEG artifacts suppression from the
decoding protocol as independent task. In this work, we take one step forward
to design a true end-to-end heterogeneous residual convolutional neural network
(HR-CNN) with spectrum decomposition and heterogeneous reconstruction
mechanism. Benefitting from the full CNN architecture and GPU acceleration, the
proposed model considerably improves the reconstruction efficiency. Numerical
experiments show that the overall reconstruction speed reaches to the same
magnitude of the standard CPU JPEG decoding protocol, while both decoding and
artifacts suppression are completed together. We formulate the JPEG artifacts
suppression task as an interactive process of decoding and image detail
reconstructions. A heterogeneous, fully convolutional, mechanism is proposed to
particularly address the uncorrelated nature of different spectral channels.
Directly starting from the JPEG code in k-space, the network first extracts the
spectral samples channel by channel, and restores the spectral snapshots with
expanded throughput. These intermediate snapshots are then heterogeneously
decoded and merged into the pixel space image. A cascaded residual learning
segment is designed to further enhance the image details. Experiments verify
that the model achieves outstanding performance in JPEG artifacts suppression,
while its full convolutional operations and elegant network structure offers
higher computational efficiency for practical online usage compared with other
deep learning models on this topic.
