  In this paper, we design a Deep Dual-Domain ($\mathbf{D^3}$) based fast
restoration model to remove artifacts of JPEG compressed images. It leverages
the large learning capacity of deep networks, as well as the problem-specific
expertise that was hardly incorporated in the past design of deep
architectures. For the latter, we take into consideration both the prior
knowledge of the JPEG compression scheme, and the successful practice of the
sparsity-based dual-domain approach. We further design the One-Step Sparse
Inference (1-SI) module, as an efficient and light-weighted feed-forward
approximation of sparse coding. Extensive experiments verify the superiority of
the proposed $D^3$ model over several state-of-the-art methods. Specifically,
our best model is capable of outperforming the latest deep model for around 1
dB in PSNR, and is 30 times faster.
