  Recently a number of CNN-based techniques were proposed to remove image
compression artifacts. As in other restoration applications, these techniques
all learn a mapping from decompressed patches to the original counterparts
under the ubiquitous $\ell_\infty$ metric. However, this approach is incapable
of restoring distinctive image details which may be statistical outliers but
have high semantic importance (e.g., tiny lesions in medical images). To
overcome this weakness, we propose to incorporate an $\ell_\infty$ fidelity
criterion in the design of neural network so that no small, distinctive
structures of the original image can be dropped or distorted. Experimental
results demonstrate that the proposed method outperforms the state-of-the-art
methods in $\ell_\infty$ error metric and perceptual quality, while being
competitive in $\ell_2$ error metric as well. It can restore subtle image
details that are otherwise destroyed or missed by other algorithms. Our
research suggests a new machine learning paradigm of ultra high fidelity image
compression that is ideally suited for applications in medicine, space, and
sciences.
