  We study the design of deep architectures for lossy image compression. We
present two architectural recipes in the context of multi-stage progressive
encoders and empirically demonstrate their importance on compression
performance. Specifically, we show that: (a) predicting the original image data
from residuals in a multi-stage progressive architecture facilitates learning
and leads to improved performance at approximating the original content and (b)
learning to inpaint (from neighboring image pixels) before performing
compression reduces the amount of information that must be stored to achieve a
high-quality approximation. Incorporating these design choices in a baseline
progressive encoder yields an average reduction of over $60\%$ in file size
with similar quality compared to the original residual encoder.
