  Deep learning provides a new avenue for image restoration, which demands a
delicate balance between fine-grained details and high-level contextualized
information during recovering the latent clear image. In practice, however,
existing methods empirically construct encapsulated end-to-end mapping networks
without deepening into the rationality, and neglect the intrinsic prior
knowledge of restoration task. To solve the above problems, inspired by
Taylor's Approximations, we unfold Taylor's Formula to construct a novel
framework for image restoration. We find the main part and the derivative part
of Taylor's Approximations take the same effect as the two competing goals of
high-level contextualized information and spatial details of image restoration
respectively. Specifically, our framework consists of two steps,
correspondingly responsible for the mapping and derivative functions. The
former first learns the high-level contextualized information and the later
combines it with the degraded input to progressively recover local high-order
spatial details. Our proposed framework is orthogonal to existing methods and
thus can be easily integrated with them for further improvement, and extensive
experiments demonstrate the effectiveness and scalability of our proposed
framework.
