  Recent work on discrete generative priors, in the form of codebooks, has
shown exciting performance for image reconstruction and restoration, as the
discrete prior space spanned by the codebooks increases the robustness against
diverse image degradations. Nevertheless, these methods require separate
training of codebooks for different image categories, which limits their use to
specific image categories only (e.g. face, architecture, etc.), and fail to
handle arbitrary natural images. In this paper, we propose AdaCode for learning
image-adaptive codebooks for class-agnostic image restoration. Instead of
learning a single codebook for each image category, we learn a set of basis
codebooks. For a given input image, AdaCode learns a weight map with which we
compute a weighted combination of these basis codebooks for adaptive image
restoration. Intuitively, AdaCode is a more flexible and expressive discrete
generative prior than previous work. Experimental results demonstrate that
AdaCode achieves state-of-the-art performance on image reconstruction and
restoration tasks, including image super-resolution and inpainting.
