  We propose the first deep learning approach for exemplar-based local
colorization. Given a reference color image, our convolutional neural network
directly maps a grayscale image to an output colorized image. Rather than using
hand-crafted rules as in traditional exemplar-based methods, our end-to-end
colorization network learns how to select, propagate, and predict colors from
the large-scale data. The approach performs robustly and generalizes well even
when using reference images that are unrelated to the input grayscale image.
More importantly, as opposed to other learning-based colorization methods, our
network allows the user to achieve customizable results by simply feeding
different references. In order to further reduce manual effort in selecting the
references, the system automatically recommends references with our proposed
image retrieval algorithm, which considers both semantic and luminance
information. The colorization can be performed fully automatically by simply
picking the top reference suggestion. Our approach is validated through a user
study and favorable quantitative comparisons to the-state-of-the-art methods.
Furthermore, our approach can be naturally extended to video colorization. Our
code and models will be freely available for public use.
