  Deep convolutional networks (CNNs) have exhibited their potential in image
inpainting for producing plausible results. However, in most existing methods,
e.g., context encoder, the missing parts are predicted by propagating the
surrounding convolutional features through a fully connected layer, which
intends to produce semantically plausible but blurry result. In this paper, we
introduce a special shift-connection layer to the U-Net architecture, namely
Shift-Net, for filling in missing regions of any shape with sharp structures
and fine-detailed textures. To this end, the encoder feature of the known
region is shifted to serve as an estimation of the missing parts. A guidance
loss is introduced on decoder feature to minimize the distance between the
decoder feature after fully connected layer and the ground-truth encoder
feature of the missing parts. With such constraint, the decoder feature in
missing region can be used to guide the shift of encoder feature in known
region. An end-to-end learning algorithm is further developed to train the
Shift-Net. Experiments on the Paris StreetView and Places datasets demonstrate
the efficiency and effectiveness of our Shift-Net in producing sharper,
fine-detailed, and visually plausible results. The codes and pre-trained models
are available at https://github.com/Zhaoyi-Yan/Shift-Net.
