  We propose a new algorithm for color transfer between images that have
perceptually similar semantic structures. We aim to achieve a more accurate
color transfer that leverages semantically-meaningful dense correspondence
between images. To accomplish this, our algorithm uses neural representations
for matching. Additionally, the color transfer should be spatially variant and
globally coherent. Therefore, our algorithm optimizes a local linear model for
color transfer satisfying both local and global constraints. Our proposed
approach jointly optimizes matching and color transfer, adopting a
coarse-to-fine strategy. The proposed method can be successfully extended from
one-to-one to one-to-many color transfer. The latter further addresses the
problem of mismatching elements of the input image. We validate our proposed
method by testing it on a large variety of image content.
