  We propose a novel solution for unpaired image-to-image (I2I) translation. To
translate complex images with a wide range of objects to a different domain,
recent approaches often use the object annotations to perform per-class
source-to-target style mapping. However, there remains a point for us to
exploit in the I2I. An object in each class consists of multiple components,
and all the sub-object components have different characteristics. For example,
a car in CAR class consists of a car body, tires, windows and head and tail
lamps, etc., and they should be handled separately for realistic I2I
translation. The simplest solution to the problem will be to use more detailed
annotations with sub-object component annotations than the simple object
annotations, but it is not possible. The key idea of this paper is to bypass
the sub-object component annotations by leveraging the original style of the
input image because the original style will include the information about the
characteristics of the sub-object components. Specifically, for each pixel, we
use not only the per-class style gap between the source and target domains but
also the pixel's original style to determine the target style of a pixel. To
this end, we present Style Harmonization for unpaired I2I translation (SHUNIT).
Our SHUNIT generates a new style by harmonizing the target domain style
retrieved from a class memory and an original source image style. Instead of
direct source-to-target style mapping, we aim for source and target styles
harmonization. We validate our method with extensive experiments and achieve
state-of-the-art performance on the latest benchmark sets. The source code is
available online: https://github.com/bluejangbaljang/SHUNIT.
