  While existing makeup style transfer models perform an image synthesis whose
results cannot be explicitly controlled, the ability to modify makeup color
continuously is a desirable property for virtual try-on applications. We
propose a new formulation for the makeup style transfer task, with the
objective to learn a color controllable makeup style synthesis. We introduce
CA-GAN, a generative model that learns to modify the color of specific objects
(e.g. lips or eyes) in the image to an arbitrary target color while preserving
background. Since color labels are rare and costly to acquire, our method
leverages weakly supervised learning for conditional GANs. This enables to
learn a controllable synthesis of complex objects, and only requires a weak
proxy of the image attribute that we desire to modify. Finally, we present for
the first time a quantitative analysis of makeup style transfer and color
control performance.
