  Adjusting the photo color to associate with some design elements is an
essential way for a graphic design to effectively deliver its message and make
it aesthetically pleasing. However, existing tools and previous works face a
dilemma between the ease of use and level of expressiveness. To this end, we
introduce an interactive language-based approach for photo recoloring, which
provides an intuitive system that can assist both experts and novices on
graphic design. Given a graphic design containing a photo that needs to be
recolored, our model can predict the source colors and the target regions, and
then recolor the target regions with the source colors based on the given
language-based instruction. The multi-granularity of the instruction allows
diverse user intentions. The proposed novel task faces several unique
challenges, including: 1) color accuracy for recoloring with exactly the same
color from the target design element as specified by the user; 2)
multi-granularity instructions for parsing instructions correctly to generate a
specific result or multiple plausible ones; and 3) locality for recoloring in
semantically meaningful local regions to preserve original image semantics. To
address these challenges, we propose a model called LangRecol with two main
components: the language-based source color prediction module and the
semantic-palette-based photo recoloring module. We also introduce an approach
for generating a synthetic graphic design dataset with instructions to enable
model training. We evaluate our model via extensive experiments and user
studies. We also discuss several practical applications, showing the
effectiveness and practicality of our approach. Code and data for this paper
are at: https://zhenwwang.github.io/langrecol.
