  We introduce LUCSS, a language-based system for interactive col- orization of
scene sketches, based on their semantic understanding. LUCSS is built upon deep
neural networks trained via a large-scale repository of scene sketches and
cartoon-style color images with text descriptions. It con- sists of three
sequential modules. First, given a scene sketch, the segmenta- tion module
automatically partitions an input sketch into individual object instances.
Next, the captioning module generates the text description with spatial
relationships based on the instance-level segmentation results. Fi- nally, the
interactive colorization module allows users to edit the caption and produce
colored images based on the altered caption. Our experiments show the
effectiveness of our approach and the desirability of its compo- nents to
alternative choices.
