  In this paper, we present an approach, namely Lexical Semantic Image
Completion (LSIC), that may have potential applications in art, design, and
heritage conservation, among several others. Existing image completion
procedure is highly subjective by considering only visual context, which may
trigger unpredictable results which are plausible but not faithful to a
grounded knowledge. To permit both grounded and controllable completion
process, we advocate generating results faithful to both visual and lexical
semantic context, i.e., the description of leaving holes or blank regions in
the image (e.g., hole description). One major challenge for LSIC comes from
modeling and aligning the structure of visual-semantic context and translating
across different modalities. We term this process as structure completion,
which is realized by multi-grained reasoning blocks in our model. Another
challenge relates to the unimodal biases, which occurs when the model generates
plausible results without using the textual description. This can be true since
the annotated captions for an image are often semantically equivalent in
existing datasets, and thus there is only one paired text for a masked image in
training. We devise an unsupervised unpaired-creation learning path besides the
over-explored paired-reconstruction path, as well as a multi-stage training
strategy to mitigate the insufficiency of labeled data. We conduct extensive
quantitative and qualitative experiments as well as ablation studies, which
reveal the efficacy of our proposed LSIC.
