  This paper proposes a novel framework for generating lingual descriptions of
indoor scenes. Whereas substantial efforts have been made to tackle this
problem, previous approaches focusing primarily on generating a single sentence
for each image, which is not sufficient for describing complex scenes. We
attempt to go beyond this, by generating coherent descriptions with multiple
sentences. Our approach is distinguished from conventional ones in several
aspects: (1) a 3D visual parsing system that jointly infers objects,
attributes, and relations; (2) a generative grammar learned automatically from
training text; and (3) a text generation algorithm that takes into account the
coherence among sentences. Experiments on the augmented NYU-v2 dataset show
that our framework can generate natural descriptions with substantially higher
ROGUE scores compared to those produced by the baseline.
