  Generating images with conditional descriptions gains increasing interests in
recent years. However, existing conditional inputs are suffering from either
unstructured forms (captions) or limited information and expensive labeling
(scene graphs). For a targeted scene, the core items, objects, are usually
definite while their interactions are flexible and hard to clearly define.
Thus, we introduce a more rational setting, generating a realistic image from
the objects and captions. Under this setting, objects explicitly define the
critical roles in the targeted images and captions implicitly describe their
rich attributes and connections. Correspondingly, a MOC-GAN is proposed to mix
the inputs of two modalities to generate realistic images. It firstly infers
the implicit relations between object pairs from the captions to build a
hidden-state scene graph. So a multi-layer representation containing objects,
relations and captions is constructed, where the scene graph provides the
structures of the scene and the caption provides the image-level guidance. Then
a cascaded attentive generative network is designed to coarse-to-fine generate
phrase patch by paying attention to the most relevant words in the caption. In
addition, a phrase-wise DAMSM is proposed to better supervise the fine-grained
phrase-patch consistency. On COCO dataset, our method outperforms the
state-of-the-art methods on both Inception Score and FID while maintaining high
visual quality. Extensive experiments demonstrate the unique features of our
proposed method.
