  Recipe generation from food images and ingredients is a challenging task,
which requires the interpretation of the information from another modality.
Different from the image captioning task, where the captions usually have one
sentence, cooking instructions contain multiple sentences and have obvious
structures. To help the model capture the recipe structure and avoid missing
some cooking details, we propose a novel framework: Decomposing Generation
Networks (DGN) with structure prediction, to get more structured and complete
recipe generation outputs. Specifically, we split each cooking instruction into
several phases, and assign different sub-generators to each phase. Our approach
includes two novel ideas: (i) learning the recipe structures with the global
structure prediction component and (ii) producing recipe phases in the
sub-generator output component based on the predicted structure. Extensive
experiments on the challenging large-scale Recipe1M dataset validate the
effectiveness of our proposed model, which improves the performance over the
state-of-the-art results.
