  This paper presents a three-tier modality alignment approach to learning
text-image joint embedding, coined as JEMA, for cross-modal retrieval of
cooking recipes and food images. The first tier improves recipe text embedding
by optimizing the LSTM networks with term extraction and ranking enhanced
sequence patterns, and optimizes the image embedding by combining the
ResNeXt-101 image encoder with the category embedding using wideResNet-50 with
word2vec. The second tier modality alignment optimizes the textual-visual joint
embedding loss function using a double batch-hard triplet loss with soft-margin
optimization. The third modality alignment incorporates two types of
cross-modality alignments as the auxiliary loss regularizations to further
reduce the alignment errors in the joint learning of the two modality-specific
embedding functions. The category-based cross-modal alignment aims to align the
image category with the recipe category as a loss regularization to the joint
embedding. The cross-modal discriminator-based alignment aims to add the
visual-textual embedding distribution alignment to further regularize the joint
embedding loss. Extensive experiments with the one-million recipes benchmark
dataset Recipe1M demonstrate that the proposed JEMA approach outperforms the
state-of-the-art cross-modal embedding methods for both image-to-recipe and
recipe-to-image retrievals.
