  Direct computer vision based-nutrient content estimation is a demanding task,
due to deformation and occlusions of ingredients, as well as high intra-class
and low inter-class variability between meal classes. In order to tackle these
issues, we propose a system for recipe retrieval from images. The recipe
information can subsequently be used to estimate the nutrient content of the
meal. In this study, we utilize the multi-modal Recipe1M dataset, which
contains over 1 million recipes accompanied by over 13 million images. The
proposed model can operate as a first step in an automatic pipeline for the
estimation of nutrition content by supporting hints related to ingredient and
instruction. Through self-attention, our model can directly process raw recipe
text, making the upstream instruction sentence embedding process redundant and
thus reducing training time, while providing desirable retrieval results.
Furthermore, we propose the use of an ingredient attention mechanism, in order
to gain insight into which instructions, parts of instructions or single
instruction words are of importance for processing a single ingredient within a
certain recipe. Attention-based recipe text encoding contributes to solving the
issue of high intra-class/low inter-class variability by focusing on
preparation steps specific to the meal. The experimental results demonstrate
the potential of such a system for recipe retrieval from images. A comparison
with respect to two baseline methods is also presented.
