  The modern saying, "You Are What You Eat" resonates on a profound level,
reflecting the intricate connection between our identities and the food we
consume. Our project, Deep Image-to-Recipe Translation, is an intersection of
computer vision and natural language generation that aims to bridge the gap
between cherished food memories and the art of culinary creation. Our primary
objective involves predicting ingredients from a given food image. For this
task, we first develop a custom convolutional network and then compare its
performance to a model that leverages transfer learning. We pursue an
additional goal of generating a comprehensive set of recipe steps from a list
of ingredients. We frame this process as a sequence-to-sequence task and
develop a recurrent neural network that utilizes pre-trained word embeddings.
We address several challenges of deep learning including imbalanced datasets,
data cleaning, overfitting, and hyperparameter selection. Our approach
emphasizes the importance of metrics such as Intersection over Union (IoU) and
F1 score in scenarios where accuracy alone might be misleading. For our recipe
prediction model, we employ perplexity, a commonly used and important metric
for language models. We find that transfer learning via pre-trained ResNet-50
weights and GloVe embeddings provide an exceptional boost to model performance,
especially when considering training resource constraints. Although we have
made progress on the image-to-recipe translation, there is an opportunity for
future exploration with advancements in model architectures, dataset
scalability, and enhanced user interaction.
