  Food image classification is the fundamental step in image-based dietary
assessment, which aims to estimate participants' nutrient intake from eating
occasion images. A common challenge of food images is the intra-class diversity
and inter-class similarity, which can significantly hinder classification
performance. To address this issue, we introduce a novel multi-modal
contrastive learning framework called FMiFood, which learns more discriminative
features by integrating additional contextual information, such as food
category text descriptions, to enhance classification accuracy. Specifically,
we propose a flexible matching technique that improves the similarity matching
between text and image embeddings to focus on multiple key information.
Furthermore, we incorporate the classification objectives into the framework
and explore the use of GPT-4 to enrich the text descriptions and provide more
detailed context. Our method demonstrates improved performance on both the
UPMC-101 and VFN datasets compared to existing methods.
