  Image-based methods to analyze food images have alleviated the user burden
and biases associated with traditional methods. However, accurate portion
estimation remains a major challenge due to the loss of 3D information in the
2D representation of foods captured by smartphone cameras or wearable devices.
In this paper, we propose a new framework to estimate both food volume and
energy from 2D images by leveraging the power of 3D food models and physical
reference in the eating scene. Our method estimates the pose of the camera and
the food object in the input image and recreates the eating occasion by
rendering an image of a 3D model of the food with the estimated poses. We also
introduce a new dataset, SimpleFood45, which contains 2D images of 45 food
items and associated annotations including food volume, weight, and energy. Our
method achieves an average error of 31.10 kCal (17.67%) on this dataset,
outperforming existing portion estimation methods.
