  Food is an integral part of our life and what and how much we eat crucially
affects our health. Our food choices largely depend on how we perceive certain
characteristics of food, such as whether it is healthy, delicious or if it
qualifies as a salad. But these perceptions differ from person to person and
one person's "single lettuce leaf" might be another person's "side salad".
Studying how food is perceived in relation to what it actually is typically
involves a laboratory setup. Here we propose to use recent advances in image
recognition to tackle this problem. Concretely, we use data for 1.9 million
images from Instagram from the US to look at systematic differences in how a
machine would objectively label an image compared to how a human subjectively
does. We show that this difference, which we call the "perception gap", relates
to a number of health outcomes observed at the county level. To the best of our
knowledge, this is the first time that image recognition is being used to study
the "misalignment" of how people describe food images vs. what they actually
depict.
