  Language models~(LMs) gradually become general-purpose interfaces in the
interactive and embodied world, where the understanding of physical concepts is
an essential prerequisite. However, it is not yet clear whether LMs can
understand physical concepts in the human world. To investigate this, we design
a benchmark VEC that covers the tasks of (i) Visual concepts, such as the shape
and material of objects, and (ii) Embodied Concepts, learned from the
interaction with the world such as the temperature of objects. Our zero
(few)-shot prompting results show that the understanding of certain visual
concepts emerges as scaling up LMs, but there are still basic concepts to which
the scaling law does not apply. For example, OPT-175B performs close to humans
with a zero-shot accuracy of 85\% on the material concept, yet behaves like
random guessing on the mass concept. Instead, vision-augmented LMs such as CLIP
and BLIP achieve a human-level understanding of embodied concepts. Analysis
indicates that the rich semantics in visual representation can serve as a
valuable source of embodied knowledge. Inspired by this, we propose a
distillation method to transfer embodied knowledge from VLMs to LMs, achieving
performance gain comparable with that by scaling up the parameters of LMs 134x.
Our dataset is available at \url{https://github.com/TobiasLee/VEC}
