  Large language models (LLMs) and vision-language models (VLMs) have been
increasingly used in robotics for high-level cognition, but their use for
low-level cognition, such as interpreting sensor information, remains
underexplored. In robotic grasping, estimating the reflectance of objects is
crucial for successful grasping, as it significantly impacts the distance
measured by proximity sensors. We investigate whether LLMs can estimate
reflectance from object names alone, leveraging the embedded human knowledge in
distributional semantics, and if the latent structure of language in VLMs
positively affects image-based reflectance estimation. In this paper, we verify
that 1) LLMs such as GPT-3.5 and GPT-4 can estimate an object's reflectance
using only text as input; and 2) VLMs such as CLIP can increase their
generalization capabilities in reflectance estimation from images. Our
experiments show that GPT-4 can estimate an object's reflectance using only
text input with a mean error of 14.7%, lower than the image-only ResNet.
Moreover, CLIP achieved the lowest mean error of 11.8%, while GPT-3.5 obtained
a competitive 19.9% compared to ResNet's 17.8%. These results suggest that the
distributional semantics in LLMs and VLMs increases their generalization
capabilities, and the knowledge acquired by VLMs benefits from the latent
structure of language.
