  There is a growing interest in applying large language models (LLMs) in
robotic tasks, due to their remarkable reasoning ability and extensive
knowledge learned from vast training corpora. Grounding LLMs in the physical
world remains an open challenge as they can only process textual input. Recent
advancements in large vision-language models (LVLMs) have enabled a more
comprehensive understanding of the physical world by incorporating visual
input, which provides richer contextual information than language alone. In
this work, we proposed a novel paradigm that leveraged GPT-4V(ision), the
state-of-the-art LVLM by OpenAI, to enable embodied agents to perceive liquid
objects via image-based environmental feedback. Specifically, we exploited the
physical understanding of GPT-4V to interpret the visual representation (e.g.,
time-series plot) of non-visual feedback (e.g., F/T sensor data), indirectly
enabling multimodal perception beyond vision and language using images as
proxies. We evaluated our method using 10 common household liquids with
containers of various geometry and material. Without any training or
fine-tuning, we demonstrated that our method can enable the robot to indirectly
perceive the physical response of liquids and estimate their viscosity. We also
showed that by jointly reasoning over the visual and physical attributes
learned through interactions, our method could recognize liquid objects in the
absence of strong visual cues (e.g., container labels with legible text or
symbols), increasing the accuracy from 69.0% -- achieved by the best-performing
vision-only variant -- to 86.0%.
