  For effective interactions with the open world, robots should understand how
interactions with known and novel objects help them towards their goal. A key
aspect of this understanding lies in detecting an object's affordances, which
represent the potential effects that can be achieved by manipulating the object
in various ways. Our approach leverages a dialogue of large language models
(LLMs) and vision-language models (VLMs) to achieve open-world affordance
detection. Given open-vocabulary descriptions of intended actions and effects,
the useful objects in the environment are found. By grounding our system in the
physical world, we account for the robot's embodiment and the intrinsic
properties of the objects it encounters. In our experiments, we have shown that
our method produces tailored outputs based on different embodiments or intended
effects. The method was able to select a useful object from a set of
distractors. Finetuning the VLM for physical properties improved overall
performance. These results underline the importance of grounding the affordance
search in the physical world, by taking into account robot embodiment and the
physical properties of objects.
