  In robotic object manipulation, human preferences can often be influenced by
the visual attributes of objects, such as color and shape. These properties
play a crucial role in operating a robot to interact with objects and align
with human intention. In this paper, we focus on the problem of inferring
underlying human preferences from a sequence of raw visual observations in
tabletop manipulation environments with a variety of object types, named Visual
Preference Inference (VPI). To facilitate visual reasoning in the context of
manipulation, we introduce the Chain-of-Visual-Residuals (CoVR) method. CoVR
employs a prompting mechanism that describes the difference between the
consecutive images (i.e., visual residuals) and incorporates such texts with a
sequence of images to infer the user's preference. This approach significantly
enhances the ability to understand and adapt to dynamic changes in its visual
environment during manipulation tasks. Furthermore, we incorporate such texts
along with a sequence of images to infer the user's preferences. Our method
outperforms baseline methods in terms of extracting human preferences from
visual sequences in both simulation and real-world environments. Code and
videos are available at:
\href{https://joonhyung-lee.github.io/vpi/}{https://joonhyung-lee.github.io/vpi/}
