  In this paper, we explore the impact of adding tactile sensation to video
prediction models for physical robot interactions. Predicting the impact of
robotic actions on the environment is a fundamental challenge in robotics.
Current methods leverage visual and robot action data to generate video
predictions over a given time period, which can then be used to adjust robot
actions. However, humans rely on both visual and tactile feedback to develop
and maintain a mental model of their physical surroundings. In this paper, we
investigate the impact of integrating tactile feedback into video prediction
models for physical robot interactions. We propose three multi-modal
integration approaches and compare the performance of these tactile-enhanced
video prediction models. Additionally, we introduce two new datasets of robot
pushing that use a magnetic-based tactile sensor for unsupervised learning. The
first dataset contains visually identical objects with different physical
properties, while the second dataset mimics existing robot-pushing datasets of
household object clusters. Our results demonstrate that incorporating tactile
feedback into video prediction models improves scene prediction accuracy and
enhances the agent's perception of physical interactions and understanding of
cause-effect relationships during physical robot interactions.
