  We propose a method to track the 6D pose of an object over time, while the
object is under non-prehensile manipulation by a robot. At any given time
during the manipulation of the object, we assume access to the robot joint
controls and an image from a camera. We use the robot joint controls to perform
a physics-based prediction of how the object might be moving. We then combine
this prediction with the observation coming from the camera, to estimate the
object pose as accurately as possible. We use a particle filtering approach to
combine the control information with the visual information. We compare the
proposed method with two baselines: (i) using only an image-based pose
estimation system at each time-step, and (ii) a particle filter which does not
perform the computationally expensive physics predictions, but assumes the
object moves with constant velocity. Our results show that making physics-based
predictions is worth the computational cost, resulting in more accurate
tracking, and estimating object pose even when the object is not clearly
visible to the camera.
