  The world around us is full of soft objects we perceive and deform with
dexterous hand movements. For a robotic hand to control soft objects, it has to
acquire online state feedback of the deforming object. While RGB-D cameras can
collect occluded point clouds at a rate of 30Hz, this does not represent a
continuously trackable object surface. Hence, in this work, we developed a
method that takes as input a template mesh which is the mesh of an object in
its non-deformed state and a deformed point cloud of the same object, and then
shapes the template mesh such that it matches the deformed point cloud. The
reconstruction of meshes from point clouds has long been studied in the field
of Computer graphics under 3D reconstruction and 4D reconstruction, however,
both lack the speed and generalizability needed for robotics applications. Our
model is designed using a point cloud auto-encoder and a Real-NVP architecture.
Our trained model can perform mesh reconstruction and tracking at a rate of
58Hz on a template mesh of 3000 vertices and a deformed point cloud of 5000
points and is generalizable to the deformations of six different object
categories which are assumed to be made of soft material in our experiments
(scissors, hammer, foam brick, cleanser bottle, orange, and dice). The object
meshes are taken from the YCB benchmark dataset. An instance of a downstream
application can be the control algorithm for a robotic hand that requires
online feedback from the state of the manipulated object which would allow
online grasp adaptation in a closed-loop manner. Furthermore, the tracking
capacity of our method can help in the system identification of deforming
objects in a marker-free approach. In future work, we will extend our trained
model to generalize beyond six object categories and additionally to real-world
deforming point clouds.
