  Vision-based tactile sensors typically utilize a deformable elastomer and a
camera mounted above to provide high-resolution image observations of contacts.
Obtaining accurate volumetric meshes for the deformed elastomer can provide
direct contact information and benefit robotic grasping and manipulation. This
paper focuses on learning to synthesize the volumetric mesh of the elastomer
based on the image imprints acquired from vision-based tactile sensors.
Synthetic image-mesh pairs and real-world images are gathered from 3D finite
element methods (FEM) and physical sensors, respectively. A graph neural
network (GNN) is introduced to learn the image-to-mesh mappings with supervised
learning. A self-supervised adaptation method and image augmentation techniques
are proposed to transfer networks from simulation to reality, from primitive
contacts to unseen contacts, and from one sensor to another. Using these
learned and adapted networks, our proposed method can accurately reconstruct
the deformation of the real-world tactile sensor elastomer in various domains,
as indicated by the quantitative and qualitative results.
