  In general, robotic dexterous hands are equipped with various sensors for
acquiring multimodal contact information such as position, force, and pose of
the grasped object. This multi-sensor-based design adds complexity to the
robotic system. In contrast, vision-based tactile sensors employ specialized
optical designs to enable the extraction of tactile information across
different modalities within a single system. Nonetheless, the decoupling design
for different modalities in common systems is often independent. Therefore, as
the dimensionality of tactile modalities increases, it poses more complex
challenges in data processing and decoupling, thereby limiting its application
to some extent. Here, we developed a multimodal sensing system based on a
vision-based tactile sensor, which utilizes visual representations of tactile
information to perceive the multimodal contact information of the grasped
object. The visual representations contain extensive content that can be
decoupled by a deep neural network to obtain multimodal contact information
such as classification, position, posture, and force of the grasped object. The
results show that the tactile sensing system can perceive multimodal tactile
information using only one single sensor and without different data decoupling
designs for different modal tactile information, which reduces the complexity
of the tactile system and demonstrates the potential for multimodal tactile
integration in various fields such as biomedicine, biology, and robotics.
