  We introduce a new class of vision-based sensor and associated algorithmic
processes that combine visual imaging with high-resolution tactile sending, all
in a uniform hardware and computational architecture. We demonstrate the
sensor's efficacy for both multi-modal object recognition and metrology. Object
recognition is typically formulated as an unimodal task, but by combining two
sensor modalities we show that we can achieve several significant performance
improvements. This sensor, named the See-Through-your-Skin sensor (STS), is
designed to provide rich multi-modal sensing of contact surfaces. Inspired by
recent developments in optical tactile sensing technology, we address a key
missing feature of these sensors: the ability to capture a visual perspective
of the region beyond the contact surface. Whereas optical tactile sensors are
typically opaque, we present a sensor with a semitransparent skin that has the
dual capabilities of acting as a tactile sensor and/or as a visual camera
depending on its internal lighting conditions. This paper details the design of
the sensor, showcases its dual sensing capabilities, and presents a deep
learning architecture that fuses vision and touch. We validate the ability of
the sensor to classify household objects, recognize fine textures, and infer
their physical properties both through numerical simulations and experiments
with a smart countertop prototype.
