  The ability to both recognize and discover terrain characteristics is an
important function required for many autonomous ground robots such as social
robots, assistive robots, autonomous vehicles, and ground exploration robots.
Recognizing and discovering terrain characteristics is challenging because
similar terrains may have very different appearances (e.g., carpet comes in
many colors), while terrains with very similar appearance may have very
different physical properties (e.g. mulch versus dirt). In order to address the
inherent ambiguity in vision-based terrain recognition and discovery, we
propose a multi-modal self-supervised learning technique that switches between
audio features extracted from a mic attached to the underside of a mobile
platform and image features extracted by a camera on the platform to cluster
terrain types. The terrain cluster labels are then used to train an image-based
convolutional neural network to predict changes in terrain types. Through
experiments, we demonstrate that the proposed self-supervised terrain type
discovery method achieves over 80% accuracy, which greatly outperforms several
baselines and suggests strong potential for assistive applications.
