  In this paper, we focus on the challenging perception problem in robotic
pouring. Most of the existing approaches either leverage visual or haptic
information. However, these techniques may suffer from poor generalization
performances on opaque containers or concerning measuring precision. To tackle
these drawbacks, we propose to make use of audio vibration sensing and design a
deep neural network PouringNet to predict the liquid height from the audio
fragment during the robotic pouring task. PouringNet is trained on our
collected real-world pouring dataset with multimodal sensing data, which
contains more than 3000 recordings of audio, force feedback, video and
trajectory data of the human hand that performs the pouring task. Each record
represents a complete pouring procedure. We conduct several evaluations on
PouringNet with our dataset and robotic hardware. The results demonstrate that
our PouringNet generalizes well across different liquid containers, positions
of the audio receiver, initial liquid heights and types of liquid, and
facilitates a more robust and accurate audio-based perception for robotic
pouring.
