  Recent focus in video captioning has been on designing architectures that can
consume both video and text modalities, and using large-scale video datasets
with text transcripts for pre-training, such as HowTo100M. Though these
approaches have achieved significant improvement, the audio modality is often
ignored in video captioning. In this work, we present an audio-visual
framework, which aims to fully exploit the potential of the audio modality for
captioning. Instead of relying on text transcripts extracted via automatic
speech recognition (ASR), we argue that learning with raw audio signals can be
more beneficial, as audio has additional information including acoustic events,
speaker identity, etc. Our contributions are twofold. First, we observed that
the model overspecializes to the audio modality when pre-training with both
video and audio modality, since the ground truth (i.e., text transcripts) can
be solely predicted using audio. We proposed a Modality Balanced Pre-training
(MBP) loss to mitigate this issue and significantly improve the performance on
downstream tasks. Second, we slice and dice different design choices of the
cross-modal module, which may become an information bottleneck and generate
inferior results. We proposed new local-global fusion mechanisms to improve
information exchange across audio and video. We demonstrate significant
improvements by leveraging the audio modality on four datasets, and even
outperform the state of the art on some metrics without relying on the text
modality as the input.
