  End-to-end Speech-to-text Translation (E2E-ST), which directly translates
source language speech to target language text, is widely useful in practice,
but traditional cascaded approaches (ASR+MT) often suffer from error
propagation in the pipeline. On the other hand, existing end-to-end solutions
heavily depend on the source language transcriptions for pre-training or
multi-task training with Automatic Speech Recognition (ASR). We instead propose
a simple technique to learn a robust speech encoder in a self-supervised
fashion only on the speech side, which can utilize speech data without
transcription. This technique termed Masked Acoustic Modeling (MAM), not only
provides an alternative solution to improving E2E-ST, but also can perform
pre-training on any acoustic signals (including non-speech ones) without
annotation. We conduct our experiments over 8 different translation directions.
In the setting without using any transcriptions, our technique achieves an
average improvement of +1.1 BLEU, and +2.3 BLEU with MAM pre-training.
Pre-training of MAM with arbitrary acoustic signals also has an average
improvement with +1.6 BLEU for those languages. Compared with ASR multi-task
learning solution, which replies on transcription during training, our
pre-trained MAM model, which does not use transcription, achieves similar
accuracy.
