  End-to-end spoken language understanding (SLU) remains elusive even with
current large pretrained language models on text and speech, especially in
multilingual cases. Machine translation has been established as a powerful
pretraining objective on text as it enables the model to capture high-level
semantics of the input utterance and associations between different languages,
which is desired for speech models that work on lower-level acoustic frames.
Motivated particularly by the task of cross-lingual SLU, we demonstrate that
the task of speech translation (ST) is a good means of pretraining speech
models for end-to-end SLU on both intra- and cross-lingual scenarios.
  By introducing ST, our models reach higher performance over baselines on
monolingual and multilingual intent classification as well as spoken question
answering using SLURP, MINDS-14, and NMSQA benchmarks. To verify the
effectiveness of our methods, we also create new benchmark datasets from both
synthetic and real sources, for speech summarization and low-resource/zero-shot
transfer from English to French or Spanish. We further show the value of
preserving knowledge for the ST pretraining task for better downstream
performance, possibly using Bayesian transfer regularizers.
