  Recently, a variety of acoustic tasks and related applications arised. For
many acoustic tasks, the labeled data size may be limited. To handle this
problem, we propose an unsupervised pre-training method using Transformer based
encoder to learn a general and robust high-level representation for all
acoustic tasks. Experiments have been conducted on three kinds of acoustic
tasks: speech emotion recognition, sound event detection and speech
translation. All the experiments have shown that pre-training using its own
training data can significantly improve the performance. With a larger
pre-training data combining MuST-C, Librispeech and ESC-US datasets, for speech
emotion recognition, the UAR can further improve absolutely 4.3% on IEMOCAP
dataset. For sound event detection, the F1 score can further improve absolutely
1.5% on DCASE2018 task5 development set and 2.1% on evaluation set. For speech
translation, the BLEU score can further improve relatively 12.2% on En-De
dataset and 8.4% on En-Fr dataset.
