  Self-supervised learning (SSL) speech models, which can serve as powerful
upstream models to extract meaningful speech representations, have achieved
unprecedented success in speech representation learning. However, their
effectiveness on non-speech datasets is relatively less explored. In this work,
we propose an ensemble framework, with a combination of ensemble techniques, to
fuse SSL speech models' embeddings. Extensive experiments on speech and
non-speech audio datasets are conducted to investigate the representation
abilities of our ensemble method and its single constituent model. Ablation
studies are carried out to evaluate the performances of different ensemble
techniques, such as feature averaging and concatenation. All experiments are
conducted during NeurIPS 2021 HEAR Challenge as a standard evaluation pipeline
provided by competition officials. Results demonstrate SSL speech models'
strong abilities on various non-speech tasks, while we also note that they fail
to deal with fine-grained music tasks, such as pitch classification and note
onset detection. In addition, feature ensemble is shown to have great potential
on producing more holistic representations, as our proposed framework generally
surpasses state-of-the-art SSL speech/audio models and has superior performance
on various datasets compared with other teams in HEAR Challenge. Our code is
available at https://github.com/tony10101105/HEAR-2021-NeurIPS-Challenge --
NTU-GURA.
