  Speech emotion recognition (SER) classifies audio into emotion categories
such as Happy, Angry, Fear, Disgust and Neutral. While Speech Emotion
Recognition (SER) is a common application for popular languages, it continues
to be a problem for low-resourced languages, i.e., languages with no pretrained
speech-to-text recognition models. This paper firstly proposes a
language-specific model that extract emotional information from multiple
pre-trained speech models, and then designs a multi-domain model that
simultaneously performs SER for various languages. Our multidomain model
employs a multi-gating mechanism to generate unique weighted feature
combination for each language, and also searches for specific neural network
structure for each language through a neural architecture search module. In
addition, we introduce a contrastive auxiliary loss to build more separable
representations for audio data. Our experiments show that our model raises the
state-of-the-art accuracy by 3% for German and 14.3% for French.
