  Advent of modern deep learning techniques has given rise to advancements in
the field of Speech Emotion Recognition (SER). However, most systems prevalent
in the field fail to generalize to speakers not seen during training. This
study focuses on handling challenges of multilingual SER, specifically on
unseen speakers. We introduce CAMuLeNet, a novel architecture leveraging
co-attention based fusion and multitask learning to address this problem.
Additionally, we benchmark pretrained encoders of Whisper, HuBERT, Wav2Vec2.0,
and WavLM using 10-fold leave-speaker-out cross-validation on five existing
multilingual benchmark datasets: IEMOCAP, RAVDESS, CREMA-D, EmoDB and CaFE and,
release a novel dataset for SER on the Hindi language (BhavVani). CAMuLeNet
shows an average improvement of approximately 8% over all benchmarks on unseen
speakers determined by our cross-validation strategy.
