  Speech emotion recognition plays a crucial role in human-computer
interactions. However, most speech emotion recognition research is biased
toward English-speaking adults, which hinders its applicability to other
demographic groups in different languages and age groups. In this work, we
analyze the transferability of emotion recognition across three different
languages--English, Mandarin Chinese, and Cantonese; and 2 different age
groups--adults and the elderly. To conduct the experiment, we develop an
English-Mandarin speech emotion benchmark for adults and the elderly, BiMotion,
and a Cantonese speech emotion dataset, YueMotion. This study concludes that
different language and age groups require specific speech features, thus making
cross-lingual inference an unsuitable method. However, cross-group data
augmentation is still beneficial to regularize the model, with linguistic
distance being a significant influence on cross-lingual transferability. We
release publicly release our code at https://github.com/HLTCHKUST/elderly_ser.
