  Emotion recognition from speech is a challenging task. Re-cent advances in
deep learning have led bi-directional recur-rent neural network (Bi-RNN) and
attention mechanism as astandard method for speech emotion recognition,
extractingand attending multi-modal features - audio and text, and thenfusing
them for downstream emotion classification tasks. Inthis paper, we propose a
simple yet efficient neural networkarchitecture to exploit both acoustic and
lexical informationfrom speech. The proposed framework using multi-scale
con-volutional layers (MSCNN) to obtain both audio and text hid-den
representations. Then, a statistical pooling unit (SPU)is used to further
extract the features in each modality. Be-sides, an attention module can be
built on top of the MSCNN-SPU (audio) and MSCNN (text) to further improve the
perfor-mance. Extensive experiments show that the proposed modeloutperforms
previous state-of-the-art methods on IEMOCAPdataset with four emotion
categories (i.e., angry, happy, sadand neutral) in both weighted accuracy (WA)
and unweightedaccuracy (UA), with an improvement of 5.0% and 5.2% respectively
under the ASR setting.
