  In this study, we propose a novel multi-modal end-to-end neural approach for
automated assessment of non-native English speakers' spontaneous speech using
attention fusion. The pipeline employs Bi-directional Recurrent Convolutional
Neural Networks and Bi-directional Long Short-Term Memory Neural Networks to
encode acoustic and lexical cues from spectrograms and transcriptions,
respectively. Attention fusion is performed on these learned predictive
features to learn complex interactions between different modalities before
final scoring. We compare our model with strong baselines and find combined
attention to both lexical and acoustic cues significantly improves the overall
performance of the system. Further, we present a qualitative and quantitative
analysis of our model.
