  Traditional visual speech recognition systems consist of two stages, feature
extraction and classification. Recently, several deep learning approaches have
been presented which automatically extract features from the mouth images and
aim to replace the feature extraction stage. However, research on joint
learning of features and classification is very limited. In this work, we
present an end-to-end visual speech recognition system based on Long-Short
Memory (LSTM) networks. To the best of our knowledge, this is the first model
which simultaneously learns to extract features directly from the pixels and
perform classification and also achieves state-of-the-art performance in visual
speech classification. The model consists of two streams which extract features
directly from the mouth and difference images, respectively. The temporal
dynamics in each stream are modelled by an LSTM and the fusion of the two
streams takes place via a Bidirectional LSTM (BLSTM). An absolute improvement
of 9.7% over the base line is reported on the OuluVS2 database, and 1.5% on the
CUAVE database when compared with other methods which use a similar visual
front-end.
