  Speech is one of the most effective ways of communication among humans. Even
though audio is the most common way of transmitting speech, very important
information can be found in other modalities, such as vision. Vision is
particularly useful when the acoustic signal is corrupted. Multi-modal speech
recognition however has not yet found wide-spread use, mostly because the
temporal alignment and fusion of the different information sources is
challenging.
  This paper presents an end-to-end audiovisual speech recognizer (AVSR), based
on recurrent neural networks (RNN) with a connectionist temporal classification
(CTC) loss function. CTC creates sparse "peaky" output activations, and we
analyze the differences in the alignments of output targets (phonemes or
visemes) between audio-only, video-only, and audio-visual feature
representations. We present the first such experiments on the large vocabulary
IBM ViaVoice database, which outperform previously published approaches on
phone accuracy in clean and noisy conditions.
