  Lip reading aims at decoding texts from the movement of a speaker's mouth. In
recent years, lip reading methods have made great progress for English, at both
word-level and sentence-level. Unlike English, however, Chinese Mandarin is a
tone-based language and relies on pitches to distinguish lexical or grammatical
meaning, which significantly increases the ambiguity for the lip reading task.
In this paper, we propose a Cascade Sequence-to-Sequence Model for Chinese
Mandarin (CSSMCM) lip reading, which explicitly models tones when predicting
sentence. Tones are modeled based on visual information and syntactic
structure, and are used to predict sentence along with visual information and
syntactic structure. In order to evaluate CSSMCM, a dataset called CMLR
(Chinese Mandarin Lip Reading) is collected and released, consisting of over
100,000 natural sentences from China Network Television website. When trained
on CMLR dataset, the proposed CSSMCM surpasses the performance of
state-of-the-art lip reading frameworks, which confirms the effectiveness of
explicit modeling of tones for Chinese Mandarin lip reading.
