  Lip reading, aiming to recognize spoken sentences according to the given
video of lip movements without relying on the audio stream, has attracted great
interest due to its application in many scenarios. Although prior works that
explore lip reading have obtained salient achievements, they are all trained in
a non-simultaneous manner where the predictions are generated requiring access
to the full video. To breakthrough this constraint, we study the task of
simultaneous lip reading and devise SimulLR, a simultaneous lip Reading
transducer with attention-guided adaptive memory from three aspects: (1) To
address the challenge of monotonic alignments while considering the syntactic
structure of the generated sentences under simultaneous setting, we build a
transducer-based model and design several effective training strategies
including CTC pre-training, model warm-up and curriculum learning to promote
the training of the lip reading transducer. (2) To learn better spatio-temporal
representations for simultaneous encoder, we construct a truncated 3D
convolution and time-restricted self-attention layer to perform the
frame-to-frame interaction within a video segment containing fixed number of
frames. (3) The history information is always limited due to the storage in
real-time scenarios, especially for massive video data. Therefore, we devise a
novel attention-guided adaptive memory to organize semantic information of
history segments and enhance the visual representations with acceptable
computation-aware latency. The experiments show that the SimulLR achieves the
translation speedup 9.10$\times$ compared with the state-of-the-art
non-simultaneous methods, and also obtains competitive results, which indicates
the effectiveness of our proposed methods.
