  Audio and visual signals complement each other in human speech perception, so
do they in speech recognition. The visual hint is less evident than the
acoustic hint, but more robust in a complex acoustic environment, as far as
speech perception is concerned. It remains a challenge how we effectively
exploit the interaction between audio and visual signals for automatic speech
recognition. There have been studies to exploit visual signals as redundant or
complementary information to audio input in a synchronous manner. Human studies
suggest that visual signal primes the listener in advance as to when and on
which frequency to attend to. We propose a Predict-and-Update Network (P&U
net), to simulate such a visual cueing mechanism for Audio-Visual Speech
Recognition (AVSR). In particular, we first predict the character posteriors of
the spoken words, i.e. the visual embedding, based on the visual signals. The
audio signal is then conditioned on the visual embedding via a novel
cross-modal Conformer, that updates the character posteriors. We validate the
effectiveness of the visual cueing mechanism through extensive experiments. The
proposed P&U net outperforms the state-of-the-art AVSR methods on both LRS2-BBC
and LRS3-BBC datasets, with the relative reduced Word Error Rate (WER)s
exceeding 10% and 40% under clean and noisy conditions, respectively.
