  In this paper, we introduce a novel audio-visual multi-modal bridging
framework that can utilize both audio and visual information, even with
uni-modal inputs. We exploit a memory network that stores source (i.e., visual)
and target (i.e., audio) modal representations, where source modal
representation is what we are given, and target modal representations are what
we want to obtain from the memory network. We then construct an associative
bridge between source and target memories that considers the interrelationship
between the two memories. By learning the interrelationship through the
associative bridge, the proposed bridging framework is able to obtain the
target modal representations inside the memory network, even with the source
modal input only, and it provides rich information for its downstream tasks. We
apply the proposed framework to two tasks: lip reading and speech
reconstruction from silent video. Through the proposed associative bridge and
modality-specific memories, each task knowledge is enriched with the recalled
audio context, achieving state-of-the-art performance. We also verify that the
associative bridge properly relates the source and target memories.
