  In this work, we present SynesLM, an unified model which can perform three
multimodal language understanding tasks: audio-visual automatic speech
recognition(AV-ASR) and visual-aided speech/machine translation(VST/VMT).
Unlike previous research that focused on lip motion as visual cues for speech
signals, our work explores more general visual information within entire
frames, such as objects and actions. Additionally, we use synthetic image data
to enhance the correlation between image and speech data. We benchmark SynesLM
against the How2 dataset, demonstrating performance on par with
state-of-the-art (SOTA) models dedicated to AV-ASR while maintaining our
multitasking framework. Remarkably, for zero-shot AV-ASR, SynesLM achieved SOTA
performance by lowering the Word Error Rate (WER) from 43.4% to 39.4% on the
VisSpeech Dataset. Furthermore, our results in VST and VMT outperform the
previous results, improving the BLEU score to 43.5 from 37.2 for VST, and to
54.8 from 54.4 for VMT.
