  Forced alignment refers to a technology that time-aligns a given
transcription with a corresponding speech. However, as the forced alignment
technologies have developed using speech audio, they might fail in alignment
when the input speech audio is noise-corrupted or is not accessible. We focus
on that there is another component that the speech can be inferred from, the
speech video (i.e., talking face video). Since the drawbacks of audio-based
forced alignment can be complemented using the visual information when the
audio signal is under poor condition, we try to develop a novel video-based
forced alignment method. However, different from audio forced alignment, it is
challenging to develop a reliable visual forced alignment technology for the
following two reasons: 1) Visual Speech Recognition (VSR) has a much lower
performance compared to audio-based Automatic Speech Recognition (ASR), and 2)
the translation from text to video is not reliable, so the method typically
used for building audio forced alignment cannot be utilized in developing
visual forced alignment. In order to alleviate these challenges, in this paper,
we propose a new method that is appropriate for visual forced alignment, namely
Deep Visual Forced Alignment (DVFA). The proposed DVFA can align the input
transcription (i.e., sentence) with the talking face video without accessing
the speech audio. Moreover, by augmenting the alignment task with anomaly case
detection, DVFA can detect mismatches between the input transcription and the
input video while performing the alignment. Therefore, we can robustly align
the text with the talking face video even if there exist error words in the
text. Through extensive experiments, we show the effectiveness of the proposed
DVFA not only in the alignment task but also in interpreting the outputs of VSR
models.
