  Speech is the most used communication method between humans and it involves
the perception of auditory and visual channels. Automatic speech recognition
focuses on interpreting the audio signals, although the video can provide
information that is complementary to the audio. Exploiting the visual
information, however, has proven challenging. On one hand, researchers have
reported that the mapping between phonemes and visemes (visual units) is
one-to-many because there are phonemes which are visually similar and
indistinguishable between them. On the other hand, it is known that some people
are very good lip-readers (e.g: deaf people). We study the limit of visual only
speech recognition in controlled conditions. With this goal, we designed a new
database in which the speakers are aware of being read and aim to facilitate
lip-reading. In the literature, there are discrepancies on whether
hearing-impaired people are better lip-readers than normal-hearing people.
Then, we analyze if there are differences between the lip-reading abilities of
9 hearing-impaired and 15 normal-hearing people. Finally, human abilities are
compared with the performance of a visual automatic speech recognition system.
In our tests, hearing-impaired participants outperformed the normal-hearing
participants but without reaching statistical significance. Human observers
were able to decode 44% of the spoken message. In contrast, the visual only
automatic system achieved 20% of word recognition rate. However, if we repeat
the comparison in terms of phonemes both obtained very similar recognition
rates, just above 50%. This suggests that the gap between human lip-reading and
automatic speech-reading might be more related to the use of context than to
the ability to interpret mouth appearance.
