  According to the World Federation of the Deaf, more than two hundred sign
languages exist. Therefore, it is challenging to understand deaf individuals,
even proficient sign language users, resulting in a barrier between the deaf
community and the rest of society. To bridge this language barrier, we propose
a novel multilingual communication system, namely MUGCAT, to improve the
communication efficiency of sign language users. By converting recognized
specific hand gestures into expressive pictures, which is universal usage and
language independence, our MUGCAT system significantly helps deaf people convey
their thoughts. To overcome the limitation of sign language usage, which is
mostly impossible to translate into complete sentences for ordinary people, we
propose to reconstruct meaningful sentences from the incomplete translation of
sign language. We also measure the semantic similarity of generated sentences
with fragmented recognized hand gestures to keep the original meaning.
Experimental results show that the proposed system can work in a real-time
manner and synthesize exquisite stunning illustrations and meaningful sentences
from a few hand gestures of sign language. This proves that our MUGCAT has
promising potential in assisting deaf communication.
