  Cued Speech (CS) is an augmented lip reading complemented by hand coding, and
it is very helpful to the deaf people. Automatic CS recognition can help
communications between the deaf people and others. Due to the asynchronous
nature of lips and hand movements, fusion of them in automatic CS recognition
is a challenging problem. In this work, we propose a novel re-synchronization
procedure for multi-modal fusion, which aligns the hand features with lips
feature. It is realized by delaying hand position and hand shape with their
optimal hand preceding time which is derived by investigating the temporal
organizations of hand position and hand shape movements in CS. This
re-synchronization procedure is incorporated into a practical continuous CS
recognition system that combines convolutional neural network (CNN) with
multi-stream hidden markov model (MSHMM). A significant improvement of about
4.6\% has been achieved retaining 76.6\% CS phoneme recognition correctness
compared with the state-of-the-art architecture (72.04\%), which did not take
into account the asynchrony of multi-modal fusion in CS. To our knowledge, this
is the first work to tackle the asynchronous multi-modal fusion in the
automatic continuous CS recognition.
