  We propose an end-to-end joint optimization framework of a multi-channel
neural speech extraction and deep acoustic model without mel-filterbank (FBANK)
extraction for overlapped speech recognition. First, based on a multi-channel
convolutional TasNet with STFT kernel, we unify the multi-channel target speech
enhancement front-end network and a convolutional, long short-term memory and
fully connected deep neural network (CLDNN) based acoustic model (AM) with the
FBANK extraction layer to build a hybrid neural network, which is thus jointly
updated only by the recognition loss. The proposed framework achieves 28% word
error rate reduction (WERR) over a separately optimized system on AISHELL-1 and
shows consistent robustness to signal to interference ratio (SIR) and angle
difference between overlapping speakers. Next, a further exploration shows that
the speech recognition is improved with a simplified structure by replacing the
FBANK extraction layer in the joint model with a learnable feature projection.
Finally, we also perform the objective measurement of speech quality on the
reconstructed waveform from the enhancement network in the joint model.
