  Dysarthric speech reconstruction (DSR) aims to transform dysarthric speech
into normal speech. It still suffers from low speaker similarity and poor
prosody naturalness. In this paper, we propose a multi-modal DSR model by
leveraging neural codec language modeling to improve the reconstruction
results, especially for the speaker similarity and prosody naturalness. Our
proposed model consists of: (i) a multi-modal content encoder to extract robust
phoneme embeddings from dysarthric speech with auxiliary visual inputs; (ii) a
speaker codec encoder to extract and normalize the speaker-aware codecs from
the dysarthric speech, in order to provide original timbre and normal prosody;
(iii) a codec language model based speech decoder to reconstruct the speech
based on the extracted phoneme embeddings and normalized codecs. Evaluations on
the commonly used UASpeech corpus show that our proposed model can achieve
significant improvements in terms of speaker similarity and prosody
naturalness.
