  Dysarthric speech reconstruction (DSR), which aims to improve the quality of
dysarthric speech, remains a challenge, not only because we need to restore the
speech to be normal, but also must preserve the speaker's identity. The speaker
representation extracted by the speaker encoder (SE) optimized for speaker
verification has been explored to control the speaker identity. However, the SE
may not be able to fully capture the characteristics of dysarthric speakers
that are previously unseen. To address this research problem, we propose a
novel multi-task learning strategy, i.e., adversarial speaker adaptation (ASA).
The primary task of ASA fine-tunes the SE with the speech of the target
dysarthric speaker to effectively capture identity-related information, and the
secondary task applies adversarial training to avoid the incorporation of
abnormal speaking patterns into the reconstructed speech, by regularizing the
distribution of reconstructed speech to be close to that of reference speech
with high quality. Experiments show that the proposed approach can achieve
enhanced speaker similarity and comparable speech naturalness with a strong
baseline approach. Compared with dysarthric speech, the reconstructed speech
achieves 22.3% and 31.5% absolute word error rate reduction for speakers with
moderate and moderate-severe dysarthria respectively. Our demo page is released
here: https://wendison.github.io/ASA-DSR-demo/
