  Dysarthric speech recognition (DSR) presents a formidable challenge due to
inherent inter-speaker variability, leading to severe performance degradation
when applying DSR models to new dysarthric speakers. Traditional speaker
adaptation methodologies typically involve fine-tuning models for each speaker,
but this strategy is cost-prohibitive and inconvenient for disabled users,
requiring substantial data collection. To address this issue, we introduce a
prototype-based approach that markedly improves DSR performance for unseen
dysarthric speakers without additional fine-tuning. Our method employs a
feature extractor trained with HuBERT to produce per-word prototypes that
encapsulate the characteristics of previously unseen speakers. These prototypes
serve as the basis for classification. Additionally, we incorporate supervised
contrastive learning to refine feature extraction. By enhancing representation
quality, we further improve DSR performance, enabling effective personalized
DSR. We release our code at https://github.com/NKU-HLT/PB-DSR.
