  The application of data-intensive automatic speech recognition (ASR)
technologies to dysarthric and elderly adult speech is confronted by their
mismatch against healthy and nonaged voices, data scarcity and large
speaker-level variability. To this end, this paper proposes two novel
data-efficient methods to learn homogeneous dysarthric and elderly
speaker-level features for rapid, on-the-fly test-time adaptation of DNN/TDNN
and Conformer ASR models. These include: 1) speaker-level variance-regularized
spectral basis embedding (VR-SBE) features that exploit a special
regularization term to enforce homogeneity of speaker features in adaptation;
and 2) feature-based learning hidden unit contributions (f-LHUC) transforms
that are conditioned on VR-SBE features. Experiments are conducted on four
tasks across two languages: the English UASpeech and TORGO dysarthric speech
datasets, the English DementiaBank Pitt and Cantonese JCCOCC MoCA elderly
speech corpora. The proposed on-the-fly speaker adaptation techniques
consistently outperform baseline iVector and xVector adaptation by
statistically significant word or character error rate reductions up to 5.32%
absolute (18.57% relative) and batch-mode LHUC speaker adaptation by 2.24%
absolute (9.20% relative), while operating with real-time factors speeding up
to 33.6 times against xVectors during adaptation. The efficacy of the proposed
adaptation techniques is demonstrated in a comparison against current ASR
technologies including SSL pre-trained systems on UASpeech, where our best
system produces a state-of-the-art WER of 23.33%. Analyses show VR-SBE features
and f-LHUC transforms are insensitive to speaker-level data quantity in
testtime adaptation. T-SNE visualization reveals they have stronger
speaker-level homogeneity than baseline iVectors, xVectors and batch-mode LHUC
transforms.
