  Parameter-efficient fine-tuning (PEFT) for personalizing automatic speech
recognition (ASR) has recently shown promise for adapting general population
models to atypical speech. However, these approaches assume a priori knowledge
of the atypical speech disorder being adapted for -- the diagnosis of which
requires expert knowledge that is not always available. Even given this
knowledge, data scarcity and high inter/intra-speaker variability further limit
the effectiveness of traditional fine-tuning. To circumvent these challenges,
we first identify the minimal set of model parameters required for ASR
adaptation. Our analysis of each individual parameter's effect on adaptation
performance allows us to reduce Word Error Rate (WER) by half while adapting
0.03% of all weights. Alleviating the need for cohort-specific models, we next
propose the novel use of a meta-learned hypernetwork to generate highly
individualized, utterance-level adaptations on-the-fly for a diverse set of
atypical speech characteristics. Evaluating adaptation at the global, cohort
and individual-level, we show that hypernetworks generalize better to
out-of-distribution speakers, while maintaining an overall relative WER
reduction of 75.2% using 0.1% of the full parameter budget.
