  We propose a Perceiver-based sequence classifier to detect abnormalities in
speech reflective of several neurological disorders. We combine this classifier
with a Universal Speech Model (USM) that is trained (unsupervised) on 12
million hours of diverse audio recordings. Our model compresses long sequences
into a small set of class-specific latent representations and a factorized
projection is used to predict different attributes of the disordered input
speech. The benefit of our approach is that it allows us to model different
regions of the input for different classes and is at the same time data
efficient. We evaluated the proposed model extensively on a curated corpus from
the Mayo Clinic. Our model outperforms standard transformer (80.9%) and
perceiver (81.8%) models and achieves an average accuracy of 83.1%. With
limited task-specific data, we find that pretraining is important and
surprisingly pretraining with the unrelated automatic speech recognition (ASR)
task is also beneficial. Encodings from the middle layers provide a mix of both
acoustic and phonetic information and achieve best prediction results compared
to just using the final layer encodings (83.1% vs. 79.6%). The results are
promising and with further refinements may help clinicians detect speech
abnormalities without needing access to highly specialized speech-language
pathologists.
