  In this paper, we propose a novel way of addressing text-dependent automatic
speaker verification (TD-ASV) by using a shared-encoder with task-specific
decoders. An autoregressive predictive coding (APC) encoder is pre-trained in
an unsupervised manner using both out-of-domain (LibriSpeech, VoxCeleb) and
in-domain (DeepMine) unlabeled datasets to learn generic, high-level feature
representation that encapsulates speaker and phonetic content. Two
task-specific decoders were trained using labeled datasets to classify speakers
(SID) and phrases (PID). Speaker embeddings extracted from the SID decoder were
scored using a PLDA. SID and PID systems were fused at the score level. There
is a 51.9% relative improvement in minDCF for our system compared to the fully
supervised x-vector baseline on the cross-lingual DeepMine dataset. However,
the i-vector/HMM method outperformed the proposed APC encoder-decoder system. A
fusion of the x-vector/PLDA baseline and the SID/PLDA scores prior to PID
fusion further improved performance by 15% indicating complementarity of the
proposed approach to the x-vector system. We show that the proposed approach
can leverage from large, unlabeled, data-rich domains, and learn speech
patterns independent of downstream tasks. Such a system can provide competitive
performance in domain-mismatched scenarios where test data is from data-scarce
domains.
