  Domain mismatch often occurs in real applications and causes serious
performance reduction on speaker verification systems. The common wisdom is to
collect cross-domain data and train a multi-domain PLDA model, with the hope to
learn a domain-independent speaker subspace. In this paper, we firstly present
an empirical study to show that simply adding cross-domain data does not help
performance in conditions with enrollment-test mismatch. Careful analysis shows
that this striking result is caused by the incoherent statistics between the
enrollment and test conditions. Based on this analysis, we present a decoupled
scoring approach that can maximally squeeze the value of cross-domain labels
and obtain optimal verification scores when the enrollment and test are
mismatched. When the statistics are coherent, the new formulation falls back to
the conventional PLDA. Experimental results on cross-channel test show that the
proposed approach is highly effective and is a principle solution to domain
mismatch.
