  J-vector has been proved to be very effective in text-dependent speaker
verification with short-duration speech. However, the current state-of-the-art
back-end classifiers, e.g. joint Bayesian model, cannot make full use of such
deep features. In this paper, we generalize the standard joint Bayesian
approach to model the multi-faceted information in the j-vector explicitly and
jointly. In our generalization, the j-vector was modeled as a result derived by
a generative Double Joint Bayesian (DoJoBa) model, which contains several kinds
of latent variables. With DoJoBa, we are able to explicitly build a model that
can combine multiple heterogeneous information from the j-vectors. In
verification step, we calculated the likelihood to describe whether the two
j-vectors having consistent labels or not. On the public RSR2015 data corpus,
the experimental results showed that our approach can achieve 0.02\% EER and
0.02\% EER for impostor wrong and impostor correct cases respectively.
