  Let $\mathbb{V}$ be a polarized variation of integral Hodge structure on a
smooth complex quasi-projective variety $S$. In this paper, we show that the
union of the non-factor special subvarieties for $(S, \mathbb{V})$, which are
of Shimura type with dominant period maps, is a finite union of special
subvarieties of $S$. This generalizes previous results of Clozel and Ullmo
arXiv:math/0404131, Ullmo \cite{Ullmo07} on the distribution of the non-factor
(in particular, strongly) special subvarieties in a Shimura variety to the
non-classical setting and also answers positively the geometric part of a
conjecture of Klingler on the Andr\'e-Oort conjecture for variations of Hodge
structures.
