  We study special subvarieties, i.e., subvarieties containing a dense subset
of CM points, of the moduli space $A_5$ of principally polarized abelian
varieties of dimension five, generically contained in the locus of intermediate
Jacobians of cubic threefolds. The analogous question for Jacobians of curves
is related to a conjecture of Coleman-Ort and has been studied by Shimura,
Mostow, De Jong-Noot, Rohde, Moonen, Oort, Frediani, Ghigi and others.
  Adapting methods of Frediani, Ghigi and Penegini, we give a sufficient
condition ensuring that the closure of the image of a family of smooth cubic
threefolds with prescribed automorphisms via the period map is a special
subvariety of $A_5$.
  By the work of Allcock, Carlson and Toledo, it is known that the family of
cyclic cubic threefolds gives rise to a special subvariety. Analyzing the
action of subgroups of the automorphism group of the Klein cubic threefold, we
discover new examples of positive-dimensional special subvarieties arising from
families of smooth cubic threefolds that are not contained in the locus of
cyclic cubic threefolds.
