  Under the condition that the Prym map is injective in characteristic $p$, we
prove that the special subvarieties in the moduli space of abelian varieties of
dimension $l$ and polarization type $D$, $A_{l,D}$, arising from families of
abelian covers of $\P^1$ are of a very restrictive nature. In other words, if
the family is one-dimensional or if it contains an eigenspace of certain type
for the group action on the cohomology of fibers, then the Shimura varieties
arising from such families can only be constructed by the group action of the
family.
