  This paper studies a class of Abelian varieties that are of $\GL_2$-type and
with isogenous classes defined over a number field $k$. We treat the cases when
their endomorphism algebras are either (1) a totally real field $K$ or (2) a
totally indefinite quaternion algebra over a totally real field $K$. Among the
isogenous class of such an Abelian variety, we identify one whose Galois
conjugates can be described in terms of actions of Atkin-Lehner operators and
the class group of $K$. Thus we deduce that such Abelian varieties are
parametrised by finite quotients of certain PEL Shimura varieties. These new
families of moduli spaces are further analysed when they are of dimension $2$.
We provide explicit numerical bounds for when they are surfaces of general
type. In addition, for two particular examples, we show that they are both
rational surfaces by computing the coordinates of inequivalent elliptic points
and studying the intersections of Hirzebruch cycles with exceptional divisors.
