  We study Shimura subvarieties of $\mathsf{A}_g$ obtained from families of
Galois coverings $f: C \rightarrow C'$ where $C'$ is a smooth complex
projective curve of genus $g' \geq 1$ and $g= g(C)$. We give the complete list
of all such families that satisfy a simple sufficient condition that ensures
that the closure of the image of the family via the Torelli map yields a
Shimura subvariety of $\mathsf{A}_g$ for $g' =1,2$ and for all $g \geq 2,4$ and
for $g' > 2$ and $g \leq 9$. In a previous work of the first and second author
together with A. Ghigi [FGP] similar computations were done in the case $g'=0$.
Here we find 6 families of Galois coverings, all with $g' = 1$ and $g=2,3,4$
and we show that these are the only families with $g'=1$ satisfying this
sufficient condition. We show that among these examples two families yield new
Shimura subvarieties of $\mathsf{A}_g$, while the other examples arise from
certain Shimura subvarieties of $\mathsf{A}_g$ already obtained as families of
Galois coverings of $\mathbb{P}^1$ in [FGP]. Finally we prove that if a family
satisfies this sufficient condition with $g'\geq 1$, then $g \leq 6g'+1$.
