  We study the modular symmetry of zero-modes on $T_1^2 \times T_2^2$ and
orbifold compactifications with magnetic fluxes, $M_1,M_2$, where modulus
parameters are identified. This identification breaks the modular symmetry of
$T^2_1 \times T^2_2$, $SL(2,\mathbb{Z})_1 \times SL(2,\mathbb{Z})_2$ to
$SL(2,\mathbb{Z})\equiv\Gamma$. Each of the wavefunctions on $T^2_1 \times
T^2_2$ and orbifolds behaves as the modular forms of weight 1 for the principal
congruence subgroup $\Gamma$($N$), $N$ being 2 times the least common multiple
of $M_1$ and $M_2$. Then, zero-modes transform each other under the modular
symmetry as multiplets of double covering groups of $\Gamma_N$ such as the
double cover of $S_4$.
