  We study orbifolding by the $\mathbb{Z}_2^{\rm (per)}$ permutaion of $T^2_1
\times T^2_2$ with magnetic fluxes and its twisted orbifolds. We classify the
possible three generation models which lead to non-vanishing Yukawa couplings
on the magnetized $T^2_1 \times T^2_2$ and orbifolds including the
$\mathbb{Z}_2^{\rm (per)}$ permutation and $\mathbb{Z}_2^{\rm (t)}$ twist. We
also study the modular symmetry on such orbifold models. As an illustrating
model, we examine the realization of quark masses and mixing angles.
