  We study modular symmetry anomalies in four-dimensional low-energy effective
field theory, which is derived from six-dimensional supersymmetric $U(N)$
Yang-Mills theory by magnetic flux compactification. The gauge symmetry $U(N)$
is broken to $U(N_a) \times U(N_b)$ by magnetic fluxes. It is found that
Abelian subgroup of the modular symmetry corresponding to discrete part of
$U(1)$ can be anomalous, but other elements independent of $U(1)$ in the
modular symmetry are always anomaly-free.
