  We study quark and lepton flavor structures on magnetized $T^2/\mathbb{Z}_2$
twisted orbifold model. There are 6,460 number of flavor models but most of
them cannot lead to realistic flavor observables because of the difficulties on
realizing mass hierarchies and small (large) mixing angles of quarks (leptons).
We find that certain zero point patterns of zero-modes of fermions and Higgs
fields give the flavor models being able to avoid these difficulties. We
classify such flavor models and show numerical example.
  Next we study four-dimensional (4D) modular symmetric quark flavor models
without fine-tuning. Mass matrices become hierarchical as close to the modular
symmetric points depending on its residual charges. Actually the residual $Z_N$
symmetries with $N\geq 6$ can originate the large quark mass hierarchies. Also
the products of residual symmetries such as $Z_3\times Z_3\times Z_3$ have such
possibility. We study the quark flavor model with $\Gamma_6$ symmetry and
$A_4\times A_4\times A_4$ symmetry. We assume the vicinity of the cusp
$\tau=i\infty$ where residual $Z_6$ and $Z_3\times Z_3\times Z_3$ symmetry
remain. Consequently we find the models realizing the order of the quark mass
ratios and the absolute values of the Cabibbo-Kobayashi-Maskawa (CKM) matrix
elements without fine-tuning.
  Finally we construct the Siegel modular forms. Zero-modes on $T^6$ at
$\vec{z}=0$ are the Siegel modular forms of weight 1/2 for the subgroup of
$Sp(6,\mathbb{Z})$. They have several moduli parameters and therefore have the
possibility realizing the flavor structures including the CP phases. We study
the Siegel modular forms transformed by $\widetilde{\Delta}(96)$ and show
numerical example. We find one of moduli parameters $\omega_1$ works on the
large mass hierarchies and $\omega_2$ works on the CP violation successfully in
our model.
