  We present a flavor model with the $S_3$ modular invariance in the framework
of SU(5) GUT. The $S_3$ modular forms of weights $2$ and $4$ give the quark and
lepton mass matrices with a common complex parameter, the modulus $\tau$. The
GUT relation of down-type quarks and charged leptons is imposed by the VEV of
adjoint 24-dimensional Higgs multiplet in addition to the VEVs of $5$ and $\bar
5$ Higgs multiples of SU(5). The observed CKM and PMNS mixing parameters as
well as the mass eigenvalues are reproduced properly. We discuss the leptonic
CP phase and the effective mass of the neutrinoless double beta decay with the
sum of neutrino masses.
