  We study the textures of SM fermion mass matrices and their mixings in a
supersymmetric adjoint SU(5) Grand Unified Theory with modular $S_4$ being the
horizontal symmetry. The Yukawa entries of both quarks and leptons are
expressed by modular forms with lower weights. Neutrino sector has an adjoint
SU(5) representation 24 as matter superfield, which is a triplet of $S_4$. The
effective light neutrino masses is generated through Type-III and Type-I seesaw
mechanism. The only common complex parameter in both charged fermion and
neutrino sectors is modulus $\tau$. Down-type quarks and charged leptons have
the same joint effective operators with adjoint scalar in them, and their mass
discrepancy in the same generation depends on Clebsch-Gordan factor. Especially
for the first two generations the respective Clebsch-Gordan factors made the
double Yukawa ratio $y_dy_{\mu}/y_ey_s=12$, in excellent agreement with the
experimental result. We reproduce proper CKM mixing parameters and all nine
Yukawa eigenvalues of quarks and charged leptons. Neutrino masses and MNS
parameters are also produced properly with normal ordering is preferred.
