  The even weight modular forms of level $N$ can be arranged into the common
irreducible representations of the inhomogeneous finite modular group
$\Gamma_N$ and the homogeneous finite modular group $\Gamma'_N$ which is the
double covering of $\Gamma_N$, and the odd weight modular forms of level $N$
transform in the new representations of $\Gamma'_N$. We find that the above
structure of modular forms can naturally generate texture zeros of the fermion
mass matrices if we properly assign the representations and weights of the
matter fields under the modular group. We perform a comprehensive analysis for
the $\Gamma'_3\cong T'$ modular symmetry. The three generations of left-handed
quarks are assumed to transform as a doublet and a singlet of $T'$, we find six
possible texture zeros structures of quark mass matrix up to row and column
permutations. We present five benchmark quark models which can produce very
good fit to the experimental data. These quark models are further extended to
include lepton sector, the resulting models can give a unified description of
both quark and lepton masses and flavor mixing simultaneously although they
contain less number of free parameters than the observables.
