  We have performed a systematical analysis of lepton and quark masses models
based on $\Gamma_4\cong S_4$ modular symmetry with gCP symmetry. We have
considered both cases that neutrinos are Majorana particles and Dirac
particles. All possible nontrivial representation assignments of matter fields
are considered, and the most general form of fermion mass matrices are given.
The phenomenologically viable models with the lowest number of free parameters
together with the results of fit are presented. We find out nine lepton models
with seven real free parameters including the real and imaginary parts of
modulus for Majorana neutrinos, which can accommodate the lepton masses and
neutrino oscillation data. The prediction for leptogenesis is studied in an
example lepton model. The observed baryon asymmetry as well as lepton masses
and mixing angles can be explained. For Dirac neutrinos, four lepton models
with five real free couplings are compatible with experimental data. Ten quark
models containing seven couplings are found to be able to accommodate the
hierarchical quark masses and mixing angles and CP violation phase.
Furthermore, the $S_4$ modular symmetry can provide a unified description of
lepton and quark flavor structure, and a benchmark model is presented.
