  We perform a comprehensive analysis of the homogeneous finite modular group
$\Gamma'_4\equiv S'_4$ which is the double covering of $S_4$ group. The weight
1 modular forms of level 4 are constructed in terms of Dedekind eta function,
and they transform as a triplet $\mathbf{\hat{3}'}$ of $S'_4$. The integral
weight modular forms until weight 6 are built from the tensor products of
weight 1 modular forms. We perform a systematical classification of $S'_4$
modular models for lepton masses and mixing with/without generalized CP, where
the left-handed leptons are assigned to triplet of $S'_4$ and right-handed
charged leptons transform as singlets under $S'_4$, and we consider both
scenarios where the neutrino masses arise from Weinberg operator or type I
seesaw mechanism. The phenomenological implications of the minimal models for
lepton masses, mixing angles, CP violation phases and neutrinoless double decay
are discussed. The $S'_4$ modular symmetry is extended to quark sector, we
present several predictive models which use nine or ten free parameters
including real and imaginary parts of $\tau$ to describe quark masses and
Cabibbo-Kobayashi-Maskawa mixing matrix. We give a quark-lepton unified model
which can explain the flavor structure of quarks and leptons simultaneously for
a common value of $\tau$.
