  We propose models in which the hierarchical structures of the masses and
mixing in both quark and lepton sectors are explained by the $S_4^\prime$
modular flavor symmetry near the fixed point $\tau \sim i\infty$. The model
provides the first explicit example which explains hierarchies of both quarks
and leptons. The hierarchies are realized by powers of $\epsilon = e^{2\pi i
\tau/4} = \mathcal{O}(0.01)$ and $2\,\mathrm{Im}\,\tau \sim 5$, where $\tau$
being the modulus. The small parameter $\epsilon$ plays a role of flavon in the
Froggatt-Nielsen mechanism under the residual $Z_4^T$ symmetry, and powers of
$2\,\mathrm{Im}\,\tau$ in the Yukawa couplings are controlled by modular
weights via the canonical normalization. The doublet quarks are identified to a
$S_4^\prime$ triplet to explain the hierarchical structure of the quark mixing
angles, while the doublet leptons are composed of three singlets for the large
mixing angles in the lepton sector. We show that the $S_4^\prime$ modular
symmetry alone can explain the hierarchies in both quark and lepton sectors by
$\mathcal{O}(1)$ coefficients.
