  We study quark and lepton mass matrices with the common modulus $\tau$ in the
$A_4$ modular symmetry. The viable quark mass matrices are composed of modular
forms of weights $2$, $4$ and $6$. It is remarked that the modulus $\tau$ is
close to $i$, which is a fixed point in the fundamental region of SL$(2,Z)$,
and the CP symmetry is not violated. Indeed, the observed CP violation is
reproduced at $\tau$ which is deviated a little bit from $\tau=i$. The charged
lepton mass matrix is also given by using modular forms of weights $2$, $4$ and
$6$, where five cases have been examined. The neutrino mass matrix is generated
in terms of the modular forms of weight $4$ through the Weinberg operator.
Lepton mass matrices are also consistent with the observed mixing angles at
$\tau$ close to $i$ for NH of neutrino masses. Allowed regions of $\tau$ of
quarks and leptons overlap each other for all cases of the charged lepton mass
matrix. However, the sum of neutrino masses is crucial to test the common
$\tau$ for quarks and leptons. The minimal sum of neutrino masses $\sum m_i$ is
$140$meV at the common $\tau$.
  The inverted hierarchy of neutrino masses is unfavorable in our framework. It
is emphasized that our result suggests the residual symmetry
$\mathbb{Z}_2^{S}=\{ I, S \}$ in the quark and lepton mass matrices.
