  We study quark and lepton mass matrices in the $A_4$ modular symmetry towards
the unification of the quark and lepton flavors. We adopt modular forms of
weights $2$ and $6$ for quarks and charged leptons, while we use modular forms
of weight $4$ for the neutrino mass matrix which is generated by the Weinberg
operator. We obtain the successful quark mass matrices, in which the down-type
quark mass matrix is constructed by modular forms of weight $2$, but the
up-type quark mass matrix is constructed by modular forms of weight $6$. Two
regions of $\tau$ are consistent with observed CKM matrix elements. The one is
close to $\tau=i$ and the other is in the larger ${\rm Im }[\tau]$. On the
other hand, lepton mass matrices work well only at nearby $\tau=i$, which
overlaps with the one of the quark sector, for the normal hierarchy of neutrino
masses. In the common $\tau$ region for quarks and leptons, the predicted sum
of neutrino masses is $87$--$120$meV taking account of its cosmological bound.
Since both the Dirac CP phase $\delta_{CP}^\ell$ and $\sin^2\theta_{23}$ are
correlated with the sum of neutrino masses, improving its cosmological bound
provides crucial tests for our scheme as well as the precise measurement of
$\sin^2\theta_{23}$ and $\delta_{CP}^\ell$. The effective neutrino mass of the
$0\nu\beta\beta$ decay is $\langle m_{ee}\rangle=15$--$31$\,meV. It is remarked
that the modulus $\tau$ is fixed at nearby $\tau=i$ in the fundamental domain
of SL$(2,Z)$, which suggests the residual symmetry $Z_2$ in the quark and
lepton mass matrices. The inverted hierarchy of neutrino masses is excluded by
the cosmological bound of the sum of neutrino masses.
