  It is shown that the so-called "Yukawaon" model can give a unified
description of masses, mixing and $CP$ violation parameters of quarks and
leptons without using any hierarchical (family number-dependent) parameters
besides the charged lepton masses. Here, we have introduced a phase matrix
$P={\rm daig}(e^{i \phi_1}, e^{i \phi_2}, e^{i \phi_3})$ with the phase
parameters $(\phi_1, \phi_2, \phi_3)$ which are described in terms of family
number-independent parameters, together with using only the charged lepton mass
parameters as the family number-dependent parameters. In this paper, the $CP$
violating phase parameters $\delta_{CP}^q$ and $\delta_{CP}^\ell$ in the
standard expression of $V_{CKM}$ and $U_{PMNS}$ are predicted as $\delta_{CP}^q
\simeq 72^\circ$ and $\delta_{CP}^\ell \simeq -76^\circ$, respectively, i.e.
$\delta_{CP}^\ell \sim - \delta_{CP}^q$.
