  We examine a common origin of four-dimensional flavor, CP, and $U(1)_R$
symmetries in the context of heterotic string theory with standard embedding.
We find that flavor and $U(1)_R$ symmetries are unified into the $Sp(2h+2,
\mathbb{C})$ modular symmetries of Calabi-Yau threefolds with $h$ being the
number of moduli fields. Together with the $\mathbb{Z}_2^{\rm CP}$ CP symmetry,
they are enhanced to $GSp(2h+2, \mathbb{C})\simeq Sp(2h+2, \mathbb{C})\rtimes
\mathbb{Z}_2^{\rm CP}$ generalized symplectic modular symmetry. We exemplify
the $S_3, S_4, T^\prime, S_9$ non-Abelian flavor symmetries on explicit
toroidal orbifolds with and without resolutions and $\mathbb{Z}_2,S_4$ flavor
symmetries on three-parameter examples of Calabi-Yau threefolds. Thus,
non-trivial flavor symmetries appear in not only the exact orbifold limit but
also a certain class of Calabi-Yau threefolds. These flavor symmetries are
further enlarged to non-Abelian discrete groups by the CP symmetry.
