  We present a comparison of the spatial distributions of Ly$\alpha$ emitters
(LAEs) and massive star-forming and quiescent galaxies (SFGs and QGs) at
$2<z<4.5$. We use the photometric redshift catalog to select SFGs and QGs and a
LAE catalog from intermediate/narrow bands obtained from the Subaru Telescope
and Isaac-Newton Telescope in Cosmic Evolution Survey (COSMOS). We derive the
auto-/cross- correlation signals of SFGs, QGs, and LAEs, and the galaxy
overdensity distributions at the position of them. Whereas the
cross-correlation signals of SFGs and QGs are explained solely by their halo
mass differences, those of SFGs and LAEs are significantly lower than those
expected from their auto-correlation signals, suggesting that some additional
physical processes are segregating these two populations. Such segregation of
SFGs and LAEs becomes stronger for rest-frame ultraviolet faint LAEs ($M_{\rm
UV}>-20$). From the overdensity distributions, LAEs are located in less dense
regions than SFGs and QGs, whereas SFGs and QGs tend to be in the same
overdensity distributions. The different spatial distributions of LAEs compared
to those of massive galaxies may be attributed to assembly bias or large
amounts of neutral hydrogen gas associated with massive halos. These results
reinforce the importance of exploring multiple galaxy populations in
quantifying the intrinsic galaxy environment of the high-$z$ universe.
