  The peculiar velocity field measured by redshift-space distortions (RSD) in
galaxy surveys provides a unique probe of the growth of large-scale structure.
However, systematic effects arise when including satellite galaxies in the
clustering analysis. Since satellite galaxies tend to reside in massive halos
with a greater halo bias, the inclusion boosts the clustering power. In
addition, virial motions of the satellite galaxies cause a significant
suppression of the clustering power due to nonlinear RSD effects. We develop a
novel method to recover the redshift-space power spectrum of halos from the
observed galaxy distribution by minimizing the contamination of satellite
galaxies. The cylinder grouping method (CGM) we study effectively excludes
satellite galaxies from a galaxy sample. However, we find that this technique
produces apparent anisotropies in the reconstructed halo distribution over all
the scales which mimic RSD. On small scales, the apparent anisotropic
clustering is caused by exclusion of halos within the anisotropic cylinder used
by the CGM. On large scales, the misidentification of different halos in the
large-scale structures, aligned along the line-of-sight, into the same CGM
group, causes the apparent anisotropic clustering via their cross-correlation
with the CGM halos. We construct an empirical model for the CGM halo power
spectrum, which includes correction terms derived using the CGM window function
at small scales as well as the linear matter power spectrum multiplied by a
simple anisotropic function at large scales. We apply this model to a mock
galaxy catalog at z=0.5, designed to resemble SDSS-III BOSS CMASS galaxies, and
find that our model can predict both the monopole and quadrupole power spectra
of the host halos up to k<0.5 h/Mpc to within 5%.
