  The clustering signals of galaxy clusters are known to be powerful tools for
self-calibrating the mass-observable relation and are complementary to cluster
abundance and lensing. In this work, we explore the possibility of combining
three correlation functions -- cluster lensing, the cluster-galaxy
cross-correlation function, and the galaxy auto-correlation function -- to
self-calibrate optical cluster selection bias, the boosted clustering and
lensing signals in a richness-selected sample mainly caused by projection
effects. We develop mock catalogues of redMaGiC-like galaxies and
redMaPPer-like clusters by applying Halo Occupation Distribution (HOD) models
to N-body simulations and using counts-in-cylinders around massive haloes as a
richness proxy. In addition to the previously known small-scale boost in
projected correlation functions, we find that the projection effects also
significantly boost 3D correlation functions out to scales 100 $h^{-1}
\mathrm{Mpc}$. We perform a likelihood analysis assuming survey conditions
similar to that of the Dark Energy Survey (DES) and show that the selection
bias can be self-consistently constrained at the 10% level. We discuss
strategies for applying this approach to real data. We expect that expanding
the analysis to smaller scales and using deeper lensing data would further
improve the constraints on cluster selection bias.
