  [Abridged] Context. Spectral differential imaging (SDI) is part of the
observing strategy of current and future high-contrast imaging instruments. It
aims to reduce the stellar speckles that prevent the detection of cool planets
by using in/out methane-band images. It attenuates the signature of off-axis
companions to the star, such as angular differential imaging (ADI). However,
this attenuation depends on the spectral properties of the low-mass companions
we are searching for. The implications of this particularity on estimating the
detection limits have been poorly explored so far. Aims. We perform an imaging
survey to search for cool (Teff<1000-1300 K) giant planets at separations as
close as 5-10 AU. We also aim to assess the sensitivity limits in SDI data
taking the photometric bias into account. This will lead to a better view of
the SDI performance. Methods. We observed a selected sample of 16 stars (age <
200 Myr, d < 25 pc) with the phase-mask coronagraph, SDI, and ADI modes of
VLT/NaCo. Results. We do not detect any companions. As for the sensitivity
limits, we argue that the SDI residual noise cannot be converted into mass
limits because it represents a differential flux, unlike the case of
single-band images. This results in degeneracies for the mass limits, which may
be removed with the use of single-band constraints. We instead employ a method
of directly determining the mass limits. The survey is sensitive to cool giant
planets beyond 10 AU for 65% and 30 AU for 100% of the sample. Conclusions. For
close-in separations, the optimal regime for SDI corresponds to SDI flux ratios
>2. According to the BT-Settl model, this translates into Teff<800 K. The
methods described here can be applied to the data interpretation of SPHERE. We
expect better performance with the dual-band imager IRDIS, thanks to more
suitable filter characteristics and better image quality.
