  Any-to-any singing voice conversion (SVC) is confronted with the challenge of
``timbre leakage'' issue caused by inadequate disentanglement between the
content and the speaker timbre. To address this issue, this study introduces
NeuCoSVC, a novel neural concatenative SVC framework. It consists of a
self-supervised learning (SSL) representation extractor, a neural harmonic
signal generator, and a waveform synthesizer. The SSL extractor condenses audio
into fixed-dimensional SSL features, while the harmonic signal generator
leverages linear time-varying filters to produce both raw and filtered harmonic
signals for pitch information. The synthesizer reconstructs waveforms using SSL
features, harmonic signals, and loudness information. During inference, voice
conversion is performed by substituting source SSL features with their nearest
counterparts from a matching pool which comprises SSL features extracted from
the reference audio, while preserving raw harmonic signals and loudness from
the source audio. By directly utilizing SSL features from the reference audio,
the proposed framework effectively resolves the ``timbre leakage" issue caused
by previous disentanglement-based approaches. Experimental results demonstrate
that the proposed NeuCoSVC system outperforms the disentanglement-based speaker
embedding approach in one-shot SVC across intra-language, cross-language, and
cross-domain evaluations.
