  Zero-shot voice conversion (VC) aims to transform source speech into
arbitrary unseen target voice while keeping the linguistic content unchanged.
Recent VC methods have made significant progress, but semantic losses in the
decoupling process as well as training-inference mismatch still hinder
conversion performance. In this paper, we propose Vec-Tok-VC+, a novel
prompt-based zero-shot VC model improved from Vec-Tok Codec, achieving voice
conversion given only a 3s target speaker prompt. We design a residual-enhanced
K-Means decoupler to enhance the semantic content extraction with a two-layer
clustering process. Besides, we employ teacher-guided refinement to simulate
the conversion process to eliminate the training-inference mismatch, forming a
dual-mode training strategy. Furthermore, we design a multi-codebook
progressive loss function to constrain the layer-wise output of the model from
coarse to fine to improve speaker similarity and content accuracy. Objective
and subjective evaluations demonstrate that Vec-Tok-VC+ outperforms the strong
baselines in naturalness, intelligibility, and speaker similarity.
