  Neural speech codec has recently gained widespread attention in generative
speech modeling domains, like voice conversion, text-to-speech synthesis, etc.
However, ensuring high-fidelity audio reconstruction of speech codecs under low
bitrate remains an open and challenging issue. In this paper, we propose
PromptCodec, a novel end-to-end neural speech codec using feature-aware prompt
encoders based on disentangled representation learning. By incorporating prompt
encoders to capture representations of additional input prompts, PromptCodec
can distribute the speech information requiring processing and enhance its
representation capabilities. Moreover, a simple yet effective adaptive feature
weighted fusion approach is introduced to integrate features of different
encoders. Meanwhile, we propose a novel disentangled representation learning
strategy based on structure similarity index measure to optimize PromptCodec's
encoders to ensure their efficiency, thereby further improving the performance
of PromptCodec. Experiments on LibriTTS demonstrate that our proposed
PromptCodec consistently outperforms state-of-the-art neural speech codec
models under all different bitrate conditions while achieving superior
performance with low bitrates.
