  The query-based audio separation usually employs specific queries to extract
target sources from a mixture of audio signals. Currently, most query-based
separation models need additional networks to obtain query embedding. In this
way, separation model is optimized to be adapted to the distribution of query
embedding. However, query embedding may exhibit mismatches with separation
models due to inconsistent structures and independent information. In this
paper, we present CaRE-SEP, a consistent and relevant embedding network for
general sound separation to encourage a comprehensive reconsideration of query
usage in audio separation. CaRE-SEP alleviates the potential mismatch between
queries and separation in two aspects, including sharing network structure and
sharing feature information. First, a Swin-Unet model with a shared encoder is
conducted to unify query encoding and sound separation into one model,
eliminating the network architecture difference and generating consistent
distribution of query and separation features. Second, by initializing CaRE-SEP
with a pretrained classification network and allowing gradient backpropagation,
the query embedding is optimized to be relevant to the separation feature,
further alleviating the feature mismatch problem. Experimental results indicate
the proposed CaRE-SEP model substantially improves the performance of
separation tasks. Moreover, visualizations validate the potential mismatch and
how CaRE-SEP solves it.
