  This work presents a framework based on feature disentanglement to learn
speaker embeddings that are robust to environmental variations. Our framework
utilises an auto-encoder as a disentangler, dividing the input speaker
embedding into components related to the speaker and other residual
information. We employ a group of objective functions to ensure that the
auto-encoder's code representation - used as the refined embedding - condenses
only the speaker characteristics. We show the versatility of our framework
through its compatibility with any existing speaker embedding extractor,
requiring no structural modifications or adaptations for integration. We
validate the effectiveness of our framework by incorporating it into two
popularly used embedding extractors and conducting experiments across various
benchmarks. The results show a performance improvement of up to 16%. We release
our code for this work to be available
https://github.com/kaistmm/voxceleb-disentangler
