  Disentangled representation learning remains challenging as the underlying
factors of variation in the data do not naturally exist. The inherent
complexity of real-world data makes it unfeasible to exhaustively enumerate and
encapsulate all its variations within a finite set of factors. However, it is
worth noting that most real-world data have linguistic equivalents, typically
in the form of textual descriptions. These linguistic counterparts can
represent the data and effortlessly decomposed into distinct tokens. In light
of this, we present Vocabulary Disentangled Retrieval (VDR), a retrieval-based
framework that harnesses natural language as proxies of the underlying data
variation to drive disentangled representation learning. Our approach employ a
bi-encoder model to represent both data and natural language in a vocabulary
space, enabling the model to distinguish dimensions that capture intrinsic
characteristics within data through its natural language counterpart, thus
facilitating disentanglement. We extensively assess the performance of VDR
across 15 retrieval benchmark datasets, covering text-to-text and cross-modal
retrieval scenarios, as well as human evaluation. Our experimental results
compellingly demonstrate the superiority of VDR over previous bi-encoder
retrievers with comparable model size and training costs, achieving an
impressive 8.7% improvement in NDCG@10 on the BEIR benchmark, a 5.3% increase
on MS COCO, and a 6.0% increase on Flickr30k in terms of mean recall in the
zero-shot setting. Moreover, The results from human evaluation indicate that
interpretability of our method is on par with SOTA captioning models.
