  Disentangled representations have recently been shown to improve fairness,
data efficiency and generalisation in simple supervised and reinforcement
learning tasks. To extend the benefits of disentangled representations to more
complex domains and practical applications, it is important to enable
hyperparameter tuning and model selection of existing unsupervised approaches
without requiring access to ground truth attribute labels, which are not
available for most datasets. This paper addresses this problem by introducing a
simple yet robust and reliable method for unsupervised disentangled model
selection. Our approach, Unsupervised Disentanglement Ranking (UDR), leverages
the recent theoretical results that explain why variational autoencoders
disentangle (Rolinek et al, 2019), to quantify the quality of disentanglement
by performing pairwise comparisons between trained model representations. We
show that our approach performs comparably to the existing supervised
alternatives across 5,400 models from six state of the art unsupervised
disentangled representation learning model classes. Furthermore, we show that
the ranking produced by our approach correlates well with the final task
performance on two different domains.
