  Learning to disentangle and represent factors of variation in data is an
important problem in AI. While many advances have been made to learn these
representations, it is still unclear how to quantify disentanglement. While
several metrics exist, little is known on their implicit assumptions, what they
truly measure, and their limits. In consequence, it is difficult to interpret
results when comparing different representations. In this work, we survey
supervised disentanglement metrics and thoroughly analyze them. We propose a
new taxonomy in which all metrics fall into one of three families:
intervention-based, predictor-based and information-based. We conduct extensive
experiments in which we isolate properties of disentangled representations,
allowing stratified comparison along several axes. From our experiment results
and analysis, we provide insights on relations between disentangled
representation properties. Finally, we share guidelines on how to measure
disentanglement.
