  The distributed and continuous representations used by neural networks are at
odds with representations employed in linguistics, which are typically
symbolic. Vector quantization has been proposed as a way to induce discrete
neural representations that are closer in nature to their linguistic
counterparts. However, it is not clear which metrics are the best-suited to
analyze such discrete representations. We compare the merits of four commonly
used metrics in the context of weakly supervised models of spoken language. We
compare the results they show when applied to two different models, while
systematically studying the effect of the placement and size of the
discretization layer. We find that different evaluation regimes can give
inconsistent results. While we can attribute them to the properties of the
different metrics in most cases, one point of concern remains: the use of
minimal pairs of phoneme triples as stimuli disadvantages larger discrete unit
inventories, unlike metrics applied to complete utterances. Furthermore, while
in general vector quantization induces representations that correlate with
units posited in linguistics, the strength of this correlation is only
moderate.
