  Methods for learning word sense embeddings represent a single word with
multiple sense-specific vectors. These methods should not only produce
interpretable sense embeddings, but should also learn how to select which sense
to use in a given context. We propose an unsupervised model that learns sense
embeddings using a modified Gumbel softmax function, which allows for
differentiable discrete sense selection. Our model produces sense embeddings
that are competitive (and sometimes state of the art) on multiple similarity
based downstream evaluations. However, performance on these downstream
evaluations tasks does not correlate with interpretability of sense embeddings,
as we discover through an interpretability comparison with competing
multi-sense embeddings. While many previous approaches perform well on
downstream evaluations, they do not produce interpretable embeddings and learn
duplicated sense groups; our method achieves the best of both worlds.
