  Interpretability methods in NLP aim to provide insights into the semantics
underlying specific system architectures. Focusing on word embeddings, we
present a supervised-learning method that, for a given domain (e.g., sports,
professions), identifies a subset of model features that strongly improve
prediction of human similarity judgments. We show this method keeps only 20-40%
of the original embeddings, for 8 independent semantic domains, and that it
retains different feature sets across domains. We then present two approaches
for interpreting the semantics of the retained features. The first obtains the
scores of the domain words (co-hyponyms) on the first principal component of
the retained embeddings, and extracts terms whose co-occurrence with the
co-hyponyms tracks these scores' profile. This analysis reveals that humans
differentiate e.g. sports based on how gender-inclusive and international they
are. The second approach uses the retained sets as variables in a probing task
that predicts values along 65 semantically annotated dimensions for a dataset
of 535 words. The features retained for professions are best at predicting
cognitive, emotional and social dimensions, whereas features retained for
fruits or vegetables best predict the gustation (taste) dimension. We discuss
implications for alignment between AI systems and human knowledge.
