  Most research on the interpretability of machine learning systems focuses on
the development of a more rigorous notion of interpretability. I suggest that a
better understanding of the deficiencies of the intuitive notion of
interpretability is needed as well. I show that visualization enables but also
impedes intuitive interpretability, as it presupposes two levels of technical
pre-interpretation: dimensionality reduction and regularization. Furthermore, I
argue that the use of positive concepts to emulate the distributed semantic
structure of machine learning models introduces a significant human bias into
the model. As a consequence, I suggest that, if intuitive interpretability is
needed, singular representations of internal model states should be avoided.
