  Deep learning models have achieved remarkable success in different areas of
machine learning over the past decade; however, the size and complexity of
these models make them difficult to understand. In an effort to make them more
interpretable, several recent works focus on explaining parts of a deep neural
network through human-interpretable, semantic attributes. However, it may be
impossible to completely explain complex models using only semantic attributes.
In this work, we propose to augment these attributes with a small set of
uninterpretable features. Specifically, we develop a novel explanation
framework ELUDE (Explanation via Labelled and Unlabelled DEcomposition) that
decomposes a model's prediction into two parts: one that is explainable through
a linear combination of the semantic attributes, and another that is dependent
on the set of uninterpretable features. By identifying the latter, we are able
to analyze the "unexplained" portion of the model, obtaining insights into the
information used by the model. We show that the set of unlabelled features can
generalize to multiple models trained with the same feature space and compare
our work to two popular attribute-oriented methods, Interpretable Basis
Decomposition and Concept Bottleneck, and discuss the additional insights ELUDE
provides.
