  The problem of explaining deep learning models, and model predictions
generally, has attracted intensive interest recently. Many successful
approaches forgo global approximations in order to provide more faithful local
interpretations of the model's behavior. LIME develops multiple interpretable
models, each approximating a large neural network on a small region of the data
manifold and SP-LIME aggregates the local models to form a global
interpretation. Extending this line of research, we propose a simple yet
effective method, NormLIME for aggregating local models into global and
class-specific interpretations. A human user study strongly favored
class-specific interpretations created by NormLIME to other feature importance
metrics. Numerical experiments confirm that NormLIME is effective at
recognizing important features.
