  Feature attribution methods, which explain an individual prediction made by a
model as a sum of attributions for each input feature, are an essential tool
for understanding the behavior of complex deep learning models. However,
ensuring that models produce meaningful explanations, rather than ones that
rely on noise, is not straightforward. Exacerbating this problem is the fact
that attribution methods do not provide insight as to why features are assigned
their attribution values, leading to explanations that are difficult to
interpret. In real-world problems we often have sets of additional information
for each feature that are predictive of that feature's importance to the task
at hand. Here, we propose the deep attribution prior (DAPr) framework to
exploit such information to overcome the limitations of attribution methods.
Our framework jointly learns a relationship between prior information and
feature importance, as well as biases models to have explanations that rely on
features predicted to be important. We find that our framework both results in
networks that generalize better to out of sample data and admits new methods
for interpreting model behavior.
