  Feature attribution explains neural network outputs by identifying relevant
input features. The attribution has to be faithful, meaning that the attributed
features must mirror the input features that influence the output. One recent
trend to test faithfulness is to fit a model on designed data with known
relevant features and then compare attributions with ground truth input
features.This idea assumes that the model learns to use all and only these
designed features, for which there is no guarantee. In this paper, we solve
this issue by designing the network and manually setting its weights, along
with designing data. The setup, AttributionLab, serves as a sanity check for
faithfulness: If an attribution method is not faithful in a controlled
environment, it can be unreliable in the wild. The environment is also a
laboratory for controlled experiments by which we can analyze attribution
methods and suggest improvements.
