  Models which estimate main effects of individual variables alongside
interaction effects have an identifiability challenge: effects can be freely
moved between main effects and interaction effects without changing the model
prediction. This is a critical problem for interpretability because it permits
"contradictory" models to represent the same function. To solve this problem,
we propose pure interaction effects: variance in the outcome which cannot be
represented by any smaller subset of features. This definition has an
equivalence with the Functional ANOVA decomposition. To compute this
decomposition, we present a fast, exact algorithm that transforms any
piecewise-constant function (such as a tree-based model) into a purified,
canonical representation. We apply this algorithm to Generalized Additive
Models with interactions trained on several datasets and show large disparity,
including contradictions, between the effects before and after purification.
These results underscore the need to specify data distributions and ensure
identifiability before interpreting model parameters.
