  Post-hoc global/local feature attribution methods are progressively being
employed to understand the decisions of complex machine learning models. Yet,
because of limited amounts of data, it is possible to obtain a diversity of
models with good empirical performance but that provide very different
explanations for the same prediction, making it hard to derive insight from
them. In this work, instead of aiming at reducing the under-specification of
model explanations, we fully embrace it and extract logical statements about
feature attributions that are consistent across all models with good empirical
performance (i.e. all models in the Rashomon Set). We show that partial orders
of local/global feature importance arise from this methodology enabling more
nuanced interpretations by allowing pairs of features to be incomparable when
there is no consensus on their relative importance. We prove that every
relation among features present in these partial orders also holds in the
rankings provided by existing approaches. Finally, we present three use cases
employing hypothesis spaces with tractable Rashomon Sets (Additive models,
Kernel Ridge, and Random Forests) and show that partial orders allow one to
extract consistent local and global interpretations of models despite their
under-specification.
