  In recent years, the number of new applications for highly complex AI systems
has risen significantly. Algorithmic decision-making systems (ADMs) are one of
such applications, where an AI system replaces the decision-making process of a
human expert. As one approach to ensure fairness and transparency of such
systems, explainable AI (XAI) has become more important. One variant to achieve
explainability are surrogate models, i.e., the idea to train a new simpler
machine learning model based on the input-output-relationship of a black box
model. The simpler machine learning model could, for example, be a decision
tree, which is thought to be intuitively understandable by humans. However,
there is not much insight into how well the surrogate model approximates the
black box.
  Our main assumption is that a good surrogate model approach should be able to
bring such a discriminating behavior to the attention of humans; prior to our
research we assumed that a surrogate decision tree would identify such a
pattern on one of its first levels. However, in this article we show that even
if the discriminated subgroup - while otherwise being the same in all
categories - does not get a single positive decision from the black box ADM
system, the corresponding question of group membership can be pushed down onto
a level as low as wanted by the operator of the system.
  We then generalize this finding to pinpoint the exact level of the tree on
which the discriminating question is asked and show that in a more realistic
scenario, where discrimination only occurs to some fraction of the
disadvantaged group, it is even more feasible to hide such discrimination.
  Our approach can be generalized easily to other surrogate models.
