  Explaining the behavior of black box machine learning models through human
interpretable rules is an important research area. Recent work has focused on
explaining model behavior locally i.e. for specific predictions as well as
globally across the fields of vision, natural language, reinforcement learning
and data science. We present a novel model-agnostic approach that derives rules
to globally explain the behavior of classification models trained on numerical
and/or categorical data. Our approach builds on top of existing local model
explanation methods to extract conditions important for explaining model
behavior for specific instances followed by an evolutionary algorithm that
optimizes an information theory based fitness function to construct rules that
explain global model behavior. We show how our approach outperforms existing
approaches on a variety of datasets. Further, we introduce a parameter to
evaluate the quality of interpretation under the scenario of distributional
shift. This parameter evaluates how well the interpretation can predict model
behavior for previously unseen data distributions. We show how existing
approaches for interpreting models globally lack distributional robustness.
Finally, we show how the quality of the interpretation can be improved under
the scenario of distributional shift by adding out of distribution samples to
the dataset used to learn the interpretation and thereby, increase robustness.
All of the datasets used in our paper are open and publicly available. Our
approach has been deployed in a leading digital marketing suite of products.
