  Given the recent interest in arguably accurate yet non-interpretable neural
models, even with textual features, for document ranking we try to answer
questions relating to how to interpret rankings. In this paper we take first
steps towards a framework for the interpretability of retrieval models with the
aim of answering 3 main questions "What is the intent of the query according to
the ranker?", "Why is a document ranked higher than another for the query?" and
"Why is a document relevant to the query?"
  Our framework is predicated on the assumption that text based retrieval model
behavior can be estimated using query expansions in conjunction with a simpler
retrieval model irrespective of the underlying ranker. We conducted experiments
with the Clueweb test collection. We show how our approach performs for both
simpler models with a closed form notation (which allows us to measure the
accuracy of the interpretation) and neural ranking models. Our results indicate
that we can indeed interpret more complex models with reasonable accuracy under
certain simplifying assumptions. In a case study we also show our framework can
be employed to interpret the results of the DRMM neural retrieval model in
various scenarios.
