  Local explanations of learning-to-rank (LTR) models are thought to extract
the most important features that contribute to the ranking predicted by the LTR
model for a single data point. Evaluating the accuracy of such explanations is
challenging since the ground truth feature importance scores are not available
for most modern LTR models. In this work, we propose a systematic evaluation
technique for explanations of LTR models. Instead of using black-box models,
such as neural networks, we propose to focus on tree-based LTR models, from
which we can extract the ground truth feature importance scores using decision
paths. Once extracted, we can directly compare the ground truth feature
importance scores to the feature importance scores generated with explanation
techniques. We compare two recently proposed explanation techniques for LTR
models when using decision trees and gradient boosting models on the MQ2008
dataset. We show that the explanation accuracy in these techniques can largely
vary depending on the explained model and even which data point is explained.
