  Deep Learning has become overly complicated and has enjoyed stellar success
in solving several classical problems like image classification, object
detection, etc. Several methods for explaining these decisions have been
proposed. Black-box methods to generate saliency maps are particularly
interesting due to the fact that they do not utilize the internals of the model
to explain the decision. Most black-box methods perturb the input and observe
the changes in the output. We formulate saliency map generation as a sequential
search problem and leverage upon Reinforcement Learning (RL) to accumulate
evidence from input images that most strongly support decisions made by a
classifier. Such a strategy encourages to search intelligently for the
perturbations that will lead to high-quality explanations. While successful
black box explanation approaches need to rely on heavy computations and suffer
from small sample approximation, the deterministic policy learned by our method
makes it a lot more efficient during the inference. Experiments on three
benchmark datasets demonstrate the superiority of the proposed approach in
inference time over state-of-the-arts without hurting the performance. Project
Page: https://cvir.github.io/projects/rexl.html
