  There is an increasing demand for interpretation of model predictions
especially in high-risk applications. Various visualization approaches have
been proposed to estimate the part of input which is relevant to a specific
model prediction. However, most approaches require model structure and
parameter details in order to obtain the visualization results, and in general
much effort is required to adapt each approach to multiple types of tasks
particularly when model backbone and input format change over tasks. In this
study, a simple yet effective visualization framework called PAMI is proposed
based on the observation that deep learning models often aggregate features
from local regions for model predictions. The basic idea is to mask majority of
the input and use the corresponding model output as the relative contribution
of the preserved input part to the original model prediction. For each input,
since only a set of model outputs are collected and aggregated, PAMI does not
require any model detail and can be applied to various prediction tasks with
different model backbones and input formats. Extensive experiments on multiple
tasks confirm the proposed method performs better than existing visualization
approaches in more precisely finding class-specific input regions, and when
applied to different model backbones and input formats. The source code will be
released publicly.
