  With the rising concern on model interpretability, the application of
eXplainable AI (XAI) tools on deepfake detection models has been a topic of
interest recently. In image classification tasks, XAI tools highlight pixels
influencing the decision given by a model. This helps in troubleshooting the
model and determining areas that may require further tuning of parameters. With
a wide range of tools available in the market, choosing the right tool for a
model becomes necessary as each one may highlight different sets of pixels for
a given image. There is a need to evaluate different tools and decide the best
performing ones among them. Generic XAI evaluation methods like insertion or
removal of salient pixels/segments are applicable for general image
classification tasks but may produce less meaningful results when applied on
deepfake detection models due to their functionality. In this paper, we perform
experiments to show that generic removal/insertion XAI evaluation methods are
not suitable for deepfake detection models. We also propose and implement an
XAI evaluation approach specifically suited for deepfake detection models.
