  Search Result Explanation (SeRE) aims to improve search sessions'
effectiveness and efficiency by helping users interpret documents' relevance.
Existing works mostly focus on factual explanation, i.e. to find/generate
supporting evidence about documents' relevance to search queries. However,
research in cognitive sciences has shown that human explanations are
contrastive i.e. people explain an observed event using some counterfactual
events; such explanations reduce cognitive load and provide actionable
insights. Though already proven effective in machine learning and NLP
communities, there lacks a strict formulation on how counterfactual
explanations should be defined and structured, in the context of web search. In
this paper, we first discuss the possible formulation of counterfactual
explanations in the IR context. Next, we formulate a suite of desiderata for
counterfactual explanation in SeRE task and corresponding automatic metrics.
With this desiderata, we propose a method named
\textbf{C}ounter\textbf{F}actual \textbf{E}diting for Search Research
\textbf{E}xplanation (\textbf{CFE2}). CFE2 provides pairwise counterfactual
explanations for document pairs within a search engine result page. Our
experiments on five public search datasets demonstrate that CFE2 can
significantly outperform baselines in both automatic metrics and human
evaluations.
