  In this paper, we address the problem of evaluating whether results served by
an e-commerce search engine for a query are good or not. This is a critical
question in evaluating any e-commerce search engine. While this question is
traditionally answered using simple metrics like query click-through rate
(CTR), we observe that in e-commerce search, such metrics can be misleading.
Upon inspection, we find cases where CTR is high but the results are poor and
vice versa. Similar cases exist for other metrics like time to click which are
often also used for evaluating search engines. We aim to learn the quality of
the results served by the search engine based on users' interactions with the
results. Although this problem has been studied in the web search context, this
is the first study for e-commerce search, to the best of our knowledge. Despite
certain commonalities with evaluating web search engines, there are several
major differences such as underlying reasons for search failure, and
availability of rich user interaction data with products (e.g. adding a product
to the cart). We study large-scale user interaction logs from Flipkart's search
engine, analyze behavioral patterns and build models to classify queries based
on user behavior signals. We demonstrate the feasibility and efficacy of such
models in accurately predicting query performance. Our classifier is able to
achieve an average AUC of 0.75 on a held-out test set.
