  Promptly and accurately answering questions on products is important for
e-commerce applications. Manually answering product questions (e.g. on
community question answering platforms) results in slow response and does not
scale. Recent studies show that product reviews are a good source for
real-time, automatic product question answering (PQA). In the literature, PQA
is formulated as a retrieval problem with the goal to search for the most
relevant reviews to answer a given product question. In this paper, we focus on
the issue of answerability and answer reliability for PQA using reviews. Our
investigation is based on the intuition that many questions may not be
answerable with a finite set of reviews. When a question is not answerable, a
system should return nil answers rather than providing a list of irrelevant
reviews, which can have significant negative impact on user experience.
Moreover, for answerable questions, only the most relevant reviews that answer
the question should be included in the result. We propose a conformal
prediction based framework to improve the reliability of PQA systems, where we
reject unreliable answers so that the returned results are more concise and
accurate at answering the product question, including returning nil answers for
unanswerable questions. Experiments on a widely used Amazon dataset show
encouraging results of our proposed framework. More broadly, our results
demonstrate a novel and effective application of conformal methods to a
retrieval task.
