  Enhancing Language Models' (LMs) ability to understand purchase intentions in
E-commerce scenarios is crucial for their effective assistance in various
downstream tasks. However, previous approaches that distill intentions from LMs
often fail to generate meaningful and human-centric intentions applicable in
real-world E-commerce contexts. This raises concerns about the true
comprehension and utilization of purchase intentions by LMs. In this paper, we
present IntentionQA, a double-task multiple-choice question answering benchmark
to evaluate LMs' comprehension of purchase intentions in E-commerce.
Specifically, LMs are tasked to infer intentions based on purchased products
and utilize them to predict additional purchases. IntentionQA consists of 4,360
carefully curated problems across three difficulty levels, constructed using an
automated pipeline to ensure scalability on large E-commerce platforms. Human
evaluations demonstrate the high quality and low false-negative rate of our
benchmark. Extensive experiments across 19 language models show that they still
struggle with certain scenarios, such as understanding products and intentions
accurately, jointly reasoning with products and intentions, and more, in which
they fall far behind human performances. Our code and data are publicly
available at https://github.com/HKUST-KnowComp/IntentionQA.
