  Analytical reasoning is an essential and challenging task that requires a
system to analyze a scenario involving a set of particular circumstances and
perform reasoning over it to make conclusions. In this paper, we study the
challenge of analytical reasoning of text and introduce a new dataset
consisting of questions from the Law School Admission Test from 1991 to 2016.
We analyze what knowledge understanding and reasoning abilities are required to
do well on this task. Furthermore, to address this reasoning challenge, we
design two different baselines: (1) a Transformer-based method which leverages
the state-of-the-art pre-trained language models and (2) Analytical Reasoning
Machine (ARM), a logical-level reasoning framework extracting symbolic
knowledge (e.g, participants, facts, logical functions) to deduce legitimate
solutions. In our experiments, we find that the Transformer-based models
struggle to solve this task as their performance is close to random guess and
ARM achieves better performance by leveraging symbolic knowledge and
interpretable reasoning steps. Results show that both methods still lag far
behind human performance, which leave further space for future research.
