Uncovering the Data-Related Limits of Human Reasoning Research: An
  Analysis based on Recommender Systems
  Understanding the fundamentals of human reasoning is central to the
development of any system built to closely interact with humans. Cognitive
science pursues the goal of modeling human-like intelligence from a
theory-driven perspective with a strong focus on explainability. Syllogistic
reasoning as one of the core domains of human reasoning research has seen a
surge of computational models being developed over the last years. However,
recent analyses of models' predictive performances revealed a stagnation in
improvement. We believe that most of the problems encountered in cognitive
science are not due to the specific models that have been developed but can be
traced back to the peculiarities of behavioral data instead.
  Therefore, we investigate potential data-related reasons for the problems in
human reasoning research by comparing model performances on human and
artificially generated datasets. In particular, we apply collaborative
filtering recommenders to investigate the adversarial effects of
inconsistencies and noise in data and illustrate the potential for data-driven
methods in a field of research predominantly concerned with gaining high-level
theoretical insight into a domain.
  Our work (i) provides insight into the levels of noise to be expected from
human responses in reasoning data, (ii) uncovers evidence for an upper-bound of
performance that is close to being reached urging for an extension of the
modeling task, and (iii) introduces the tools and presents initial results to
pioneer a new paradigm for investigating and modeling reasoning focusing on
predicting responses for individual human reasoners.
