  Machine learning models tend to over-rely on statistical shortcuts. These
spurious correlations between parts of the input and the output labels does not
hold in real-world settings. We target this issue on the recent open-ended
visual counting task which is well suited to study statistical shortcuts. We
aim to develop models that learn a proper mechanism of counting regardless of
the output label. First, we propose the Modifying Count Distribution (MCD)
protocol, which penalizes models that over-rely on statistical shortcuts. It is
based on pairs of training and testing sets that do not follow the same count
label distribution such as the odd-even sets. Intuitively, models that have
learned a proper mechanism of counting on odd numbers should perform well on
even numbers. Secondly, we introduce the Spatial Counting Network (SCN), which
is dedicated to visual analysis and counting based on natural language
questions. Our model selects relevant image regions, scores them with fusion
and self-attention mechanisms, and provides a final counting score. We apply
our protocol on the recent dataset, TallyQA, and show superior performances
compared to state-of-the-art models. We also demonstrate the ability of our
model to select the correct instances to count in the image. Code and datasets
are available: https://github.com/cdancette/spatial-counting-network
