  While neural symbolic methods demonstrate impressive performance in visual
question answering on synthetic images, their performance suffers on real
images. We identify that the long-tail distribution of visual concepts and
unequal importance of reasoning steps in real data are the two key obstacles
that limit the models' real-world potentials. To address these challenges, we
propose a new paradigm, Calibrating Concepts and Operations (CCO), which
enables neural symbolic models to capture underlying data characteristics and
to reason with hierarchical importance. Specifically, we introduce an executor
with learnable concept embedding magnitudes for handling distribution
imbalance, and an operation calibrator for highlighting important operations
and suppressing redundant ones. Our experiments show CCO substantially boosts
the performance of neural symbolic methods on real images. By evaluating models
on the real world dataset GQA, CCO helps the neural symbolic method NSCL
outperforms its vanilla counterpart by 9.1% (from 47.0% to 56.1%); this result
also largely reduces the performance gap between symbolic and non-symbolic
methods. Additionally, we create a perturbed test set for better understanding
and analyzing model performance on real images. Code is available at
https://github.com/Lizw14/CaliCO.git .
