  Concept learning constructs visual representations that are connected to
linguistic semantics, which is fundamental to vision-language tasks. Although
promising progress has been made, existing concept learners are still
vulnerable to attribute perturbations and out-of-distribution compositions
during inference. We ascribe the bottleneck to a failure of exploring the
intrinsic semantic hierarchy of visual concepts, e.g. \{red, blue,...\} $\in$
`color' subspace yet cube $\in$ `shape'. In this paper, we propose a visual
superordinate abstraction framework for explicitly modeling semantic-aware
visual subspaces (i.e. visual superordinates). With only natural visual
question answering data, our model first acquires the semantic hierarchy from a
linguistic view, and then explores mutually exclusive visual superordinates
under the guidance of linguistic hierarchy. In addition, a quasi-center visual
concept clustering and a superordinate shortcut learning schemes are proposed
to enhance the discrimination and independence of concepts within each visual
superordinate. Experiments demonstrate the superiority of the proposed
framework under diverse settings, which increases the overall answering
accuracy relatively by 7.5\% on reasoning with perturbations and 15.6\% on
compositional generalization tests.
