  A number of studies point out that current Visual Question Answering (VQA)
models are severely affected by the language prior problem, which refers to
blindly making predictions based on the language shortcut. Some efforts have
been devoted to overcoming this issue with delicate models. However, there is
no research to address it from the angle of the answer feature space learning,
despite of the fact that existing VQA methods all cast VQA as a classification
task. Inspired by this, in this work, we attempt to tackle the language prior
problem from the viewpoint of the feature space learning. To this end, an
adapted margin cosine loss is designed to discriminate the frequent and the
sparse answer feature space under each question type properly. As a result, the
limited patterns within the language modality are largely reduced, thereby less
language priors would be introduced by our method. We apply this loss function
to several baseline models and evaluate its effectiveness on two VQA-CP
benchmarks. Experimental results demonstrate that our adapted margin cosine
loss can greatly enhance the baseline models with an absolute performance gain
of 15\% on average, strongly verifying the potential of tackling the language
prior problem in VQA from the angle of the answer feature space learning.
