  Question answering methods are well-known for leveraging data bias, such as
the language prior in visual question answering and the position bias in
machine reading comprehension (extractive question answering). Current
debiasing methods often come at the cost of significant in-distribution
performance to achieve favorable out-of-distribution generalizability, while
non-debiasing methods sacrifice a considerable amount of out-of-distribution
performance in order to obtain high in-distribution performance. Therefore, it
is challenging for them to deal with the complicated changing real-world
situations. In this paper, we propose a simple yet effective novel loss
function with adaptive loose optimization, which seeks to make the best of both
worlds for question answering. Our main technical contribution is to reduce the
loss adaptively according to the ratio between the previous and current
optimization state on mini-batch training data. This loose optimization can be
used to prevent non-debiasing methods from overlearning data bias while
enabling debiasing methods to maintain slight bias learning. Experiments on the
visual question answering datasets, including VQA v2, VQA-CP v1, VQA-CP v2,
GQA-OOD, and the extractive question answering dataset SQuAD demonstrate that
our approach enables QA methods to obtain state-of-the-art in- and
out-of-distribution performance in most cases. The source code has been
released publicly in \url{https://github.com/reml-group/ALO}.
