  Recent studies reveal that various biases exist in different NLP tasks, and
over-reliance on biases results in models' poor generalization ability and low
adversarial robustness. To mitigate datasets biases, previous works propose
lots of debiasing techniques to tackle specific biases, which perform well on
respective adversarial sets but fail to mitigate other biases. In this paper,
we propose a new debiasing method Sparse Mixture-of-Adapters (SMoA), which can
mitigate multiple dataset biases effectively and efficiently. Experiments on
Natural Language Inference and Paraphrase Identification tasks demonstrate that
SMoA outperforms full-finetuning, adapter tuning baselines, and prior strong
debiasing methods. Further analysis indicates the interpretability of SMoA that
sub-adapter can capture specific pattern from the training data and specialize
to handle specific bias.
