  We conduct a large empirical evaluation to investigate the landscape of
distributional robustness in question answering. Our investigation spans over
350 models and 16 question answering datasets, including a diverse set of
architectures, model sizes, and adaptation methods (e.g., fine-tuning, adapter
tuning, in-context learning, etc.). We find that, in many cases, model
variations do not affect robustness and in-distribution performance alone
determines out-of-distribution performance. Moreover, our findings indicate
that i) zero-shot and in-context learning methods are more robust to
distribution shifts than fully fine-tuned models; ii) few-shot prompt
fine-tuned models exhibit better robustness than few-shot fine-tuned span
prediction models; iii) parameter-efficient and robustness enhancing training
methods provide no significant robustness improvements. In addition, we
publicly release all evaluations to encourage researchers to further analyze
robustness trends for question answering models.
