  Many of the top question answering systems today utilize ensembling to
improve their performance on tasks such as the Stanford Question Answering
Dataset (SQuAD) and Natural Questions (NQ) challenges. Unfortunately most of
these systems do not publish their ensembling strategies used in their
leaderboard submissions. In this work, we investigate a number of ensembling
techniques and demonstrate a strategy which improves our F1 score for short
answers on the dev set for NQ by 2.3 F1 points over our single model (which
outperforms the previous SOTA by 1.9 F1 points).
