  Machine Reading Comprehension (MRC) is an active field in natural language
processing with many successful developed models in recent years. Despite their
high in-distribution accuracy, these models suffer from two issues: high
training cost and low out-of-distribution accuracy. Even though some approaches
have been presented to tackle the generalization problem, they have high,
intolerable training costs. In this paper, we investigate the effect of
ensemble learning approach to improve generalization of MRC systems without
retraining a big model. After separately training the base models with
different structures on different datasets, they are ensembled using weighting
and stacking approaches in probabilistic and non-probabilistic settings. Three
configurations are investigated including heterogeneous, homogeneous, and
hybrid on eight datasets and six state-of-the-art models. We identify the
important factors in the effectiveness of ensemble methods. Also, we compare
the robustness of ensemble and fine-tuned models against data distribution
shifts. The experimental results show the effectiveness and robustness of the
ensemble approach in improving the out-of-distribution accuracy of MRC systems,
especially when the base models are similar in accuracies.
