  Machine Reading Comprehension(MRC) has achieved a remarkable result since
some powerful models, such as BERT, are proposed. However, these models are not
robust enough and vulnerable to adversarial input perturbation and
generalization examples. Some works tried to improve the performance on
specific types of data by adding some related examples into training data while
it leads to degradation on the original dataset, because the shift of data
distribution makes the answer ranking based on the softmax probability of model
unreliable. In this paper, we propose a method to improve the robustness by
using a calibrator as the post-hoc reranker, which is implemented based on
XGBoost model. The calibrator combines both manual features and representation
learning features to rerank candidate results. Experimental results on
adversarial datasets show that our model can achieve performance improvement by
more than 10\% and also make improvement on the original and generalization
datasets.
