  Machine question answering is an essential yet challenging task in natural
language processing. Recently, Pre-trained Contextual Embeddings (PCE) models
like Bidirectional Encoder Representations from Transformers (BERT) and A Lite
BERT (ALBERT) have attracted lots of attention due to their great performance
in a wide range of NLP tasks. In our Paper, we utilized the fine-tuned ALBERT
models and implemented combinations of additional layers (e.g. attention layer,
RNN layer) on top of them to improve model performance on Stanford Question
Answering Dataset (SQuAD 2.0). We implemented four different models with
different layers on top of ALBERT-base model, and two other models based on
ALBERT-xlarge and ALBERT-xxlarge. We compared their performance to our baseline
model ALBERT-base-v2 + ALBERT-SQuAD-out with details. Our best-performing
individual model is ALBERT-xxlarge + ALBERT-SQuAD-out, which achieved an F1
score of 88.435 on the dev set. Furthermore, we have implemented three
different ensemble algorithms to boost overall performance. By passing in
several best-performing models' results into our weighted voting ensemble
algorithm, our final result ranks first on the Stanford CS224N Test PCE SQuAD
Leaderboard with F1 = 90.123.
