  In this paper we explore the parameter efficiency of BERT arXiv:1810.04805 on
version 2.0 of the Stanford Question Answering dataset (SQuAD2.0). We evaluate
the parameter efficiency of BERT while freezing a varying number of final
transformer layers as well as including the adapter layers proposed in
arXiv:1902.00751. Additionally, we experiment with the use of context-aware
convolutional (CACNN) filters, as described in arXiv:1709.08294v3, as a final
augmentation layer for the SQuAD2.0 tasks.
  This exploration is motivated in part by arXiv:1907.10597, which made a
compelling case for broadening the evaluation criteria of artificial
intelligence models to include various measures of resource efficiency. While
we do not evaluate these models based on their floating point operation
efficiency as proposed in arXiv:1907.10597, we examine efficiency with respect
to training time, inference time, and total number of model parameters. Our
results largely corroborate those of arXiv:1902.00751 for adapter modules,
while also demonstrating that gains in F1 score from adding context-aware
convolutional filters are not practical due to the increase in training and
inference time.
