  We present a highly parameter efficient approach for Question Answering that
significantly reduces the need for extended BERT fine-tuning. Our method uses
information from the hidden state activations of each BERT transformer layer,
which is discarded during typical BERT inference. Our best model achieves
maximal BERT performance at a fraction of the training time and GPU or TPU
expense. Performance is further improved by ensembling our model with BERTs
predictions. Furthermore, we find that near optimal performance can be achieved
for QA span annotation using less training data. Our experiments show that this
approach works well not only for span annotation, but also for classification,
suggesting that it may be extensible to a wider range of tasks.
