  We extract an optimal subset of architectural parameters for the BERT
architecture from Devlin et al. (2018) by applying recent breakthroughs in
algorithms for neural architecture search. This optimal subset, which we refer
to as "Bort", is demonstrably smaller, having an effective (that is, not
counting the embedding layer) size of $5.5\%$ the original BERT-large
architecture, and $16\%$ of the net size. Bort is also able to be pretrained in
$288$ GPU hours, which is $1.2\%$ of the time required to pretrain the
highest-performing BERT parametric architectural variant, RoBERTa-large (Liu et
al., 2019), and about $33\%$ of that of the world-record, in GPU hours,
required to train BERT-large on the same hardware. It is also $7.9$x faster on
a CPU, as well as being better performing than other compressed variants of the
architecture, and some of the non-compressed variants: it obtains performance
improvements of between $0.3\%$ and $31\%$, absolute, with respect to
BERT-large, on multiple public natural language understanding (NLU) benchmarks.
