  Neural architecture search (NAS) has allowed for the automatic creation of
new and effective neural network architectures, offering an alternative to the
laborious process of manually designing complex architectures. However,
traditional NAS algorithms are slow and require immense amounts of computing
power. Recent research has investigated training-free NAS metrics for image
classification architectures, drastically speeding up search algorithms. In
this paper, we investigate training-free NAS metrics for recurrent neural
network (RNN) and BERT-based transformer architectures, targeted towards
language modeling tasks. First, we develop a new training-free metric, named
hidden covariance, that predicts the trained performance of an RNN architecture
and significantly outperforms existing training-free metrics. We experimentally
evaluate the effectiveness of the hidden covariance metric on the NAS-Bench-NLP
benchmark. Second, we find that the current search space paradigm for
transformer architectures is not optimized for training-free neural
architecture search. Instead, a simple qualitative analysis can effectively
shrink the search space to the best performing architectures. This conclusion
is based on our investigation of existing training-free metrics and new metrics
developed from recent transformer pruning literature, evaluated on our own
benchmark of trained BERT architectures. Ultimately, our analysis shows that
the architecture search space and the training-free metric must be developed
together in order to achieve effective results.
