  In recent years, zero-cost metrics are gaining ground in neural architecture
search (NAS). There metrics allow finding the optimal neural network for a
given task faster and with a lesser computational load than conventional NAS
methods. Equally important is that they also shed some light on the internal
workings of neural architectures. This paper presents a zero-cost metric that
highly correlated with the train set accuracy across the NAS-Bench-101,
NAS-Bench-201 and NAS-Bench-NLP benchmark datasets. We evaluate a neural
achitecture's potential based on the outputs' statistics after two constant
shared weights initialisations. For this, we only use an unlabelled mini-batch
of data. We observe that the dispersion of the outputs between two
initialisations positively correlates with trained accuracy. The correlation
further improves when we normalise dispersion by average output magnitude. The
resulting metric, epsilon, does not require gradients computation and unbinds
the NAS procedure from training hyperparameters, loss metrics and
human-labelled data. Our method is easy to integrate within existing NAS
algorithms and takes a fraction of a second to evaluate a single network. The
code supporting this study can be found on GitHub at
https://github.com/egracheva/epsinas.
