  Neural Architecture Search (NAS) benchmarks significantly improved the
capability of developing and comparing NAS methods while at the same time
drastically reduced the computational overhead by providing meta-information
about thousands of trained neural networks. However, tabular benchmarks have
several drawbacks that can hinder fair comparisons and provide unreliable
results. These usually focus on providing a small pool of operations in heavily
constrained search spaces -- usually cell-based neural networks with
pre-defined outer-skeletons. In this work, we conducted an empirical analysis
of the widely used NAS-Bench-101, NAS-Bench-201 and TransNAS-Bench-101
benchmarks in terms of their generability and how different operations
influence the performance of the generated architectures. We found that only a
subset of the operation pool is required to generate architectures close to the
upper-bound of the performance range. Also, the performance distribution is
negatively skewed, having a higher density of architectures in the upper-bound
range. We consistently found convolution layers to have the highest impact on
the architecture's performance, and that specific combination of operations
favors top-scoring architectures. These findings shed insights on the correct
evaluation and comparison of NAS methods using NAS benchmarks, showing that
directly searching on NAS-Bench-201, ImageNet16-120 and TransNAS-Bench-101
produces more reliable results than searching only on CIFAR-10. Furthermore,
with this work we provide suggestions for future benchmark evaluations and
design. The code used to conduct the evaluations is available at
https://github.com/VascoLopes/NAS-Benchmark-Evaluation.
