  The searching procedure of neural architecture search (NAS) is notoriously
time consuming and cost prohibitive.To make the search space continuous, most
existing gradient-based NAS methods relax the categorical choice of a
particular operation to a softmax over all possible operations and calculate
the weighted sum of multiple features, resulting in a large memory requirement
and a huge computation burden. In this work, we propose an efficient and novel
search strategy with meta kernels. We directly encode the supernet from the
perspective on convolution kernels and "shrink" multiple convolution kernel
candidates into a single one before these candidates operate on the input
feature. In this way, only a single feature is generated between two
intermediate nodes. The memory for storing intermediate features and the
resource budget for conducting convolution operations are both reduced
remarkably. Despite high efficiency, our search strategy can search in a more
fine-grained way than existing works and increases the capacity for
representing possible networks. We demonstrate the effectiveness of our search
strategy by conducting extensive experiments. Specifically, our method achieves
77.0% top-1 accuracy on ImageNet benchmark dataset with merely 357M FLOPs,
outperforming both EfficientNet and MobileNetV3 under the same FLOPs
constraints. Compared to models discovered by the start-of-the-art NAS method,
our method achieves the same (sometimes even better) performance, while faster
by three orders of magnitude.
