  Recent studies on deep convolutional neural networks present a simple
paradigm of architecture design, i.e., models with more MACs typically achieve
better accuracy, such as EfficientNet and RegNet. These works try to enlarge
all the stages in the model with one unified rule by sampling and statistical
methods. However, we observe that some network architectures have similar MACs
and accuracies, but their allocations on computations for different stages are
quite different. In this paper, we propose to enlarge the capacity of CNN
models by improving their width, depth and resolution on stage level. Under the
assumption that the top-performing smaller CNNs are a proper subcomponent of
the top-performing larger CNNs, we propose an greedy network enlarging method
based on the reallocation of computations. With step-by-step modifying the
computations on different stages, the enlarged network will be equipped with
optimal allocation and utilization of MACs. On EfficientNet, our method
consistently outperforms the performance of the original scaling method. In
particular, with application of our method on GhostNet, we achieve
state-of-the-art 80.9% and 84.3% ImageNet top-1 accuracies under the setting of
600M and 4.4B MACs, respectively.
