  We propose a new building block, IdleBlock, which naturally prunes
connections within the block. To fully utilize the IdleBlock we break the
tradition of monotonic design in state-of-the-art networks, and introducing
hybrid composition with IdleBlock. We study hybrid composition on MobileNet v3
and EfficientNet-B0, two of the most efficient networks. Without any neural
architecture search, the deeper "MobileNet v3" with hybrid composition design
surpasses possibly all state-of-the-art image recognition network designed by
human experts or neural architecture search algorithms. Similarly, the
hybridized EfficientNet-B0 networks are more efficient than previous
state-of-the-art networks with similar computation budgets. These results
suggest a new simpler and more efficient direction for network design and
neural architecture search.
