  Satisfying the high computation demand of modern deep learning architectures
is challenging for achieving low inference latency. The current approaches in
decreasing latency only increase parallelism within a layer. This is because
architectures typically capture a single-chain dependency pattern that prevents
efficient distribution with a higher concurrency (i.e., simultaneous execution
of one inference among devices). Such single-chain dependencies are so
widespread that even implicitly biases recent neural architecture search (NAS)
studies. In this visionary paper, we draw attention to an entirely new space of
NAS that relaxes the single-chain dependency to provide higher concurrency and
distribution opportunities. To quantitatively compare these architectures, we
propose a score that encapsulates crucial metrics such as communication,
concurrency, and load balancing. Additionally, we propose a new generator and
transformation block that consistently deliver superior architectures compared
to current state-of-the-art methods. Finally, our preliminary results show that
these new architectures reduce the inference latency and deserve more
attention.
