  Machine learning applied to architecture design presents a promising
opportunity with broad applications. Recent deep reinforcement learning (DRL)
techniques, in particular, enable efficient exploration in vast design spaces
where conventional design strategies may be inadequate. This paper proposes a
novel deep reinforcement framework, taking routerless networks-on-chip (NoC) as
an evaluation case study. The new framework successfully resolves problems with
prior design approaches being either unreliable due to random searches or
inflexible due to severe design space restrictions. The framework learns
(near-)optimal loop placement for routerless NoCs with various design
constraints. A deep neural network is developed using parallel threads that
efficiently explore the immense routerless NoC design space with a Monte Carlo
search tree. Experimental results show that, compared with conventional mesh,
the proposed deep reinforcement learning (DRL) routerless design achieves a
3.25x increase in throughput, 1.6x reduction in packet latency, and 5x
reduction in power. Compared with the state-of-the-art routerless NoC, DRL
achieves a 1.47x increase in throughput, 1.18x reduction in packet latency, and
1.14x reduction in average hop count albeit with slightly more power overhead.
