  In recent years, there has been increasing demand for automatic architecture
search in deep learning. Numerous approaches have been proposed and led to
state-of-the-art results in various applications, including image
classification and language modeling. In this paper, we propose a novel way of
architecture search by means of weighted networks (WeNet), which consist of a
number of networks, with each assigned a weight. These weights are updated with
back-propagation to reflect the importance of different networks. Such weighted
networks bear similarity to mixture of experts. We conduct experiments on Penn
Treebank and WikiText-2. We show that the proposed WeNet can find recurrent
architectures which result in state-of-the-art performance.
