  The quest for performant networks has been a significant force that drives
the advancements of deep learning in recent years. While rewarding, improving
network design has never been an easy journey. The large design space combined
with the tremendous cost required for network training poses a major obstacle
to this endeavor. In this work, we propose a new approach to this problem,
namely, predicting the performance of a network before training, based on its
architecture. Specifically, we develop a unified way to encode individual
layers into vectors and bring them together to form an integrated description
via LSTM. Taking advantage of the recurrent network's strong expressive power,
this method can reliably predict the performances of various network
architectures. Our empirical studies showed that it not only achieved accurate
predictions but also produced consistent rankings across datasets -- a key
desideratum in performance prediction.
