  Neural Architecture Search (NAS), the process of automating architecture
engineering, is an appealing next step to advancing end-to-end Automatic Speech
Recognition (ASR), replacing expert-designed networks with learned,
task-specific architectures. In contrast to early computational-demanding NAS
methods, recent gradient-based NAS methods, e.g., DARTS (Differentiable
ARchiTecture Search), SNAS (Stochastic NAS) and ProxylessNAS, significantly
improve the NAS efficiency. In this paper, we make two contributions. First, we
rigorously develop an efficient NAS method via Straight-Through (ST) gradients,
called ST-NAS. Basically, ST-NAS uses the loss from SNAS but uses ST to
back-propagate gradients through discrete variables to optimize the loss, which
is not revealed in ProxylessNAS. Using ST gradients to support sub-graph
sampling is a core element to achieve efficient NAS beyond DARTS and SNAS.
Second, we successfully apply ST-NAS to end-to-end ASR. Experiments over the
widely benchmarked 80-hour WSJ and 300-hour Switchboard datasets show that the
ST-NAS induced architectures significantly outperform the human-designed
architecture across the two datasets. Strengths of ST-NAS such as architecture
transferability and low computation cost in memory and time are also reported.
