  Autoregressive neural network models have been used successfully for sequence
generation, feature extraction, and hypothesis scoring. This paper presents yet
another use for these models: allocating more computation to more difficult
inputs. In our model, an autoregressive model is used both to extract features
and to predict observations in a stream of input observations. The surprisal of
the input, measured as the negative log-likelihood of the current observation
according to the autoregressive model, is used as a measure of input
difficulty. This in turn determines whether a small, fast network, or a big,
slow network, is used. Experiments on two speech recognition tasks show that
our model can match the performance of a baseline in which the big network is
always used with 15% fewer FLOPs.
