  State-of-the-art sequence-to-sequence models often require autoregressive
decoding, which can be highly expensive. However, for some downstream tasks
such as out-of-distribution (OOD) detection and resource allocation, the actual
decoding output is not needed just a scalar attribute of this sequence. In
these scenarios, where for example knowing the quality of a system's output to
predict poor performance prevails over knowing the output itself, is it
possible to bypass the autoregressive decoding? We propose Non-Autoregressive
Proxy (NAP) models that can efficiently predict general scalar-valued
sequence-level attributes. Importantly, NAPs predict these metrics directly
from the encodings, avoiding the expensive autoregressive decoding stage. We
consider two sequence-to-sequence task: Machine Translation (MT); and Automatic
Speech Recognition (ASR). In OOD for MT, NAPs outperform a deep ensemble while
being significantly faster. NAPs are also shown to be able to predict
performance metrics such as BERTScore (MT) or word error rate (ASR). For
downstream tasks, such as data filtering and resource optimization, NAPs
generate performance predictions that outperform predictive uncertainty while
being highly inference efficient.
