  Language models trained on large amounts of data require careful tuning to be
safely deployed in real world. We revisit the guided decoding paradigm, where
the goal is to augment the logits of the base language model using the scores
from a task-specific reward model. We propose a simple but efficient
parameterization of the autoregressive reward model enabling fast and effective
guided decoding. On detoxification and sentiment control tasks, we show that
our efficient parameterization performs on par with RAD, a strong but less
efficient guided decoding approach.
