  Applying language models to natural language processing tasks typically
relies on the representations in the final model layer, as intermediate hidden
layer representations are presumed to be less informative. In this work, we
argue that due to the gradual improvement across model layers, additional
information can be gleaned from the contrast between higher and lower layers
during inference. Specifically, in choosing between the probable next token
predictions of a generative model, the predictions of lower layers can be used
to highlight which candidates are best avoided. We propose a novel approach
that utilizes the contrast between layers to improve text generation outputs,
and show that it mitigates degenerative behaviors of the model in open-ended
generation, significantly improving the quality of generated texts.
Furthermore, our results indicate that contrasting between model layers at
inference time can yield substantial benefits to certain aspects of general
language model capabilities, more effectively extracting knowledge during
inference from a given set of model parameters.
