  While instruction-tuned language models have demonstrated impressive
zero-shot generalization, these models often struggle to generate accurate
responses when faced with instructions that fall outside their training set.
This paper presents Instructive Decoding (ID), a simple yet effective approach
that augments the efficacy of instruction-tuned models. Specifically, ID
adjusts the logits for next-token prediction in a contrastive manner, utilizing
predictions generated from a manipulated version of the original instruction,
referred to as a noisy instruction. This noisy instruction aims to elicit
responses that could diverge from the intended instruction yet remain
plausible. We conduct experiments across a spectrum of such noisy instructions,
ranging from those that insert semantic noise via random words to others like
'opposite' that elicit the deviated responses. Our approach achieves
considerable performance gains across various instruction-tuned models and
tasks without necessitating any additional parameter updates. Notably,
utilizing 'opposite' as the noisy instruction in ID, which exhibits the maximum
divergence from the original instruction, consistently produces the most
significant performance gains across multiple models and tasks.
