  As one of the most exciting features of large language models (LLMs),
in-context learning is a mixed blessing. While it allows users to
fast-prototype a task solver with only a few training examples, the performance
is generally sensitive to various configurations of the prompt such as the
choice or order of the training examples. In this paper, we for the first time
theoretically and empirically identify that such a paradox is mainly due to the
label shift of the in-context model to the data distribution, in which LLMs
shift the label marginal $p(y)$ while having a good label conditional $p(x|y)$.
With this understanding, we can simply calibrate the in-context predictive
distribution by adjusting the label marginal, which is estimated via
Monte-Carlo sampling over the in-context model, i.e., generation of LLMs. We
call our approach as generative calibration. We conduct exhaustive experiments
with 12 text classification tasks and 12 LLMs scaling from 774M to 33B,
generally find that the proposed method greatly and consistently outperforms
the ICL as well as state-of-the-art calibration methods, by up to 27% absolute
in macro-F1. Meanwhile, the proposed method is also stable under different
prompt configurations.
