  In-Context Learning (ICL) typically utilizes classification criteria from
probabilities of manually selected label tokens. However, we argue that such
token-based classification criteria lead to suboptimal decision boundaries,
despite delicate calibrations through translation and constrained rotation. To
address this problem, we propose Hidden Calibration, which renounces token
probabilities and uses the nearest centroid classifier on the LM's last hidden
states. In detail, we use the nearest centroid classification on the hidden
states, assigning the category of the nearest centroid previously observed from
a few-shot calibration set to the test sample as the predicted label. Our
experiments on 3 models and 10 classification datasets indicate that Hidden
Calibration consistently outperforms current token-based calibrations by about
20%. Our further analysis demonstrates that Hidden Calibration finds better
classification criteria with less inter-categories overlap, and LMs provide
linearly separable intra-category clusters with the help of demonstrations,
which supports Hidden Calibration and gives new insights into the conventional
ICL.
