  Large language models (LLMs) are capable to perform complex reasoning by
in-context learning (ICL) when provided with a few input-output demonstrations
(demos) and more powerful when intermediate reasoning steps ("chain of thoughts
(CoT)") of the demos are given. Is it necessary to use multi-demo in ICL? In
this paper, we study ICL using fewer demos for each test query on the tasks
in~\cite{wei2022chain}. Surprisingly, we do not observe significant degradation
when using only one randomly chosen demo. To study this phenomenon, for each
test query, we categorize demos into "correct demos" leading to the correct
answer, and "wrong demos" resulting in wrong answers. Our analysis reveals an
inherent bias in those widely studied datasets: most demos are correct for a
majority of test queries, which explains the good performance of using one
random demo. Moreover, ICL (with and w/o CoT) using only one correct demo
significantly outperforms all-demo ICL adopted by most previous works,
indicating the weakness of LLMs in finding correct demo(s) for input queries,
which is difficult to evaluate on the biased datasets. Furthermore, we observe
a counterintuitive behavior of ICL using multi-demo, i.e., its accuracy
degrades(improves) when given more correct(wrong) demos. This implies that ICL
can be easily misguided by interference among demos and their spurious
correlations. Our analyses highlight several fundamental challenges that need
to be addressed in LLMs training, ICL, and benchmark design.
