  In-context learning (ICL) empowers generative models to address new tasks
effectively and efficiently on the fly, without relying on any artificially
crafted optimization techniques. In this paper, we study extending ICL to
address a broader range of tasks with an extended learning horizon and higher
improvement potential, namely General-Purpose In-Context Learning (GPICL). To
this end, we introduce two lightweight benchmarks specifically crafted to train
and evaluate GPICL functionalities. Each benchmark encompasses a vast number of
tasks characterized by significant task variance, facilitating meta-training
that minimizes inductive bias. These tasks are also crafted to promote
long-horizon in-context learning through continuous generation and interaction.
These characteristics necessitate the models to leverage contexts and history
interactions to enhance their capabilities, across domains such as language
modeling, decision-making, and world modeling. Our experiments on the baseline
models demonstrate that meta-training with minimal inductive bias and ICL from
the ground up is feasible across all the domains we've discussed. Additionally,
our findings indicate that the scale of parameters alone may not be crucial for
ICL or GPICL, suggesting alternative approaches such as increasing the scale of
contexts and memory states.
