  Anaphora resolution is an important task for information extraction across a
range of languages, text genres, and domains, motivating the need for methods
that do not require large annotated datasets. In-context learning has emerged
as a promising approach, yet there are a number of challenges in applying
in-context learning to resolve anaphora. For example, encoding a single
in-context demonstration that consists of: an anaphor, a paragraph-length
context, and a list of corresponding antecedents, requires conditioning a
language model on a long sequence of tokens, limiting the number of
demonstrations per prompt. In this paper, we present MICE (Mixtures of
In-Context Experts), which we demonstrate is effective for few-shot anaphora
resolution in scientific protocols (Tamari et al., 2021). Given only a handful
of training examples, MICE combines the predictions of hundreds of in-context
experts, yielding a 30% increase in F1 score over a competitive prompt
retrieval baseline. Furthermore, we show MICE can be used to train compact
student models without sacrificing performance. As far as we are aware, this is
the first work to present experimental results demonstrating the effectiveness
of in-context learning on the task of few-shot anaphora resolution in
scientific protocols.
