  Recently, work in NLP has shifted to few-shot (in-context) learning, with
large language models (LLMs) performing well across a range of tasks. However,
while fairness evaluations have become a standard for supervised methods,
little is known about the fairness of LLMs as prediction systems. Further,
common standard methods for fairness involve access to models weights or are
applied during finetuning, which are not applicable in few-shot learning. Do
LLMs exhibit prediction biases when used for standard NLP tasks? In this work,
we explore the effect of shots, which directly affect the performance of
models, on the fairness of LLMs as NLP classification systems. We consider how
different shot selection strategies, both existing and new demographically
sensitive methods, affect model fairness across three standard fairness
datasets. We discuss how future work can include LLM fairness evaluations.
