  Large Language Models are affected by the phenomena of memorizing and
forgetting their training data. But how do these vary by model size? We work
towards this question by investigating how the model size affects the model's
ability to discriminate a word's meaning in a given context. We introduce a
dataset called DeltaWords, which evaluates a model's ability to follow
instructions to select a sentence which replaces the target word with its
antonym. We show a weak inverse scaling trend, where task accuracy degrades as
model size increase, under extremely few-shot prompting regimes. We show that
increasing the number of examples tend to disproportionately benefit larger
models than smaller models.
