  Despite their general capabilities, LLMs still struggle on biomedical NER
tasks, which are difficult due to the presence of specialized terminology and
lack of training data. In this work we set out to improve LLM performance on
biomedical NER in limited data settings via a new knowledge augmentation
approach which incorporates definitions of relevant concepts on-the-fly. During
this process, to provide a test bed for knowledge augmentation, we perform a
comprehensive exploration of prompting strategies. Our experiments show that
definition augmentation is useful for both open source and closed LLMs. For
example, it leads to a relative improvement of 15\% (on average) in GPT-4
performance (F1) across all (six) of our test datasets. We conduct extensive
ablations and analyses to demonstrate that our performance improvements stem
from adding relevant definitional knowledge. We find that careful prompting
strategies also improve LLM performance, allowing them to outperform fine-tuned
language models in few-shot settings. To facilitate future research in this
direction, we release our code at https://github.com/allenai/beacon.
