  Instruction tuning has become an integral part of training pipelines for
Large Language Models (LLMs) and has been shown to yield strong performance
gains. In an orthogonal line of research, Annotation Error Detection (AED) has
emerged as a tool for detecting quality problems in gold standard labels. So
far, however, the application of AED methods has been limited to classification
tasks. It is an open question how well AED methods generalize to language
generation settings, which are becoming more widespread via LLMs. In this
paper, we present a first and novel benchmark for AED on instruction tuning
data: DONKII. It comprises three instruction-tuning datasets enriched with
error annotations by experts and semi-automatic methods. We also provide a
novel taxonomy of error types for instruction-tuning data. We find that all
three datasets contain clear errors, which sometimes propagate directly into
instruction-tuned LLMs. We propose four AED baselines for the generative
setting and evaluate them extensively on the newly introduced dataset. Our
results show that the choice of the right AED method and model size is indeed
crucial and derive practical recommendations for how to use AED methods to
clean instruction-tuning data.
