  Large-scale datasets are essential to modern day deep learning. Advocates
argue that understanding these methods requires dataset transparency (e.g.
"dataset curation, motivation, composition, collection process, etc...").
However, almost no one has suggested the release of the detailed definitions
and visual category examples provided to annotators - information critical to
understanding the structure of the annotations present in each dataset. These
labels are at the heart of public datasets, yet few datasets include the
instructions that were used to generate them. We introduce a new task, Labeling
Instruction Generation, to address missing publicly available labeling
instructions. In Labeling Instruction Generation, we take a reasonably
annotated dataset and: 1) generate a set of examples that are visually
representative of each category in the dataset; 2) provide a text label that
corresponds to each of the examples. We introduce a framework that requires no
model training to solve this task and includes a newly created rapid retrieval
system that leverages a large, pre-trained vision and language model. This
framework acts as a proxy to human annotators that can help to both generate a
final labeling instruction set and evaluate its quality. Our framework
generates multiple diverse visual and text representations of dataset
categories. The optimized instruction set outperforms our strongest baseline
across 5 folds by 7.06 mAP for NuImages and 12.9 mAP for COCO.
