  One of the core goals of responsible AI development is ensuring high-quality
training datasets. Many researchers have pointed to the importance of the
annotation step in the creation of high-quality data, but less attention has
been paid to the work that enables data annotation. We define this work as the
design of ground truth schema and explore the challenges involved in the
creation of datasets in the medical domain even before any annotations are
made. Based on extensive work in three health-tech organisations, we describe
five external and internal factors that condition medical dataset creation
processes. Three external factors include regulatory constraints, the context
of creation and use, and commercial and operational pressures. These factors
condition medical data collection and shape the ground truth schema design. Two
internal factors include epistemic differences and limits of labelling. These
directly shape the design of the ground truth schema. Discussions of what
constitutes high-quality data need to pay attention to the factors that shape
and constrain what is possible to be created, to ensure responsible AI design.
