  Aligning model representations to humans has been found to improve robustness
and generalization. However, such methods often focus on standard observational
data. Synthetic data is proliferating and powering many advances in machine
learning; yet, it is not always clear whether synthetic labels are perceptually
aligned to humans -- rendering it likely model representations are not human
aligned. We focus on the synthetic data used in mixup: a powerful regularizer
shown to improve model robustness, generalization, and calibration. We design a
comprehensive series of elicitation interfaces, which we release as HILL MixE
Suite, and recruit 159 participants to provide perceptual judgments along with
their uncertainties, over mixup examples. We find that human perceptions do not
consistently align with the labels traditionally used for synthetic points, and
begin to demonstrate the applicability of these findings to potentially
increase the reliability of downstream models, particularly when incorporating
human uncertainty. We release all elicited judgments in a new data hub we call
H-Mix.
