  Label-free model evaluation, or AutoEval, estimates model accuracy on
unlabeled test sets, and is critical for understanding model behaviors in
various unseen environments. In the absence of image labels, based on dataset
representations, we estimate model performance for AutoEval with regression. On
the one hand, image feature is a straightforward choice for such
representations, but it hampers regression learning due to being unstructured
(\ie no specific meanings for component at certain location) and of
large-scale. On the other hand, previous methods adopt simple structured
representations (like average confidence or average feature), but insufficient
to capture the data characteristics given their limited dimensions. In this
work, we take the best of both worlds and propose a new semi-structured dataset
representation that is manageable for regression learning while containing rich
information for AutoEval. Based on image features, we integrate distribution
shapes, clusters, and representative samples for a semi-structured dataset
representation. Besides the structured overall description with distribution
shapes, the unstructured description with clusters and representative samples
include additional fine-grained information facilitating the AutoEval task. On
three existing datasets and 25 newly introduced ones, we experimentally show
that the proposed representation achieves competitive results. Code and dataset
are available at
https://github.com/sxzrt/Semi-Structured-Dataset-Representations.
