  Current work on human-machine alignment aims at understanding machine-learned
latent spaces and their correspondence to human representations.
G{\"a}rdenfors' conceptual spaces is a prominent framework for understanding
human representations. Convexity of object regions in conceptual spaces is
argued to promote generalizability, few-shot learning, and interpersonal
alignment. Based on these insights, we investigate the notion of convexity of
concept regions in machine-learned latent spaces. We develop a set of tools for
measuring convexity in sampled data and evaluate emergent convexity in layered
representations of state-of-the-art deep networks. We show that convexity is
robust to basic re-parametrization and, hence, meaningful as a quality of
machine-learned latent spaces. We find that approximate convexity is pervasive
in neural representations in multiple application domains, including models of
images, audio, human activity, text, and medical images. Generally, we observe
that fine-tuning increases the convexity of label regions. We find evidence
that pretraining convexity of class label regions predicts subsequent
fine-tuning performance.
