  Do different generative image models secretly learn similar underlying
representations? We investigate this by measuring the latent space similarity
of four different models: VAEs, GANs, Normalizing Flows (NFs), and Diffusion
Models (DMs). Our methodology involves training linear maps between frozen
latent spaces to "stitch" arbitrary pairs of encoders and decoders and
measuring output-based and probe-based metrics on the resulting "stitched''
models. Our main findings are that linear maps between latent spaces of
performant models preserve most visual information even when latent sizes
differ; for CelebA models, gender is the most similarly represented probe-able
attribute. Finally we show on an NF that latent space representations converge
early in training.
