  In this work, we investigate how the central stellar metallicity ([Z/H]) of
1363 galaxies from the SAMI galaxy survey is related to their stellar mass and
a proxy for the gravitational potential, $\Phi$ = log10(M/M*) -
log10($r_e$/kpc). In agreement with previous studies, we find that passive and
star-forming galaxies occupy different areas of the [Z/H]-M* plane, with
passive galaxies having higher [Z/H] than star-forming galaxies at fixed mass
(a difference of 0.23 dex at log10(M/M*)=10.3). We show for the first time that
all galaxies lie on the same relation between [Z/H] and $\Phi$, and show that
the offset in [Z/H] between passive and star-forming galaxies at fixed $\Phi$
is smaller than or equal to the offset in [Z/H] at fixed mass (an average
$\Delta$[Z/H] of 0.11 dex at fixed $\Phi$ compared to 0.21 dex at fixed mass).
We then build a simple model of galaxy evolution to explain and understand our
results. By assuming that [Z/H] traces $\Phi$ over cosmic time and that the
probability that a galaxy quenches depends on both its mass and size, we are
able to reproduce these offsets in stellar metallicity with a model containing
instantaneous quenching. We therefore conclude that an offset in metallicity at
fixed mass cannot by itself be used as evidence of slow quenching processes, in
contrast to previous studies. Instead, our model implies that metal-rich
galaxies have always been the smallest objects for their mass in a population.
Our findings reiterate the need to consider galaxy size when studying stellar
populations.
