  The remarkably tight relationship between galaxy optical color and stellar
mass-to-light ratio ($M_*/L$) is widely used for efficient stellar mass
estimates. However, it remains unclear whether this low scatter comes from a
natural order in the galaxy population, or whether it is driven by simple
relationships in the models used to describe them. In this work we investigate
the origins of the relationship by contrasting the derived relationship from a
simple 4-parameter SED model with a more sophisticated 14-dimensional
Prospector-$\alpha$ model including nonparametric star formation histories
(SFHs). We apply these models to 63,430 galaxies at $0.5<z<3$ and fit a
hierarchical Bayesian model (HBM) to the population distribution in the
$(g-r)$--$\log(M/L_g)$ plane. We find that Prospector-$\alpha$ infers
systematically higher $M_*/L$ by 0.12 dex, a result of nonparametric SFHs
producing older ages, and also systematically redder rest-frame $(g-r)$ by 0.06
mag owing to the contribution from nebular emission. Surprisingly, the combined
effects of the $M_*/L$ and $(g-r)$ offsets produce a similar average
relationship for the two models, though Prospector-$\alpha$ produces a higher
scatter of 0.28 dex compared to the simple model of 0.12 dex. Also, unlike the
simple model, the Prospector-$\alpha$ relationship shows substantial redshift
evolution due to stellar aging. These expected and testable effects produce
overall older and redder galaxies, though the color--$M_*/L$ relationship is
measured only at $0.5<z<3$. Finally, we demonstrate that the HBM produces
substantial shrinkage in the individual posteriors of faint galaxies, an
important first step toward using the observed galaxy population directly to
inform the SED fitting priors.
