  One of the most robust observations of the stellar initial mass function
(IMF) is its near-universality in the Milky Way and neighboring galaxies. But
recent observations of early-type galaxies can be interpreted to imply a
bottom-heavy IMF, while others of ultra-faint dwarfs could imply a top-heavy
IMF. This would impose powerful constraints on star formation models. We
explore what sort of cloud-scale IMF models could possibly satisfy these
constraints. We utilize simulated galaxies which reproduce (broadly) the
observed galaxy properties, while they also provide the detailed star formation
history and properties of each progenitor star-forming cloud. We then consider
generic models where the characteristic mass of the IMF is some arbitrary
power-law function of progenitor cloud properties, along with well-known
literature IMF models, which scale with Jeans mass, turbulent Bonnor-Ebert
mass, temperature, the opacity limit, metallicity, or the protostellar heating
mass. We show that no IMF models currently in the literature - nor any model
where the turnover mass is an arbitrary power-law function of a combination of
cloud temperature/density/size/metallicity/velocity dispersion/magnetic field -
can reproduce the claimed IMF variation in ellipticals or dwarfs without
severely violating observational constraints in the Milky Way. Specifically,
they predict too much variation in the extreme environments of the Galaxy,
compared to that observed. Either the IMF varies in a more complicated manner,
or alternative interpretations of the extragalactic observations must be
explored.
