  There is observational evidence for inside-out growth of elliptical galaxies
since $z \gtrsim 2-3$, which is not driven by in-situ star formation. Many
systems at high redshift have small sizes $\sim 1kpc$ and surface brightness
profiles with low Sersic indices n. The most likely descendants have, on
average, grown by a factor of two in mass and a factor of four in size,
indicating $r \propto M^{\alpha}$ with $\alpha \gtrsim 2$. They also have
surface brightness profiles with $n \gtrsim 5$. This evolution can be
qualitatively explained on the basis of two assumptions: compact ellipticals
predominantly grow by collisionless minor or intermediate 'dry' mergers, and
they are embedded in massive dark matter halos. We draw these conclusions from
idealized collisionless mergers spheroidal galaxies - with and without dark
matter - with mass ratios of 1:1, 1:5, and 1:10. The sizes evolve as $r \propto
M^{\alpha}$ with $\alpha < 2$ for mass-ratios of 1:1. For minor mergers of
galaxies embedded in dark matter halos, the sizes grow significantly faster and
the profile shapes change more rapidly. Mergers with moderate mass-ratios of
1:5 give $\alpha \sim 2.3$ and a final Sersic index of $n = 9.5$ after doubling
the stellar mass. This is accompanied by a significant increase of the dark
matter fraction within the stellar half-mass radius, driven by the strong size
increase probing larger, dark matter dominated regions. Only a few intermediate
mass-ratio mergers of galaxies embedded in massive dark matter halos can result
in the observed concurrent inside-out growth and the rapid evolution in profile
shapes. Apart from negative stellar metallicity gradients such a 'minor' merger
scenario also predicts significantly lower dark matter fractions for $z \sim 2$
compact quiescent galaxies and their rare present day analogues (abbreviated).
