  We study how half-mass radii, central mass densities ($\Sigma_1$), and color
gradients change as galaxies evolve. We separate $\sim7,000$ galaxies into
sixteen groups with similar spectral shapes; each group represents a different
evolutionary stage. We find that different galaxy types populate different
regions of both size-mass and $\Sigma_1$-mass space. The nine star-forming
groups lie along the integrated star-forming $\Sigma_1$-mass relation. However,
these star-forming groups form steep parallel relations in the size-mass plane,
with slopes similar to the quiescent size-mass relation. These steep slopes can
be explained as a transformation of the star-forming $\Sigma_1$-mass relation
and its scatter. We identify three types of transitional galaxies. Green valley
and post-starburst galaxies are similarly compact at $z>1.5$; however, their
distinct color gradients indicate that the two populations represent different
pathways to quenching. Post-starburst galaxies have flat color gradients and
compact structures, consistent with a fast quenching pathway which requires
structural change and operates primarily at high redshift. Green valley
galaxies have negative color gradients, and are both larger and more numerous
towards lower redshift. These galaxies are consistent with slow quenching
without significant structural change. We find that dusty star-forming galaxies
at $z\gtrsim2$ are very compact, and may represent the "burst" before
post-starburst galaxies; at $z\lesssim2$, dusty star-forming galaxies are
extended and have shallow color gradients consistent with slow quenching. Our
results suggest that star-forming galaxies grow gradually up the \sig-mass
relation until (a) they naturally reach the high $\Sigma_1$ values required for
quiescence, or (b) a compaction-type event rapidly increases their $\Sigma_1$.
