  We present the UV-to-NIR size evolution of a sample of 161 quiescent galaxies
(QGs) with $M_*>10^{10}M_\odot$ over $0.5<z<5$. With deep multi-band NIRCam
images in GOODS-South from JADES, we measure the effective radii ($R_e$) of the
galaxies at rest-frame 0.3, 0.5 and 1$\mu m$. On average, QGs are 45% (15%)
more compact at rest-frame 1$\mu m$ than they are at 0.3$\mu m$ (0.5$\mu m$).
Regardless of wavelengths, the $R_e$ of QGs strongly evolves with redshift, and
this evolution depends on stellar mass. For lower-mass QGs with
$M_*=10^{10}-10^{10.6}M_\odot$, the evolution follows $R_e\sim(1+z)^{-1.1}$,
whereas it becomes steeper, following $R_e\sim(1+z)^{-1.7}$, for higher-mass
QGs with $M_*>10^{10.6}M_\odot$. To constrain the physical mechanisms driving
the apparent size evolution, we study the relationship between $R_e$ and the
formation redshift ($z_{form}$) of QGs. For lower-mass QGs, this relationship
is broadly consistent with $R_e\sim(1+z_{form})^{-1}$, in line with the
expectation of the progenitor effect. For higher-mass QGs, the relationship
between $R_e$ and $z_{form}$ depends on stellar age. Older QGs have a steeper
relationship between $R_e$ and $z_{form}$ than that expected from the
progenitor effect alone, suggesting that mergers and/or post-quenching
continuous gas accretion drive additional size growth in very massive systems.
We find that the $z>3$ QGs in our sample are very compact, with mass surface
densities $\Sigma_e\gtrsim10^{10} M_\odot/\rm{kpc}^2$, and their $R_e$ are
possibly even smaller than anticipated from the size evolution measured for
lower-redshift QGs. Finally, we take a close look at the structure of GS-9209,
one of the earliest confirmed massive QGs at $z_{spec}\sim4.7$. From UV to NIR,
GS-9209 becomes increasingly compact, and its light profile becomes more
spheroidal, showing that the color gradient is already present in this earliest
massive QG.
