  We present the rest-frame optical sizes of massive quiescent galaxies (QGs)
at $z\sim4$ measured at $K'$-band with the Infrared Camera and Spectrograph
(IRCS) and AO188 on the Subaru telescope. Based on a deep multi-wavelength
catalog in the Subaru XMM-Newton Deep Survey Field (SXDS), covering a wide
wavelength range from the $u$-band to the IRAC $8.0\mu m$ over 0.7 deg$^2$, we
evaluate photometric redshift to identify massive ($M_{\star}\sim10^{11}\
M_\odot$) galaxies with suppressed star formation. These galaxies show a
prominent 4000$\rm \AA$ break feature at $z\sim4$, suggestive of an evolved
stellar population. We then conduct follow-up $K'$-band imaging with adaptive
optics for the five brightest galaxies ($K_{AB,total}=22.5\sim23.4$). Compared
to lower redshift ones, QGs at $z\sim4$ have smaller physical sizes of
effective radii $r_{eff}=0.2$ to $1.8$ kpc. The mean size measured by stacking
the four brightest objects is $r_{eff}=0.7\rm\ kpc$. This is the first
measurement of the rest-frame optical sizes of QGs at $z\sim4$. We evaluate the
robustness of our size measurements using simulations and find that our size
estimates are reasonably accurate with an expected systematic bias of $\sim0.2$
kpc. If we account for the stellar mass evolution, massive QGs at $z\sim4$ are
likely to evolve into the most massive galaxies today. We find their size
evolution with cosmic time in a form of $\log(r_e/{\rm kpc})= -0.44+1.77
\log(t/\rm Gyr)$. Their size growth is proportional to the square of stellar
mass, indicating the size-stellar mass growth driven by minor dry mergers.
