  The presence of extremely compact galaxies at z~2 and their subsequent growth
in physical size has been the cause of much puzzlement. We revisit the question
using deep infrared Wide Field Camera 3 data to probe the rest-frame optical
structure of 935 host galaxies selected with 0.4<z<2.5 and stellar masses M* >
10^10.7 Msol using optical and near-infrared photometry in the UKIRT Ultra Deep
Survey and GOODS-South fields of the CANDELS survey. At each redshift, the most
compact sources are those with little or no star formation, and we find that
the mean size of these systems grows by a factor of 3.5 +- 0.3 over this
redshift interval. The new data are sufficiently deep to enable us to identify
companions to these hosts whose stellar masses are ten times smaller, while
still yielding suitably accurate photometric redshifts to define a likely
physical association. By searching for faint companions around 404 quiescent
hosts within a projected physical annulus 10 < R < 30 kpc/h, we estimate the
minor merger rate over the redshift range 0.4 < z < 2. After correcting for
contamination from projected pairs, we find that 13-18% of quiescent hosts have
likely physical companions with stellar mass ratios of 0.1 or greater. Mergers
of these companions will typically increase the host mass by 6+-2% per merger
timescale. We estimate the minimum growth rate necessary to explain the
declining abundance of compact galaxies. Using a simple model of merging
motivated by recent numerical simulations, we then assess whether mergers of
the faint companions with their hosts are sufficient to explain this minimal
rate. We find that mergers with mass ratios > 0.1 may explain most of the size
evolution observed at z >~ 1 if a relatively short merger timescale is assumed,
but the rapid growth seen at higher redshift likely requires additional
physical processes.
