  A fundamental feature of galaxies is their structure, yet we are just now
understanding the evolution of structural properties in quantitative ways. As
such, we explore the quantitative non-parametric structural evolution of 16,778
galaxies up to $z\sim3$ in all five CANDELS fields, the largest collection of
high resolution images of distant galaxies to date. Our goal is to investigate
how the structure, as opposed to size, surface brightness, or mass, changes
with time. In particular, we investigate how the concentration and asymmetry of
light evolve in the rest-frame optical. To interpret our galaxy structure
measurements, we also run and analyse 300 simulation realisations from
IllustrisTNG to determine the timescale of mergers for the CAS system. We
measure that from $z=0-3$, the median asymmetry merger timescale is
$0.56^{+0.23}_{-0.18}$Gyr, and find it does not vary with redshift. Using this
data, we find that galaxies become progressively asymmetric at a given mass at
higher redshifts and we derive merger rates which scale as
$\sim(1+z)^{1.87\pm0.04}$Gyr$^{-1}$, which agrees well with recent machine
learning and galaxy pair approaches, removing previous inconsistencies. We also
show that far-infrared selected galaxies that are invisible to \textit{HST}
have a negligible effect on our measurements. We also find that galaxies are
more concentrated at higher redshifts. We interpret this as a sign of how their
formation occurs from a smaller initial galaxy that later grows into a larger
one through mergers, consistent with the size growth of galaxies from
`inside-out', suggesting that the centres are the oldest parts of most
galaxies.
