  We present an analysis of the structures, sizes, star formation rates, and
local environmental properties of galaxies at z~4-6, utilising deep Hubble
Space Telescope imaging of the Hubble Ultra Deep Field. The galaxies we study
are selected with the Lyman-break drop-out technique, using galaxies which are
B-,V-, and i-drops, which effectively selects UV bright starbursting galaxies
between z=4 and z=6. Our primary observational finding is that starbursting
galaxies at z>4 have a diversity in structure, with roughly 30% appearing
distorted and asymmetric, while the majority are smooth and apparently
undisturbed systems. We utilize several methods to compute the inferred
assembly rates for these distorted early galaxies including utilising the CAS
system and pair counts. Overall, we find a similar fraction of galaxies which
are in pairs as the fraction which have a distorted structure. Using the CAS
methodology, and our best-estimate for merger time-scales, we find that the
total number of inferred effective mergers for M_{*} > 10^9-10 M_sol galaxies
at z<6 is N_m = 4.2^{+4.1}_{-1.4}. The more common symmetrical systems display
a remarkable scaling relation between the concentration of light and their
half-light radii, revealing the earliest known galaxy scaling relationship, and
demonstrating that some galaxies at z>4 are likely in a relaxed state. Systems
which are asymmetric do not display a correlation between size and half-light
radii, and are generally larger than the symmetric smooth systems. The
time-scale for the formation of these smooth systems is 0.5-1 Gyr, suggesting
that most of these galaxies are formed through coordinated very rapid gas
collapses and star formation over a size of 1-2 kpc, or from merger events at z
> 10. (Abridged)
