  We present an analysis of $\sim$1500 H160-selected photometric galaxies
detected to a limiting magnitude of 27.8 in the HUDF, using imaging from the
HST WFC3/IR camera in combination with archival UV, optical, and NIR imaging.
We fit photometric redshifts and stellar population estimates for all galaxies
with well-determined Spitzer IRAC fluxes, allowing for the determination of the
cumulative mass function within the range $1<z<6$. By selecting samples of
galaxies at a constant cumulative number density, we explore the co-evolution
of stellar masses and star formation rates from z$\sim$6. We find a steady
increase in the SFRs of galaxies at constant number density from z$\sim$6 to
z$\sim$3. The peak epoch of star formation is found to shift to later times for
galaxies with increasing number densities, in agreement with the expectations
from cosmic downsizing. The observed SFRs can fully account for the mass growth
to z$\sim$2 amongst galaxies with cumulative number densities greater than
10$^{-3.5}$ Mpc$^{-3}$. For galaxies with a lower constant number density we
find the observed stellar masses are $\sim$3 times greater than that which may
be accounted for by the observed star formation alone at late times, implying
that growth from mergers plays an important role at $z<2$. We additionally
observe a decreasing sSFR, equivalent to approximately one order of magnitude,
from z$\sim$6 to z$\sim$2 amongst galaxies with number densities less than
10$^{-3.5}$ Mpc$^{-3}$ along with significant evidence that at any redshift the
sSFR is higher for galaxies at higher number density. The combination of these
findings can qualitatively explain the previous findings of a sSFR plateau at
high redshift. Tracing the evolution of the fraction of quiescent galaxies for
samples matched in cumulative number density over this redshift range, we find
no unambiguous examples of quiescent galaxies at $z>4$.
