  We use a sample of 8298 galaxies observed in the HST GOODS NICMOS Survey
(GNS) to construct the galaxy stellar mass function as a function of both
redshift and stellar mass up to z=3.5 and down to masses of Mstar=10^8.5 Msun
at z~1. We discover that a significant fraction of all massive Mstar>10^11 Msun
galaxies are in place up to the highest redshifts we probe, with a decreasing
fraction of lower mass galaxies present at all redshifts. This is an example of
`galaxy mass downsizing', and is the result of massive galaxies forming before
lower mass ones, and not just simply ending their star formation earlier as in
traditional downsizing scenarios. We find that the faint end slope is
significantly steeper than what is found in previous investigations. We
demonstrate that this steeper mass function better matches the stellar mass
added due to star formation, thereby alleviating some of the mismatch between
these two measures of the evolution of galaxy mass. We furthermore examine the
stellar mass function divided into blue/red systems, as well as for star
forming and non-star forming galaxies. We find a similar mass downsizing
present for both blue/red and star-forming/non-star forming galaxies, and that
the low mass galaxies are mostly all blue, and are therefore creating the steep
mass functions. We furthermore show that, although there is a downsizing such
that high mass galaxies are nearer their z=0 values at high redshift, this
turns over at masses Mstar~10^10 Msun, such that the lowest mass galaxies are
more common than galaxies at slight higher masses, creating a `dip' in the
observed galaxy mass function. We argue that the galaxy assembly process may be
driven by different mechanisms at low and high masses, and that the efficiency
of the galaxy formation process is lowest at masses Mstar~10^10 Msun at 1<z<3.
(Abridged)
