  Using a sample of ~28,000 sources selected at 3.6-4.5 microns with Spitzer
observations of the HDF-N, the CDF-S, and the Lockman Hole (surveyed area: ~664
arcmin^2), we study the evolution of the stellar mass content of the Universe
at 0<z<4. We calculate stellar masses and photometric redshifts, based on
~2,000 templates built with stellar and dust emission models fitting the
UV-to-MIR SEDs of galaxies with spectroscopic redshifts. We estimate stellar
mass functions for different redshift intervals. We find that 50% of the local
stellar mass density was assembled at 0<z<1 (average SFR:0.048 M_sun/yr/Mpc^3),
and at least another 40% at 1<z<4 (average SFR: 0.074 M_sun/yr/Mpc^3). Our
results confirm and quantify the ``downsizing'' scenario of galaxy formation.
The most massive galaxies (M>10^12.0 M_sun) assembled the bulk of their stellar
content rapidly (in 1-2 Gyr) beyond z~3 in very intense star formation events
(producing high specific SFRs). Galaxies with 10^11.5<M/M_sun<10^12.0 assembled
half of their stellar mass before z~1.5, and more than 90% of their mass was
already in place at z~0.6. Galaxies with M<10^11.5 M_sun evolved more slowly
(presenting smaller specific SFRs), assembling half of their stellar mass below
z~1. About 40% of the local stellar mass density of 10^9.0<M/M_sun<10^11.0
galaxies was assembled below z~0.4, most probably through accretion of small
satellites producing little star formation. The cosmic stellar mass density at
z>2.5 is dominated by optically faint (R>25) red galaxies (Distant Red Galaxies
or BzK sources) which account for ~30% of the global population of galaxies,
but contribute at least 60% to the cosmic stellar mass density. Bluer galaxies
(e.g., Lyman Break Galaxies) are more numerous but less massive, contributing
less than 50% to the global stellar mass density at high redshift.
