  The Spitzer Matching Survey of the UltraVISTA Ultra-deep Stripes (SMUVS) has
obtained the largest ultra-deep Spitzer maps to date in a single field of the
sky. We considered the sample of about 66,000 SMUVS sources at $z=2-6$ to
investigate the evolution of dusty and non-dusty galaxies with stellar mass
through the analysis of the galaxy stellar mass function (GSMF). We further
divide our non-dusty galaxy sample with rest-frame optical colours to isolate
red quiescent (`passive') galaxies. At each redshift, we identify a
characteristic stellar mass in the GSMF above which dusty galaxies dominate, or
are at least as important as non-dusty galaxies. Below that stellar mass,
non-dusty galaxies comprise about 80% of all sources, at all redshifts except
at $z=4-5$. The percentage of dusty galaxies at $z=4-5$ is unusually high:
30-40% for $M_{*}=10^9 - 10^{10.5} \, \rm M_\odot$ and $>80\%$ at $M_*>10^{11}
\, \rm M_\odot$, which indicates that dust obscuration is of major importance
in this cosmic period. The overall percentage of massive ($\log_{10}
(M_*/M_\odot)>10.6$) galaxies that are quiescent increases with decreasing
redshift, reaching $>30\%$ at $z\sim2$. Instead, the quiescent percentage among
intermediate-mass galaxies (with $\log_{10} (M_*/M_\odot)=9.7-10.6$) stays
roughly constant at a $\sim 10\%$ level. Our results indicate that massive and
intermediate-mass galaxies clearly have different evolutionary paths in the
young Universe, and are consistent with the scenario of galaxy downsizing.
