  Combining the public JWST/NIRCam imaging programs CEERS, PRIMER and JADES,
spanning a total area of $\sim$500 arcmin$^2$, we obtain a sample of $>$30,000
galaxies at $z\sim4-9$ that allows us to perform a complete, rest-optical
selected census of the galaxy population at $z>3$. Comparing the stellar mass
$M_*$ and the UV-slope $\beta$ distributions between JWST- and HST-selected
samples, we generally find very good agreement and no significant biases.
Nevertheless, JWST enables us to probe a small population of UV-red galaxies
that was missing from previous HST-based LBG samples. We measure galaxy stellar
mass functions (SMFs) at $z\sim4-9$ and show that they are broadly consistent
with existing literature results. However, UV-red galaxies dominate the
high-mass end of the SMF at least out to $z\sim6$. In particular the most
massive galaxies typically show very red colors between
$\lambda_{obs}\sim1.5\mu$m and $\sim4.5\mu$m, and thus JWST's unprecedented
resolution and sensitivity at these wavelengths yields more accurate
constraints on their abundance and masses. The implied redshift evolution of
the high-mass end of the SMF suggests a rapid build-up of massive dust-obscured
as well as quiescent galaxies from $z\sim6$ to $z\sim4$ as well as an enhanced
efficiency of star formation towards earlier times ($z\gtrsim6$). We find the
SMFs to be steep over the entire redshift range, and slightly steepening with
redshift from $z\sim 4-6$, reaching values of $\approx-2$ at $z\gtrsim6$.
Finally, we show that the galaxy mass density grows by a factor $\sim20\times$
in the $\sim1$ Gyr of cosmic time from $z\sim9$ to $z\sim4$. Our results
emphasize the importance of rest-frame optically-selected samples in inferring
accurate distributions of physical properties and studying the mass build-up of
galaxies in the first 1.5 Gyr of cosmic history.
