  (Abridged) We present the evolution of the volume averaged properties of the
rest-frame optically luminous galaxy population to z~3, determined from four
disjoint deep fields with optical to near-infrared wavelength coverage. We
select galaxies above a rest-frame V-band luminosity of 3x10^10 Lsol and
characterize their rest-frame UV through optical properties via the mean
spectral energy distribution (SED). To measure evolution we apply the same
selection criteria to a sample of galaxies from the Sloan Digital Sky Survey
and COMBO-17. The mean rest-frame 2200Ang through V-band SED becomes steadily
bluer with increasing redshift but at z<3 the mean SED falls within the range
defined by ``normal'' galaxies in the nearby Universe. We measure stellar
mass-to-light ratios (Mstar/L) by fitting models to the rest-frame UV-optical
SEDs and derive the stellar mass density. The stellar mass density in luminous
galaxies has increased by a factor of 3.5-7.9 from z=3 to z=0.1, including
field-to-field variance uncertainties. After correcting to total, the measured
mass densities at z<2 lie below the integral of the star formation rate (SFR)
density as a function of redshift as derived from UV selected samples. This may
indicate a systematic error in the mass densities or SFR(z) estimates. We find
large discrepancies between recent model predictions for the evolution of the
mass density and our results, even when our observational selection is applied
to the models. Finally we determine that Distant Red Galaxies (selected to have
J_s - K_s>2.3) in our LV selected samples contribute 30% and 64% of the stellar
mass budget at z~2 and z~ 2.8 respectively. These galaxies are largely absent
from UV surveys and this result highlights the need for mass selection of high
redshift galaxies.
