  Computer generated holography has long been touted as the future of augmented
and virtual reality (AR/VR) displays, but has yet to be realized in practice.
Previous high-quality, color holographic displays have made either a 3$\times$
sacrifice on frame rate by using a sequential color illumination scheme or used
more than one spatial light modulator (SLM) and/or bulky, complex optical
setups. The reduced frame rate of sequential color introduces distracting
judder and color fringing in the presence of head motion while the form factor
of current simultaneous color systems is incompatible with a head-mounted
display. In this work, we propose a framework for simultaneous color holography
that allows the use of the full SLM frame rate while maintaining a compact and
simple optical setup. Simultaneous color holograms are optimized through the
use of a perceptual loss function, a physics-based neural network wavefront
propagator, and a camera-calibrated forward model. We measurably improve
hologram quality compared to other simultaneous color methods and move one step
closer to the realization of color holographic displays for AR/VR.
