  Imaging depth and spectrum have been extensively studied in isolation from
each other for decades. Recently, hyperspectral-depth (HS-D) imaging emerges to
capture both information simultaneously by combining two different imaging
systems; one for depth, the other for spectrum. While being accurate, this
combinational approach induces increased form factor, cost, capture time, and
alignment/registration problems. In this work, departing from the combinational
principle, we propose a compact single-shot monocular HS-D imaging method. Our
method uses a diffractive optical element (DOE), the point spread function of
which changes with respect to both depth and spectrum. This enables us to
reconstruct spectrum and depth from a single captured image. To this end, we
develop a differentiable simulator and a neural-network-based reconstruction
that are jointly optimized via automatic differentiation. To facilitate
learning the DOE, we present a first HS-D dataset by building a benchtop HS-D
imager that acquires high-quality ground truth. We evaluate our method with
synthetic and real experiments by building an experimental prototype and
achieve state-of-the-art HS-D imaging results.
