  Spontaneous parametric down-conversion in quantum optics is an invaluable
resource for the realization of high-dimensional qudits with spatial modes of
light. One of the main open challenges is how to directly generate a desirable
qudit state in the SPDC process. This problem can be addressed through advanced
computational learning methods; however, due to difficulties in modeling the
SPDC process by a fully differentiable algorithm that takes into account all
interaction effects, progress has been limited. Here, we overcome these
limitations and introduce a physically-constrained and differentiable model,
validated against experimental results for shaped pump beams and structured
crystals, capable of learning every interaction parameter in the process. We
avoid any restrictions induced by the stochastic nature of our physical model
and integrate the dynamic equations governing the evolution under the SPDC
Hamiltonian. We solve the inverse problem of designing a nonlinear quantum
optical system that achieves the desired quantum state of down-converted photon
pairs. The desired states are defined using either the second-order
correlations between different spatial modes or by specifying the required
density matrix. By learning nonlinear volume holograms as well as different
pump shapes, we successfully show how to generate maximally entangled states.
Furthermore, we simulate all-optical coherent control over the generated
quantum state by actively changing the profile of the pump beam. Our work can
be useful for applications such as novel designs of high-dimensional quantum
key distribution and quantum information processing protocols. In addition, our
method can be readily applied for controlling other degrees of freedom of light
in the SPDC process, such as the spectral and temporal properties, and may even
be used in condensed-matter systems having a similar interaction Hamiltonian.
