  Realization of deep learning with coherent diffraction has achieved
remarkable development nowadays, which benefits on the fact that matrix
multiplication can be optically executed in parallel as well as with little
power consumption. Coherent optical field propagated in the form of
complex-value entity can be manipulated into a task-oriented output with
statistical inference. In this paper, we present a unitary learning protocol on
deep diffractive neural network, meeting the physical unitary prior in coherent
diffraction. Unitary learning is a backpropagation serving to unitary weights
update through the gradient translation between Euclidean and Riemannian space.
The temporal-space evolution characteristic in unitary learning is formulated
and elucidated. Particularly a compatible condition on how to select the
nonlinear activations in complex space is unveiled, encapsulating the
fundamental sigmoid, tanh and quasi-ReLu in complex space. As a preliminary
application, deep diffractive neural network with unitary learning is
tentatively implemented on the 2D classification and verification tasks.
