  Diffractive deep neural networks (D2NNs) are composed of successive
transmissive layers optimized using supervised deep learning to all-optically
implement various computational tasks between an input and output field-of-view
(FOV). Here, we present a pyramid-structured diffractive optical network design
(which we term P-D2NN), optimized specifically for unidirectional image
magnification and demagnification. In this design, the diffractive layers are
pyramidally scaled in alignment with the direction of the image magnification
or demagnification. This P-D2NN design creates high-fidelity magnified or
demagnified images in only one direction, while inhibiting the image formation
in the opposite direction - achieving the desired unidirectional imaging
operation using a much smaller number of diffractive degrees of freedom within
the optical processor volume. Furthermore, P-D2NN design maintains its
unidirectional image magnification/demagnification functionality across a large
band of illumination wavelengths despite being trained with a single
wavelength. We also designed a wavelength-multiplexed P-D2NN, where a
unidirectional magnifier and a unidirectional demagnifier operate
simultaneously in opposite directions, at two distinct illumination
wavelengths. Furthermore, we demonstrate that by cascading multiple
unidirectional P-D2NN modules, we can achieve higher magnification factors. The
efficacy of the P-D2NN architecture was also validated experimentally using
terahertz illumination, successfully matching our numerical simulations. P-D2NN
offers a physics-inspired strategy for designing task-specific visual
processors.
