  Differentiable simulations of optical systems can be combined with deep
learning-based reconstruction networks to enable high performance computational
imaging via end-to-end (E2E) optimization of both the optical encoder and the
deep decoder. This has enabled imaging applications such as 3D localization
microscopy, depth estimation, and lensless photography via the optimization of
local optical encoders. More challenging computational imaging applications,
such as 3D snapshot microscopy which compresses 3D volumes into single 2D
images, require a highly non-local optical encoder. We show that existing deep
network decoders have a locality bias which prevents the optimization of such
highly non-local optical encoders. We address this with a decoder based on a
shallow neural network architecture using global kernel Fourier convolutional
neural networks (FourierNets). We show that FourierNets surpass existing deep
network based decoders at reconstructing photographs captured by the highly
non-local DiffuserCam optical encoder. Further, we show that FourierNets enable
E2E optimization of highly non-local optical encoders for 3D snapshot
microscopy. By combining FourierNets with a large-scale multi-GPU
differentiable optical simulation, we are able to optimize non-local optical
encoders 170$\times$ to 7372$\times$ larger than prior state of the art, and
demonstrate the potential for ROI-type specific optical encoding with a
programmable microscope.
