  Deep neural networks (DNNs) have substantial computational requirements,
which greatly limit their performance in resource-constrained environments.
Recently, there are increasing efforts on optical neural networks and optical
computing based DNNs hardware, which bring significant advantages for deep
learning systems in terms of their power efficiency, parallelism and
computational speed. Among them, free-space diffractive deep neural networks
(D$^2$NNs) based on the light diffraction, feature millions of neurons in each
layer interconnected with neurons in neighboring layers. However, due to the
challenge of implementing reconfigurability, deploying different DNNs
algorithms requires re-building and duplicating the physical diffractive
systems, which significantly degrades the hardware efficiency in practical
application scenarios. Thus, this work proposes a novel hardware-software
co-design method that enables robust and noise-resilient Multi-task Learning in
D$^2$NNs. Our experimental results demonstrate significant improvements in
versatility and hardware efficiency, and also demonstrate the robustness of
proposed multi-task D$^2$NN architecture under wide noise ranges of all system
components. In addition, we propose a domain-specific regularization algorithm
for training the proposed multi-task architecture, which can be used to
flexibly adjust the desired performance for each task.
