  Diffractive optical neural networks have shown promising advantages over
electronic circuits for accelerating modern machine learning (ML) algorithms.
However, it is challenging to achieve fully programmable all-optical
implementation and rapid hardware deployment. Furthermore, understanding the
threat of adversarial ML in such system becomes crucial for real-world
applications, which remains unexplored. Here, we demonstrate a large-scale,
cost-effective, complex-valued, and reconfigurable diffractive all-optical
neural networks system in the visible range based on cascaded transmissive
twisted nematic liquid crystal spatial light modulators. With the assist of
categorical reparameterization, we create a physics-aware training framework
for the fast and accurate deployment of computer-trained models onto optical
hardware. Furthermore, we theoretically analyze and experimentally demonstrate
physics-aware adversarial attacks onto the system, which are generated from a
complex-valued gradient-based algorithm. The detailed adversarial robustness
comparison with conventional multiple layer perceptrons and convolutional
neural networks features a distinct statistical adversarial property in
diffractive optical neural networks. Our full stack of software and hardware
provides new opportunities of employing diffractive optics in a variety of ML
tasks and enabling the research on optical adversarial ML.
