  Recently, there are increasing efforts on advancing optical neural networks
(ONNs), which bring significant advantages for machine learning (ML) in terms
of power efficiency, parallelism, and computational speed. With the
considerable benefits in computation speed and energy efficiency, there are
significant interests in leveraging ONNs into medical sensing, security
screening, drug detection, and autonomous driving. However, due to the
challenge of implementing reconfigurability, deploying multi-task learning
(MTL) algorithms on ONNs requires re-building and duplicating the physical
diffractive systems, which significantly degrades the energy and cost
efficiency in practical application scenarios. This work presents a novel ONNs
architecture, namely, \textit{RubikONNs}, which utilizes the physical
properties of optical systems to encode multiple feed-forward functions by
physically rotating the hardware similarly to rotating a \textit{Rubik's Cube}.
To optimize MTL performance on RubikONNs, two domain-specific physics-aware
training algorithms \textit{RotAgg} and \textit{RotSeq} are proposed. Our
experimental results demonstrate more than 4$\times$ improvements in energy and
cost efficiency with marginal accuracy degradation compared to the
state-of-the-art approaches.
