  We present both a theoretical and a methodological framework that addresses a
critical challenge in applying deep learning to physical systems: the
reconciliation of non-linear expressiveness with SO(3)-equivariance in
predictions of SO(3)-equivariant quantities. Inspired by covariant theory in
physics, we address this problem by exploring the mathematical relationships
between SO(3)-invariant and SO(3)-equivariant quantities and their
representations. We first construct theoretical SO(3)-invariant quantities
derived from the SO(3)-equivariant regression targets, and use these invariant
quantities as supervisory labels to guide the learning of high-quality
SO(3)-invariant features. Given that SO(3)-invariance is preserved under
non-linear operations, the encoding process for invariant features can
extensively utilize non-linear mappings, thereby fully capturing the non-linear
patterns inherent in physical systems. Building on this foundation, we propose
a gradient-based mechanism to induce SO(3)-equivariant encodings of various
degrees from the learned SO(3)-invariant features. This mechanism can
incorporate non-linear expressive capabilities into SO(3)-equivariant
representations, while theoretically preserving their equivariant properties as
we prove. We apply our theory and method to the electronic-structure
Hamiltonian prediction tasks, experimental results on eight benchmark databases
covering multiple types of elements and challenging scenarios show dramatic
breakthroughs on the state-of-the-art prediction accuracy, with improvements of
up to 40% in predicting Hamiltonians and up to 76% in predicting downstream
physical quantities such as occupied orbital energy. Our approach goes beyond
handling physical systems and offers a promising general solution to the
critical dilemma between equivariance and non-linear expressiveness for the
deep learning paradigm.
