  Data-driven machine learning approaches are being increasingly used to solve
partial differential equations (PDEs). They have shown particularly striking
successes when training an operator, which takes as input a PDE in some family,
and outputs its solution. However, the architectural design space, especially
given structural knowledge of the PDE family of interest, is still poorly
understood. We seek to remedy this gap by studying the benefits of weight-tied
neural network architectures for steady-state PDEs. To achieve this, we first
demonstrate that the solution of most steady-state PDEs can be expressed as a
fixed point of a non-linear operator. Motivated by this observation, we propose
FNO-DEQ, a deep equilibrium variant of the FNO architecture that directly
solves for the solution of a steady-state PDE as the infinite-depth fixed point
of an implicit operator layer using a black-box root solver and differentiates
analytically through this fixed point resulting in $\mathcal{O}(1)$ training
memory. Our experiments indicate that FNO-DEQ-based architectures outperform
FNO-based baselines with $4\times$ the number of parameters in predicting the
solution to steady-state PDEs such as Darcy Flow and steady-state
incompressible Navier-Stokes. Finally, we show FNO-DEQ is more robust when
trained with datasets with more noisy observations than the FNO-based
baselines, demonstrating the benefits of using appropriate inductive biases in
architectural design for different neural network based PDE solvers. Further,
we show a universal approximation result that demonstrates that FNO-DEQ can
approximate the solution to any steady-state PDE that can be written as a fixed
point equation.
