  We propose quadratic residual networks (QRes) as a new type of
parameter-efficient neural network architecture, by adding a quadratic residual
term to the weighted sum of inputs before applying activation functions. With
sufficiently high functional capacity (or expressive power), we show that it is
especially powerful for solving forward and inverse physics problems involving
partial differential equations (PDEs). Using tools from algebraic geometry, we
theoretically demonstrate that, in contrast to plain neural networks, QRes
shows better parameter efficiency in terms of network width and depth thanks to
higher non-linearity in every neuron. Finally, we empirically show that QRes
shows faster convergence speed in terms of number of training epochs especially
in learning complex patterns.
