  Neural networks readily learn a subset of the modular arithmetic tasks, while
failing to generalize on the rest. This limitation remains unmoved by the
choice of architecture and training strategies. On the other hand, an
analytical solution for the weights of Multi-layer Perceptron (MLP) networks
that generalize on the modular addition task is known in the literature. In
this work, we (i) extend the class of analytical solutions to include modular
multiplication as well as modular addition with many terms. Additionally, we
show that real networks trained on these datasets learn similar solutions upon
generalization (grokking). (ii) We combine these "expert" solutions to
construct networks that generalize on arbitrary modular polynomials. (iii) We
hypothesize a classification of modular polynomials into learnable and
non-learnable via neural networks training; and provide experimental evidence
supporting our claims.
