  In this paper, we propose a new super-expressive activation function called
the Parametric Elementary Universal Activation Function (PEUAF). We demonstrate
the effectiveness of PEUAF through systematic and comprehensive experiments on
various industrial and image datasets, including CIFAR10, Tiny-ImageNet, and
ImageNet. Moreover, we significantly generalize the family of super-expressive
activation functions, whose existence has been demonstrated in several recent
works by showing that any continuous function can be approximated to any
desired accuracy by a fixed-size network with a specific super-expressive
activation function. Specifically, our work addresses two major bottlenecks in
impeding the development of super-expressive activation functions: the limited
identification of super-expressive functions, which raises doubts about their
broad applicability, and their often peculiar forms, which lead to skepticism
regarding their scalability and practicality in real-world applications.
