  Nonlinear activation functions are pivotal to the success of deep neural
nets, and choosing the appropriate activation function can significantly affect
their performance. Most networks use fixed activation functions (e.g., ReLU,
GELU, etc.), and this choice might limit their expressiveness. Furthermore,
different layers may benefit from diverse activation functions. Consequently,
there has been a growing interest in trainable activation functions. In this
paper, we introduce DiTAC, a trainable highly-expressive activation function
based on an efficient diffeomorphic transformation (called CPAB). Despite
introducing only a negligible number of trainable parameters, DiTAC enhances
model expressiveness and performance, often yielding substantial improvements.
It also outperforms existing activation functions (regardless whether the
latter are fixed or trainable) in tasks such as semantic segmentation, image
generation, regression problems, and image classification. Our code is
available at https://github.com/BGU-CS-VIL/DiTAC.
