  Activation functions are non-linearities in neural networks that allow them
to learn complex mapping between inputs and outputs. Typical choices for
activation functions are ReLU, Tanh, Sigmoid etc., where the choice generally
depends on the application domain. In this work, we propose a
framework/strategy that unifies several works on activation functions and
theoretically explains the performance benefits of these works. We also propose
novel techniques that originate from the framework and allow us to obtain
``extensions'' (i.e. special generalizations of a given neural network) of
neural networks through operations on activation functions. We theoretically
and empirically show that ``extensions'' of neural networks have performance
benefits compared to vanilla neural networks with insignificant space and time
complexity costs on standard test functions. We also show the benefits of
neural network ``extensions'' in the time-series domain on real-world datasets.
